Author Topic: REXX and double quoted INPUT parameter strings  (Read 8006 times)

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
REXX and double quoted INPUT parameter strings
« on: September 14, 2020, 02:18:09 am »
So let me set the context of the question: I wrote a little REXX script to automate the execution of RSYNC. The goal is to have this scheduled in CRON2 to allow me to copy the contents of the LOCAL drives over to the NAS box, or if needed to a LOCAL to LOCAL copy (such as a backup run to another HDD).

The INPUT parameters to the REXX script are various LOCAL folders as well as the NAS box folder where I'm going to copy the stuff to, basically the SOURCE and TARGET folder destinations.

So while I have this working well enough to handle any folder names where no space exists in the name:

1) G:\DESKTOP\AUTOMOTIVE

...I have ran into some difficulties trying to get this working for the folders that do have a space:

2) G:\DESKTOP\MISC INFO

Clearly passing such a folder name as an un-double quoted value just causes the REXX script to interpret the first space as a argument separator, subsequently:

CLI  => rsync_backup NAS G:\DESKTOP\MISC INFO\ V:\PUBLIC\DOCUMENTS\MISC INFO

causes the following args to be identified:

Code: [Select]
SOURCE => G:\DESKTOP\MISC
TARGET => INFO\ V:\PUBLIC\DOCUMENTS\MISC INFO

Now if I do apply the double quotes, just like one would need to when attempting to work with such a folder through a CLI window:

CLI => rsync_backup NAS "G:\DESKTOP\MISC INFO\" "V:\PUBLIC\DOCUMENTS\MISC INFO"

causes the following args to be identified:

Code: [Select]
SOURCE => "G:\DESKTOP\MISC
TARGET => INFO\" "V:\PUBLIC\DOCUMENTS\MISC INFO"

The REXX code I use to pull in the arguments is pretty basic:

Code: [Select]
/* parse the command line to pick up the BACKUP_TYPE, SOURCE and TARGET locations */
arg backup_type source target

I looked through all my REXX PDFs and INFs but haven't found anything that spells out how to get around this.

So...any ideas how to handle this?

Thanks!

FYI: I've attached the actual rsync_backup.cmd file, but with a TXT extension.

Pete

  • Hero Member
  • *****
  • Posts: 1281
  • Karma: +9/-0
    • View Profile
Re: REXX and double quoted INPUT parameter strings
« Reply #1 on: September 14, 2020, 04:10:28 am »
Hi Dariusz

Have you tried omitting the 2 middle double quotes ie

 rsync_backup NAS "G:\DESKTOP\MISC INFO\ V:\PUBLIC\DOCUMENTS\MISC INFO"


Regards

Pete


Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
Re: REXX and double quoted INPUT parameter strings
« Reply #2 on: September 14, 2020, 05:35:12 am »
Hey Pete!

..Have you tried omitting the 2 middle double quotes ie

 rsync_backup NAS "G:\DESKTOP\MISC INFO\ V:\PUBLIC\DOCUMENTS\MISC INFO"
...

Yup, tried that as well, the darn space is still being treated as the parm separator:

Code: [Select]
SOURCE => "G:\DESKTOP\MISC
TARGET => INFO\ V:\PUBLIC\DOCUMENTS\MISC INFO"

BUUUTTTT...that gave me an idea: why not try to use the arg separator to tell REXX which special character divides up the SOURCE from TARGET?

So I now have the following code:

Code: [Select]
arg backup_type source"-"target

where the "-" tells REXX to look for that dash in the input CLI.

Now I can actually kick things off with: rsync_backup NAS "G:\DESKTOP\MISC INFO\"-"V:\PUBLIC\DOCUMENTS\MISC INFO"

which gives me the following result:

Code: [Select]
SOURCE => "G:\DESKTOP\MISC INFO\"
TARGET => "V:\PUBLIC\DOCUMENTS\MISC INFO"

So at least I passed that hurdle, what remains now is to figure out why rsync itself chokes on those parameters being passed as-is:

Code: [Select]
...
RSYNC CLI => --dry-run --archive --delete --progress --stats --whole-file --no-xattrs --human-readable --quiet --exclude-from=G:\mptn\etc\rsync_exclude --log-file=G:\tmp\log\rsync_NAS_backup.log  "G:\DESKTOP\MISC INFO\" "V:\PUBLIC\DOCUMENTS\MISC INFO"

RSYNC started:     09/13/20 - 23:43:39...
[sender] change_dir chdir failed for G:/DESKTOP/MISC INFO" V:/PUBLIC/DOCUMENTS at util.c(1117)
rsync: change_dir "G:/DESKTOP/MISC INFO" V:/PUBLIC/DOCUMENTS" failed: No such file or directory (2)
[sender] change_dir G:/DESKTOP/MISC INFO" V:/PUBLIC/DOCUMENTS failed at flist.c(404)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1168) [sender=3.0.9.1]
...

It looks to me like it does strip out some of the double quotes, but I'm not quite sure what logic it uses to do so. Trying with a single quote did not work, result was different, just not what I needed.
« Last Edit: September 14, 2020, 05:54:49 am by Dariusz Piatkowski »

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: REXX and double quoted INPUT parameter strings
« Reply #3 on: September 14, 2020, 07:27:38 am »
Why don't you just use strip (with " leading and trailing) after separating the arguments?

Andreas Schnellbacher

  • Hero Member
  • *****
  • Posts: 827
  • Karma: +14/-0
    • View Profile
Re: REXX and double quoted INPUT parameter strings
« Reply #4 on: September 14, 2020, 11:40:47 am »
Here's how to parse enquoted args correctly. It allows for multiple args, each may be enclosed in double quotes.

First the output:
Code: [Select]
> TestParseArgs.cmd "G:\DESKTOP\MISC INFO\" "V:\PUBLIC\DOCUMENTS\MISC INFO"
1: Rest = ["G:\DESKTOP\MISC INFO\" "V:\PUBLIC\DOCUMENTS\MISC INFO"]
1: Next = [G:\DESKTOP\MISC INFO\]
2: Rest = ["V:\PUBLIC\DOCUMENTS\MISC INFO"]
2: Next = [V:\PUBLIC\DOCUMENTS\MISC INFO]
Arg.1 = G:\DESKTOP\MISC INFO\
Arg.2 = V:\PUBLIC\DOCUMENTS\MISC INFO

Here's the script:
Code: [Select]
/* Parse multiple args optionally enclosed in double quotes */

/* Parse args */
PARSE ARG Args
Args = STRIP( Args)

/* Init vars */
Arg. = ''
Arg.0 = 0
Rest = Args

/* Parse Rest */
DO WHILE Rest <> ''
   /* Advance arg counter */
   NumArgs = Arg.0 + 1
   SAY NumArgs': Rest = ['Rest']'
   
   /* Parse Next from Rest */
   IF LEFT( Rest, 1) = '"' THEN
      PARSE VAR Rest '"'Next'"' Rest
   ELSE   
      PARSE VAR Rest Next Rest
   SAY NumArgs': Next = ['Next']'

   /* Strip multiple spaces between args */     
   Rest = STRIP( Rest)

   /* Save Next to Arg. array */
   Arg.NumArgs = Next
   Arg.0 = NumArgs
END

/* Test Arg. array */
DO i = 1 TO Arg.0
   SAY 'Arg.'i' = 'Arg.i
END

EXIT


Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
Re: REXX and double quoted INPUT parameter strings
« Reply #5 on: September 14, 2020, 03:46:05 pm »
Hey Lars,

Why don't you just use strip (with " leading and trailing) after separating the arguments?

I'm not sure I follow.

If you are suggesting that I use strip to remove the " after I successfully recognize the SOURCE & TARGET folder names alright, I am following you.

The problem though becomes that rysnc still fails since the OS/2 way of handling such a path name is to wrap in double quotes.

Therefore, the following CLI:

rsync --dry-run --archive --delete --progress --stats --whole-file --no-xattrs --human-readable --quiet --exclude-from=G:\mptn\etc\rsync_exclude --log-file=G:tmp\log\rsync_NAS_backup.log  G:\DESKTOP\MISC INFO\ V:\PUBLIC\DOCUMENTS\MISC INFO

still gives me the following error from rsync:

Code: [Select]
rsync: link_stat "G:/DESKTOP/MISC" failed: No such file or directory (2)
[sender] change_dir chdir failed for INFO at util.c(1146)
rsync: change_dir "G:/INFO" failed: No such file or directory (2)
[sender] change_dir INFO failed at flist.c(404)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1168) [sender=3.0.9.1]

So I wonder if perhaps the bigger problem here is that our port of rsync does not handle the native OS/2 way of passing path names which contain a space in it?

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
Re: REXX and double quoted INPUT parameter strings
« Reply #6 on: September 14, 2020, 03:47:39 pm »
Hi Andreas!!!

Here's how to parse enquoted args correctly. It allows for multiple args, each may be enclosed in double quotes....

Aha...SUPER!

Yes, I will use that approach, it beats having to have that awkward separator in the CLI.

Thank you.

Andreas Schnellbacher

  • Hero Member
  • *****
  • Posts: 827
  • Karma: +14/-0
    • View Profile
Re: REXX and double quoted INPUT parameter strings
« Reply #7 on: September 14, 2020, 04:27:26 pm »
Code: [Select]
rsync --dry-run --archive --delete --progress --stats --whole-file --no-xattrs
 --human-readable --quiet --exclude-from=G:\mptn\etc\rsync_exclude
 --log-file=G:tmp\log\rsync_NAS_backup.log
  G:\DESKTOP\MISC INFO\ V:\PUBLIC\DOCUMENTS\MISC INFO

still gives me the following error from rsync:
[...]
Sure. Have you tried to enquote the dirnames?
Code: [Select]
rsync ... "G:\DESKTOP\MISC INFO\" "V:\PUBLIC\DOCUMENTS\MISC INFO"

If you call that from a REXX script, you have two choices: Use surrounding single quotes for the entire rsync call including args or double all contained double quotes and surround it with double quotes.

The single quote option would be much easier to read.

For automatically enquoting a filename, if required, I use something like this:
Code: [Select]
Enquote:
   rc = 0

   PARSE ARG Filename

   IF VERIFY( Filename, ' &|<>', 'M') <> 0 THEN
      Filename = '"'Filename'"'

   RETURN( Filename)

Then you could change the rsync call to:
Code: [Select]
Cmd = 'rsync ... 'Enquote( 'G:\DESKTOP\MISC INFO\') Enquote( 'V:\PUBLIC\DOCUMENTS\MISC INFO')
Cmd
and use either single or double quotes to make the interpreter submit that to CMD.EXE.

Edit: Damned. The third version of the Cmd should finally work. (Much easier when you have syntax highlighting.)
« Last Edit: September 16, 2020, 07:40:52 pm by Andreas Schnellbacher »

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
Re: REXX and double quoted INPUT parameter strings
« Reply #8 on: September 26, 2020, 08:20:13 pm »
Alright you guys!!!

Andreas,
Thank you sir, I wholly took advantage of your sample code and made good use of it.

I've attached my current version of the rsync copy routine (rename the 'TXT' extension to 'CMD'). By all means, take a look, give me some feedback. Flip on the DEBUG flag in the code to get a pile more info on what's going on (as usual).

If you are going to run it keep in mind that the RSYNC '--dry-run' option is currently in-place to make sure that no actual changes are done. If you want RSYNC to actually modify the filesystem adjust accordingly by simply removing that one option.

Pretty usable at the moment, but the rsync options build could use a little refinement, so I'm going to re-work this to make it a little bit more self-explanatory to someone who does not know much about RSYNC.

Anyways, here is how you run it (sample params for my use here):

Code: [Select]
rsync_backup.cmd NAS "G:/DESKTOP/Misc Info" "V:/public/documents"

But to make it usable for the cron2 utility I simply tossed this into a two liner CMD file as follows:

Code: [Select]
@echo off
rsync_backup.cmd NAS "G:/DESKTOP/Misc Info" "V:/public/documents"

This is the output in the console:

Code: [Select]
[G:\util\misc]rsync_backup_AUTOMOTIVE.cmd
Starting RSYNC for LOCAL to NAS backup!

SOURCE_FOLDER => G:/DESKTOP/Automotive
TARGET_FOLDER => V:/public/documents

===============================================================================
RSYNC started:     09/26/20 - 13:58:59...

SOURCE_FOLDER => G:/DESKTOP/Automotive
TARGET_FOLDER => V:/public/documents

RSYNC completed:   09/26/20 - 14:00:16...

RSYNC took 77.260000 seconds to complete the NAS backup task!
===============================================================================

...while this gets sent to the rsync LOG:

Code: [Select]
===============================================================================
RSYNC started:     09/26/20 - 13:58:59...

SOURCE_FOLDER => G:/DESKTOP/Automotive
TARGET_FOLDER => V:/public/documents

RSYNC CLI =>--dry-run --archive --delete --progress --stats --whole-file --no-xattrs --human-readable --quiet --exclude-from=G:\mptn\etc\rsync_exclude --log-file=G:\tmp\log\rsync_NAS_backup.log  G:/DESKTOP/Automotive V:/public/documents

-----------------------------------------
2020/09/26 13:58:59 [516] building file list
2020/09/26 13:59:29 [516] .d..t...... Automotive/Mopar/Tech Info/ENGINE/Oil System/
2020/09/26 14:00:16 [516] Number of files: 7281
2020/09/26 14:00:16 [516] Number of files transferred: 5
2020/09/26 14:00:16 [516] Total file size: 14.42G bytes
2020/09/26 14:00:16 [516] Total transferred file size: 340.09K bytes
2020/09/26 14:00:16 [516] Literal data: 0 bytes
2020/09/26 14:00:16 [516] Matched data: 0 bytes
2020/09/26 14:00:16 [516] File list size: 195.46K
2020/09/26 14:00:16 [516] File list generation time: 0.001 seconds
2020/09/26 14:00:16 [516] File list transfer time: 0.000 seconds
2020/09/26 14:00:16 [516] Total bytes sent: 197.78K
2020/09/26 14:00:16 [516] Total bytes received: 2.32K
2020/09/26 14:00:16 [516] sent 197.78K bytes  received 2.32K bytes  2.55K bytes/sec  elapsed 00:01:18
2020/09/26 14:00:16 [516] total size is 14.42G  speedup is 72068.45 (DRY RUN)
-----------------------------------------

RSYNC completed:   09/26/20 - 14:00:16...

RSYNC took 77.260000 seconds to complete the NAS backup task!
===============================================================================

Andreas Schnellbacher

  • Hero Member
  • *****
  • Posts: 827
  • Karma: +14/-0
    • View Profile
Re: REXX and double quoted INPUT parameter strings
« Reply #9 on: September 27, 2020, 07:49:41 pm »
I simply tossed this into a two liner CMD
An option is to insert '@ECHO OFF' as the second line in the REXX file.

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
Re: REXX and double quoted INPUT parameter strings
« Reply #10 on: October 12, 2020, 10:44:08 pm »
Hi everyone!

Alright, so I took a little more time to re-do parts of the logic, we have a lot more user control now that allow you to fine-tune the script to your particular system configuration. Right at the front in the 'main:' you've got some variables that point to RSYNC, various control files you might be using, etc. Stuff that was previously hard-coded is now defined with some defaults (still specific to my config), but which are all now found in one spot and can be easily adjusted.

Take a look please at the attached script, change the extension if you intend to run.

By DEFAULT this will execute RSYNC with --dry-run option, so no actual writes will happen. If you want to use this just head on over to the build_rsync_parameter_arrays procedure which actually defines this using an array. This gives us a fully dynamic way to adjust what level of optional other RSYNC parameters you want to utilize. If you need more, just execute 'man rsync' at CLI, grab whatever options you want and toss that in as an extra array entry. Don't forget to adjust the TOTAL count of all options in the matching rsync_param_local.0.0 or rsync_param_remote.0.0 entry.

By all means feedback is always welcome. This was a fun exercise to develop a much better REXX on our platform understanding. Appreciate everyone's help in getting me there!

Andreas Schnellbacher

  • Hero Member
  • *****
  • Posts: 827
  • Karma: +14/-0
    • View Profile
Re: REXX and double quoted INPUT parameter strings
« Reply #11 on: October 13, 2020, 01:57:11 am »
Hi Dariusz,

Please exchange the quotes in the Enquote func

from:
      Filename = "'"Filename"'"    (double quote - single quote - double quote)
to:
      Filename = '"'Filename'"'    (single quote - double quote - single quote)

Your version won't work for CMD.

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
Re: REXX and double quoted INPUT parameter strings
« Reply #12 on: October 13, 2020, 03:21:20 am »
Hi Andreas,

...Please exchange the quotes in the Enquote func

from:
      Filename = "'"Filename"'"    (double quote - single quote - double quote)
to:
      Filename = '"'Filename'"'    (single quote - double quote - single quote)

Your version won't work for CMD.

Hmm...I had originally used exactly the solution you initially suggested, but I found that it did not work for me. The only way I could make it work was by reversing these into that double-single-double combination. That is actually how I have been running it here ever since.

Please explain further the last comment...how does this not work for CMD? Do you mean the regular OS/2 CMD? This is the only thing I use here, and it's working fine.

In particular, the following code is what actually executes RSYNC:

Code: [Select]
      /* check to see if we have a user supplier RSYNC path */
      IF rsync_exe_file <> '' THEN
         '@'||rsync_exe_file||' '||rsync_cli
      ELSE
         /* if not, rely on RSYNC being somewhere in the PATH */
         '@'||'rsync'||' '||rsync_cli

and in case where either the SOURCE or TARGET folder names have a space in them, RSYNC needs that single quote around that folder name, for example:

Code: [Select]
===============================================================================
RSYNC started:     10/12/20 - 21:16:29...

SOURCE_FOLDER => G:/DESKTOP/Misc Info
TARGET_FOLDER => V:/public/documents

RSYNC CLI =>--dry-run --archive --delete --progress --stats --whole-file --no-xattrs --human-readable --quiet  --exclude-from=G:\mptn\etc\rsync_exclude --log-file=G:\tmp\log\rsync_REMOTE_backup.log 'G:/DESKTOP/Misc Info' V:/public/documents
-----------------------------------------
2020/10/12 21:16:29 [443] building file list
2020/10/12 21:16:45 [443] Number of files: 2382
2020/10/12 21:16:45 [443] Number of files transferred: 0
2020/10/12 21:16:45 [443] Total file size: 1.10G bytes
2020/10/12 21:16:45 [443] Total transferred file size: 0 bytes
2020/10/12 21:16:45 [443] Literal data: 0 bytes
2020/10/12 21:16:45 [443] Matched data: 0 bytes
2020/10/12 21:16:45 [443] File list size: 61.97K
2020/10/12 21:16:45 [443] File list generation time: 0.001 seconds
2020/10/12 21:16:45 [443] File list transfer time: 0.000 seconds
2020/10/12 21:16:45 [443] Total bytes sent: 62.99K
2020/10/12 21:16:45 [443] Total bytes received: 1.02K
2020/10/12 21:16:45 [443] sent 62.99K bytes  received 1.02K bytes  3.66K bytes/sec  elapsed 00:00:17
2020/10/12 21:16:45 [443] total size is 1.10G  speedup is 17243.86 (DRY RUN)
-----------------------------------------

RSYNC completed:   10/12/20 - 21:16:45...

RSYNC took 15.810000 seconds to complete the REMOTE backup task!
===============================================================================
But let me make that tweak and try it out...

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
Re: REXX and double quoted INPUT parameter strings
« Reply #13 on: October 13, 2020, 04:24:31 pm »
Hmm...not sure what to tell ya, other than: it still works fine with the double-quotes.

Here is the updated version with some additional clean-up and re-work.