OS2World OLD-STATIC-BACKUP Forum

OS/2 - SIGs => Rexx => Topic started by: RobertM on 2010.02.13, 04:27:03

Title: REXX DLL that can do https with form data?
Post by: RobertM on 2010.02.13, 04:27:03
Hey all,

This may be a silly question... it may even be one I asked (or even answered) in the past... so apologies if it's been asked and answered before (couldnt find it on the boards).

I am looking for a REXX Extension DLL that provides HTTPS support in a similar fashion that curl does. As a matter of fact, REXXCurl would be wonderful, if I could find a compiled version of it.

I am writing a small script that needs to do such things. Currently, it works great using Curl, but thus requires RxQueueing the output or sending the output to a file to be read. I'd prefer a better option.

I am presuming wGet will also work fine - but also seems to lack a REXX interface DLL.

Any suggestions are greatly appreciated.

Thanks,
Robert
Title: Re: REXX DLL that can do https with form data?
Post by: RobertM on 2010.02.13, 10:21:51
Basically, I've written a little script to read contacts from gMail, create new contacts, (and soon) update existing contacts.

Problem is the following (which is what requires the use of curl or wget):
(1) it has to send custom header information (the authentication stuff)
(2) it supposedly needs an https connection (may work still with http, but I think that feature will disappear)
(3) it has to "POST" data to the URL with "Content-Type: application/atom+xml" specified
(4) I need to be able to read the return code (400, 409, 201 generally)

The only other option I can think of is using RXSOCK stuff, but the problem I have had with using that in the past is it does not seem to honor the timeout value specified, meaning if the request hangs (Google acting flakey or whatever) the script will hang until it is killed.

Anyways, I figure with a little work, these scripts can be used to be able to pull one, all or certain Google/gMail contacts for use in other programs (by outputting them to whatever format the program uses). So, for instance, one can create a comma delimited file (or whatever other format) of email addresses for an email merge... or (if they actually put other info into their gMail contacts like I do since I use them on my Android G1 phone as well) pull the address info for a postal service mail merge type of thing.

And since gMail allows a plethora of information stored for each contact (actually, a LOT more than what the gMail interface itself allows) as well as custom information, maybe it can be a really neat tool.

My plans are using it with a web interface for a very simple invoice program I'm writing up so that when a new customer comes into our computer shop, I can create an entry in gMail to (a) send them email reminders for service, (b) have their phone number and address already in my phone from that point forward and so on.

So... that's the plan... worst case scenario, I can simply RxQueue the stuff and pop the response off the queue... not as elegant, but it will work.

REALLY not too elegant as I have to use curl's verbose mode, and RxQueue both the "displayable" response from the server as well as the "raw" output (the server's response codes the the "web browser") so I can read the return code (201, 400, 404, 409, etc). That leaves me TWO RxQueues to create and read and parse.

Well... that's where I am with that. And why I ask... any ideas are greatly welcomed.

Best,
Robert
Title: Re: REXX DLL that can do https with form data?
Post by: warpcafe on 2010.02.13, 14:59:42
Hey Robert,

I suppose you already went through the list of "usual suspects" on hobbes et cetera, right?
I did't check in details, but what about http://www.igor.co.nz/rexxweb.htm ? That just came to my mind...

Other than that I'm wondering if session/cookie handling etc. is something that relates as well... might be tricky to get this done. I was thinking of suggesting you to use something like Canoo webtest (a "robot" kinda thing based on htmlunit and others) for doing this... but then, even if it works after having managed the steep learning curve, it still is java (Ant) but not rexx...

Cheers,
Thomas
Title: Re: REXX DLL that can do https with form data?
Post by: RobertM on 2010.02.13, 18:55:23
Hi Thomas,

Quote from: warpcafe on 2010.02.13, 14:59:42
Hey Robert,

I suppose you already went through the list of "usual suspects" on hobbes et cetera, right?
I did't check in details, but what about http://www.igor.co.nz/rexxweb.htm ? That just came to my mind...

Yep, checked that out... it doesnt mention whether or not it can do https, but that aside, sending the custom header when it initiates the connection is an aspect that is is lacking (and this is the only place I've seen such a thing used for web during connection negotiation).

Quote from: warpcafe on 2010.02.13, 14:59:42
Other than that I'm wondering if session/cookie handling etc. is something that relates as well... might be tricky to get this done. I was thinking of suggesting you to use something like Canoo webtest (a "robot" kinda thing based on htmlunit and others) for doing this... but then, even if it works after having managed the steep learning curve, it still is java (Ant) but not rexx...

Cheers,
Thomas

Yeah, I've found Java implementations that work... (Google provides those) and even PHP and Python ones... I could go with any of those I guess, or I could even use their sample Javascript code which will integrate into my web apps nicely.

But, I figured I'd try to finish this in a way that others (without the need for running their own web server and creating their own web app) could use. That way, anyone else can simply grab the code... make it web enabled or not make it web enabled.

If I can keep it purely REXX (or REXX with a more elegant method of integrating with curl or wget), then anyone with a little programming experience can interact with most of Google's services (it's very very similar code for Google Buzz, Picasa, Google Calendar and so on).

Best,
Robert
Title: Re: REXX DLL that can do https with form data?
Post by: warpcafe on 2010.02.13, 21:01:14
Hey Robert,

that sounds like a plan. Wishing you all the best to achieve this ambitious target... sounds like a lot of work when thinking about it. Also, to some extent it's a re-invention of the wheel, albeit "forced to have to re-invent" since rexx doesn't offer the required features. I wonder if netrexx could ease the job... but then again, you'll end up having something Java and not useable like e.g. a simple rexx-DLL.
On the other hand, when looking at the wealth of free stuff floating around, there should be a lot of open source to take snippets from or even perhaps port them. But that would require to understand C and make it work on OS/2 before even thinking about how to create a rexx-callable DLL. There is so much stuff that I use from day to day at work which exist as plugins for Firefox (TamperData, web developer, xpath explorer), let alone the curl, wget and other stuff... ghee... there should be something almost ready-to-use.

Regards,
Thomas
Title: Re: REXX DLL that can do https with form data?
Post by: RobertM on 2010.02.14, 01:43:29
Quote from: warpcafe on 2010.02.13, 21:01:14
Hey Robert,

that sounds like a plan. Wishing you all the best to achieve this ambitious target... sounds like a lot of work when thinking about it. Also, to some extent it's a re-invention of the wheel, albeit "forced to have to re-invent" since rexx doesn't offer the required features.

Sadly, REXX seems to (and not) at the same time. RxSock will work admirably (IF I can figure out why it does not honor a connection timeout value - which may be an error on my part). Long before the days of RxWeb and so on, I used pure REXX code through RXSOCK to communicate with web, mail and newsgroup servers. Not too difficult, and thusly also very flexible.

The only remaining obstacle using RxSOCK is probably https support... but just like another little problem I have (which I am using PHP to get around) where I need to build a "special" sort of hash, without all the info and specs available, it's tough going that route. I'm pretty sure that by simply borrowing the certs table from curl or the likes and knowing how https actually works behind the scenes, that RxSock can do all of the same stuff... but dont have time to play with that.


Ah... what I'd do for a port of RexxCurl to OS/2.

Quote from: warpcafe on 2010.02.13, 21:01:14
I wonder if netrexx could ease the job... but then again, you'll end up having something Java and not useable like e.g. a simple rexx-DLL.
On the other hand, when looking at the wealth of free stuff floating around, there should be a lot of open source to take snippets from or even perhaps port them. But that would require to understand C and make it work on OS/2 before even thinking about how to create a rexx-callable DLL.


Exactly... and I am still trying to get some help on another project to create a rexx-callable dll. Need to find the time to get the source files to someone to work on it.

As for me, I haven't done C++ in a decade... got too lazy using REXX and never went back.

Quote from: warpcafe on 2010.02.13, 21:01:14
There is so much stuff that I use from day to day at work which exist as plugins for Firefox (TamperData, web developer, xpath explorer), let alone the curl, wget and other stuff... ghee... there should be something almost ready-to-use.

Regards,
Thomas

And there are some great PHP helper functions that would be really cool to have REXX enabled, like the PDF writer stuff and many of the plugins for Apache. Ah well...
Title: Re: REXX DLL that can do https with form data?
Post by: rwklein on 2010.02.14, 18:20:03
Quote from: RobertM on 2010.02.13, 04:27:03
Hey all,

This may be a silly question... it may even be one I asked (or even answered) in the past... so apologies if it's been asked and answered before (couldnt find it on the boards).

I am looking for a REXX Extension DLL that provides HTTPS support in a similar fashion that curl does. As a matter of fact, REXXCurl would be wonderful, if I could find a compiled version of it.

I am writing a small script that needs to do such things. Currently, it works great using Curl, but thus requires RxQueueing the output or sending the output to a file to be read. I'd prefer a better option.

I am presuming wGet will also work fine - but also seems to lack a REXX interface DLL.

Any suggestions are greatly appreciated.

Thanks,
Robert


Could you not somehow use stunnel ? It can connect to an SSL service. If you have a static address, like for example maybe you can hook it up gmail.com (I only quickly skimmed the thread) then let stunnel listen on the localhost and you let rexx talk to for example 127.0.0.1:81 ? What ever ? The trick is stunnel (if you can make it work) will do the SSL traffic and the traffic is also encrypted and the HTTP code will work just fine.

Goto smedley.info to find a build of stunnel.exe see www.stunnel.org for details and configuration examples.

Roderick
Title: Re: REXX DLL that can do https with form data?
Post by: RobertM on 2010.02.14, 23:44:18
Hi Roderick,

That should indeed do the trick, but still requires me figuring out why RxSock does not adhere to timeout settings... for now, I think I will curl and RxQueue the stuff until I have more time to dig into things...

Best,
Robert
Title: Re: REXX DLL that can do https with form data?
Post by: jep on 2010.02.15, 09:51:36
Hi Robert,

WGet does it nicely, but the lack of dll inteface has also bothered me.
Would redirection of the output from WGet suffice for you?

For example:
load a rexx dll (rxtnsion.dll)
call a function that start wget with the parameters you want, but return as soon as wget has started. (rxExecPrgm)
loop until done (rc=number).
 call another function that retrieve the text output from wget (last 1024 characters or so) for each call. (rxPrgmOut)
 parse the output and do whatever...
next
read the html file retrieved (if any).

I wrote those functions to test with CE2mp3 (http://www.os2world.com/forum/index.php/topic,1655.msg11722.html#msg11722) here in this forum.
The downside is that that it require the application to output 1024 characters for each call before it return with its present design. I can easily change that for you though. The "new" behaviour will probably then bee to specify a smaller value (<1024), unless someone else can tell me how to redesign the source code below to return data within a specified time frame instead... ;-)

/**
* function: rxExecPrgm( app_name, <parameters>, <environment> )
*
* This function returns an empty string if it couldn't start the application.
*/
APIRET APIENTRY __export rxExecPrgm( CHAR *name, ULONG numargs, RXSTRING  args[], char* queuename, RXSTRING *retstr )
{

  if( numargs < 1 )
     return INVALID_ROUTINE;

  var appName( args[0].strptr );
  var params( ( numargs > 1 ) ? args[1].strptr : "" );
  var appEnv( ( numargs > 2 ) ? args[2].strptr : "" );

   HPIPE hpW;
   CHAR szFailName[256];

   HFILE hfSave = -1, hfNew = HF_STDOUT;

   DosDupHandle( HF_STDOUT, &hfSave ); /* Saves standard output handle      */

   DosCreatePipe( &Output.hpR, &hpW, PIPESIZE ); /* Creates pipe                      */

   DosDupHandle( hpW, &hfNew ); /* Duplicates standard output handle */

   DosExecPgm( szFailName, sizeof( szFailName ), EXEC_ASYNCRESULT, params, appEnv, &Output.resCode, appName );  /* Starts child process      */

   DosClose( hpW );                /* Closes write handle to ensure     */
                                 /* Notification at child termination */
   DosDupHandle( hfSave, &hfNew );     /* Brings stdout back                */

  appName.cpy( "pid=" ).cat( (int) Output.resCode.codeTerminate ).set( retstr );

  Output.ulRead = 1;

  return VALID_ROUTINE;
}

/**
* function: rxPrgmOut( <none> )
*
* This function returns an empty string if it couldn't start the application.
*/
APIRET APIENTRY __export rxPrgmOut( CHAR *name, ULONG numargs, RXSTRING  args[], char* queuename, RXSTRING *retstr )
{
  char achBuf[1024];
  PID pidChild = 0;
  var tmpBuf;

  if ( Output.ulRead )
     DosRead( Output.hpR, achBuf, sizeof( achBuf ), &Output.ulRead ); //<- Need a redesign?

  if ( Output.ulRead ) {
     tmpBuf.cpy( Output.buffer ).cat( achBuf, Output.ulRead );
     Output.buffer.cpy( tmpBuf.splitAtLast( '\n', true ) );   //Remove delimiter
  } else {
     DosWaitChild( DCWA_PROCESS,    /* Look at only the process specified */
                DCWW_WAIT,       /* Wait until a child terminates      */
                &Output.resCode,        /* Termination codes returned         */
                &pidChild,       /* pid of terminating process         */
                Output.resCode.codeTerminate );   /* Process (pid) to look at */

     tmpBuf.cpy( "rc=" ).cat( (int) Output.resCode.codeTerminate );
  }

  tmpBuf.set( retstr );

  return VALID_ROUTINE;
}


//Jan-Erik