OS2World OLD-STATIC-BACKUP Forum

OS/2 - SIGs => Rexx => Topic started by: jep on 2008.05.21, 15:37:00

Title: Fetching things from the net
Post by: jep on 2008.05.21, 15:37:00
Marked as: Advanced
Hello,

here's some code for you to fetch pages with rexx. I used it in my script to retrieve translations but removed it in favour of wget to simplify that script. The included script can be used if you don't want to rely on other software and shouldn't fetch large files over the net. It use rxSock that's included with your eComStation and newer OS/2 installation and you can read more about it in the help files. You can also look for wwwget.zip at hobbes to get another hands on example, or just Google the net.

You need to be connected to the net to try this script as it'll retrieve a page and save it to your harddrive. The page ( http://www.edm2.com/0202/tcpiprexx.html ) contain an article about rxSock from edm2.

Solution included in this script for you to study:
How to load the external library (dll) for rxSock. It's very similar to rexxutil.
Retrieving data from the net
Determine different parts of a string that represent the URL in this case.
Reuse information to retrieve/build portions of the string for futher processing (GetURLComponent)
Translate %20, %28, %29 etc. to real characters

You can use rxSock to create a simple chat application or a script to retrieve data over the net. There's even a server called "The SREhttp/2 WWW Server" written in rexx. It's not that known nor fast, but I use it as it work.

Note that the script may require some tweaking to be compatible with various web servers.

//Jan-Erik

/* Retrieve HTML page from the net */

/* Load REXX Utility library */
CALL RxFuncAdd 'SysLoadFuncs', 'REXXUTIL', 'SysLoadFuncs'
CALL SysLoadFuncs

/* Load REXX Socket library */
CALL RxFuncAdd 'SockLoadFuncs', 'rxSock', 'SockLoadFuncs'
CALL SockLoadFuncs

Call SysCLs

IF Resolve( 'http://www.edm2.com/0202/tcpiprexx.html' ) = 1 THEN
  IF Connect() = 0 THEN
    IF Fetch() = 0 THEN DO
      CALL rxOut 3, 'Saving 'FILESPEC( 'N', host.path )
      CALL CHAROUT FILESPEC( 'N', host.path ), host.file
    END
IF rc \= 0 THEN CALL SAY rc
RETURN 0


/*************** Fetch file **********************/
Fetch:
elapsed = TIME( 'R' )
rc = SockSend( socket, 'GET /'||host.path||' HTTP/1.1'||"0D0A"x||'HOST:'||host.location||"0D0A0D0A"x )
host.file = ''
k = 0
DO WHILE k = 0
  IF SockRecv( socket, 'sReceive', 256 ) < 1 THEN
    LEAVE
  host.file = host.file||CR2CRLF( sReceive )
  PARSE VALUE TRANSLATE( host.file ) WITH 'CONTENT-LENGTH: 'size . '<' .
  k = STRIP( size ) /* Different servers may answer differently, sometimes in HEX so you have to convert the value with X2D() */
END
j = POS( '<', host.file )
chunked = ( POS( 'TRANSFER-ENCODING: CHUNKED', TRANSLATE( err ) ) > 0 )
IF j > 0 THEN
  host.file = SUBSTR( host.file, j )
DO FOREVER
   IF TIME( 'E' ) > elapsed + 1 THEN
   DO
      CALL rxMeter j, k, 2
      elapsed = TIME( 'E' )
   END
   rc = SockRecv( socket, 'sReceive', 256 )
   host.file = host.file||CR2CRLF( sReceive )
   j = LENGTH( host.file )
   IF rc < 1 THEN
      LEAVE
END
j = LENGTH( host.file )
CALL rxMeter j, k, 2
RETURN rc

/*************** Disconnect from host **********************/
Disconnect:
rc = SockShutDown( socket, 2 )
rc = SockClose( socket )
RETURN rc

/*************** Connect to host **********************/
Connect:
socket = SockSocket( host.family, 'SOCK_STREAM', 0 )
IF socket < 0 THEN
  rc = -3
ELSE IF SockConnect( socket, 'host.' ) < 0 THEN DO
  CALL Disconnect( socket )
  rc = -4
END
RETURN 0


/*************** Resolve host **********************/
Resolve:
IF ARG( 1 ) = '' THEN
  RETURN 0
ELSE DO
  CALL GetURLComponents DeCodeURL( SPACE( TRANSLATE( ARG( 1 ), '', ' "' ) ) )
  host.family = 'AF_INET'
  rc = 1
  IF SockGetHostByName( host.location, 'host.' ) <> 1 THEN
    rc = -2
END
RETURN rc

/*************** Helper functions below **********************/

/* CR -> CRLF */
CR2CRLF:
RETURN TRANSLATE( ARG( 1 ),,"0D0A"x )

/* Translate %20, %28, %29 etc. to real characters */
DeCodeURL: PROCEDURE
PARSE ARG url
DO WHILE POS( '%', url ) > 0
   i = POS( '%', url )
   url = SUBSTR( url, 1, i - 1 ) || X2C(SUBSTR( url, i + 1, 2 )) || SUBSTR( url, i + 3 )
END
RETURN url

GetURLComponents: PROCEDURE EXPOSE host.
/* URL syntax consists of these components:          */
/* <scheme>://<location>:<port>/<path><params>?<query>#<target> */
 
PARSE VALUE ARG( 1 ) WITH host.scheme '://' host.location '/' host.path
host.location = strip( host.location, 'L', '/' ) /* in case of scheme like "file:///" */

IF host.location = '' THEN
  PARSE VALUE ARG( 1 ) WITH host.location '/' host.path

PARSE VALUE host.location WITH host.location ':' host.port
IF host.port = '' THEN
  IF TRANSLATE( host.scheme ) = 'HTTP' then
    host.port = 80
  ELSE IF TRANSLATE( host.scheme ) = 'FTP' then
    host.port = 21

PARSE VALUE host.path WITH host.file '' host.params '?' host.query '#' host.target

IF host.params = '' THEN
  PARSE VALUE host.path WITH host.file '?' host.query '#' host.target

IF host.query = '' THEN
  PARSE VALUE host.path WITH host.file '#' host.target
RETURN

GetURLComponent: PROCEDURE EXPOSE host.

CALL GetURLComponents ARG( 1 )

SELECT
  WHEN TRANSLATE( ARG( 2 ) ) = 'S' THEN
    RETURN host.scheme
  WHEN TRANSLATE( ARG( 2 ) ) = 'H' THEN
    RETURN host.location||':'||host.port
  WHEN TRANSLATE( ARG( 2 ) ) = 'N' THEN
    RETURN host.location
  WHEN TRANSLATE( ARG( 2 ) ) = 'L' THEN
    RETURN host.port
  WHEN TRANSLATE( ARG( 2 ) ) = 'P' THEN
    RETURN host.param
  WHEN TRANSLATE( ARG( 2 ) ) = 'Q' THEN
    RETURN host.query
  WHEN TRANSLATE( ARG( 2 ) ) = 'T' THEN
    RETURN host.target
  WHEN TRANSLATE( ARG( 2 ) ) = 'F' THEN
    RETURN FILESPEC( 'N', host.path )
OTHERWISE
  RETURN host.path
END

ToLower: PROCEDURE
RETURN TRANSLATE( ARG(1), XRANGE( 'a', 'z' )||'äöüß', XRANGE( 'A', 'Z' )||'ÄÖÜß' )

rxMeter: Procedure
    progress = 76 * ARG(1) % ARG(2)
    call rxOut ARG(3), left( copies( '█', progress )||copies( '░', 76 - progress ), 76 )||right( ( ( 100 * ARG(1) ) % ARG(2) )||'%', 4 )
Return 0
   
rxOut: procedure
    if datatype( ARG(1), 'W' ) then
        parse value SysCurPos( ARG(1), 0 ) with prev_row prev_col
    say left( ARG(2), max( 79, min( length( ARG(2) ), 80 ) ) )
    if datatype( ARG(1), 'W' ) then
        call SysCurPos prev_row, prev_col
Return 0