Marked as: Advanced
Hello,
here's some code for you to fetch pages with rexx. I used it in my script to retrieve translations but removed it in favour of wget to simplify that script. The included script can be used if you don't want to rely on other software and shouldn't fetch large files over the net. It use rxSock that's included with your eComStation and newer OS/2 installation and you can read more about it in the help files. You can also look for wwwget.zip at hobbes to get another hands on example, or just Google the net.
You need to be connected to the net to try this script as it'll retrieve a page and save it to your harddrive. The page ( http://www.edm2.com/0202/tcpiprexx.html ) contain an article about rxSock from edm2.
Solution included in this script for you to study:
How to load the external library (dll) for rxSock. It's very similar to rexxutil.
Retrieving data from the net
Determine different parts of a string that represent the URL in this case.
Reuse information to retrieve/build portions of the string for futher processing (GetURLComponent)
Translate %20, %28, %29 etc. to real characters
You can use rxSock to create a simple chat application or a script to retrieve data over the net. There's even a server called "The SREhttp/2 WWW Server" written in rexx. It's not that known nor fast, but I use it as it work.
Note that the script may require some tweaking to be compatible with various web servers.
//Jan-Erik
/* Retrieve HTML page from the net */
/* Load REXX Utility library */
CALL RxFuncAdd 'SysLoadFuncs', 'REXXUTIL', 'SysLoadFuncs'
CALL SysLoadFuncs
/* Load REXX Socket library */
CALL RxFuncAdd 'SockLoadFuncs', 'rxSock', 'SockLoadFuncs'
CALL SockLoadFuncs
Call SysCLs
IF Resolve( 'http://www.edm2.com/0202/tcpiprexx.html' ) = 1 THEN
IF Connect() = 0 THEN
IF Fetch() = 0 THEN DO
CALL rxOut 3, 'Saving 'FILESPEC( 'N', host.path )
CALL CHAROUT FILESPEC( 'N', host.path ), host.file
END
IF rc \= 0 THEN CALL SAY rc
RETURN 0
/*************** Fetch file **********************/
Fetch:
elapsed = TIME( 'R' )
rc = SockSend( socket, 'GET /'||host.path||' HTTP/1.1'||"0D0A"x||'HOST:'||host.location||"0D0A0D0A"x )
host.file = ''
k = 0
DO WHILE k = 0
IF SockRecv( socket, 'sReceive', 256 ) < 1 THEN
LEAVE
host.file = host.file||CR2CRLF( sReceive )
PARSE VALUE TRANSLATE( host.file ) WITH 'CONTENT-LENGTH: 'size . '<' .
k = STRIP( size ) /* Different servers may answer differently, sometimes in HEX so you have to convert the value with X2D() */
END
j = POS( '<', host.file )
chunked = ( POS( 'TRANSFER-ENCODING: CHUNKED', TRANSLATE( err ) ) > 0 )
IF j > 0 THEN
host.file = SUBSTR( host.file, j )
DO FOREVER
IF TIME( 'E' ) > elapsed + 1 THEN
DO
CALL rxMeter j, k, 2
elapsed = TIME( 'E' )
END
rc = SockRecv( socket, 'sReceive', 256 )
host.file = host.file||CR2CRLF( sReceive )
j = LENGTH( host.file )
IF rc < 1 THEN
LEAVE
END
j = LENGTH( host.file )
CALL rxMeter j, k, 2
RETURN rc
/*************** Disconnect from host **********************/
Disconnect:
rc = SockShutDown( socket, 2 )
rc = SockClose( socket )
RETURN rc
/*************** Connect to host **********************/
Connect:
socket = SockSocket( host.family, 'SOCK_STREAM', 0 )
IF socket < 0 THEN
rc = -3
ELSE IF SockConnect( socket, 'host.' ) < 0 THEN DO
CALL Disconnect( socket )
rc = -4
END
RETURN 0
/*************** Resolve host **********************/
Resolve:
IF ARG( 1 ) = '' THEN
RETURN 0
ELSE DO
CALL GetURLComponents DeCodeURL( SPACE( TRANSLATE( ARG( 1 ), '', ' "' ) ) )
host.family = 'AF_INET'
rc = 1
IF SockGetHostByName( host.location, 'host.' ) <> 1 THEN
rc = -2
END
RETURN rc
/*************** Helper functions below **********************/
/* CR -> CRLF */
CR2CRLF:
RETURN TRANSLATE( ARG( 1 ),,"0D0A"x )
/* Translate %20, %28, %29 etc. to real characters */
DeCodeURL: PROCEDURE
PARSE ARG url
DO WHILE POS( '%', url ) > 0
i = POS( '%', url )
url = SUBSTR( url, 1, i - 1 ) || X2C(SUBSTR( url, i + 1, 2 )) || SUBSTR( url, i + 3 )
END
RETURN url
GetURLComponents: PROCEDURE EXPOSE host.
/* URL syntax consists of these components: */
/* <scheme>://<location>:<port>/<path><params>?<query>#<target> */
PARSE VALUE ARG( 1 ) WITH host.scheme '://' host.location '/' host.path
host.location = strip( host.location, 'L', '/' ) /* in case of scheme like "file:///" */
IF host.location = '' THEN
PARSE VALUE ARG( 1 ) WITH host.location '/' host.path
PARSE VALUE host.location WITH host.location ':' host.port
IF host.port = '' THEN
IF TRANSLATE( host.scheme ) = 'HTTP' then
host.port = 80
ELSE IF TRANSLATE( host.scheme ) = 'FTP' then
host.port = 21
PARSE VALUE host.path WITH host.file '' host.params '?' host.query '#' host.target
IF host.params = '' THEN
PARSE VALUE host.path WITH host.file '?' host.query '#' host.target
IF host.query = '' THEN
PARSE VALUE host.path WITH host.file '#' host.target
RETURN
GetURLComponent: PROCEDURE EXPOSE host.
CALL GetURLComponents ARG( 1 )
SELECT
WHEN TRANSLATE( ARG( 2 ) ) = 'S' THEN
RETURN host.scheme
WHEN TRANSLATE( ARG( 2 ) ) = 'H' THEN
RETURN host.location||':'||host.port
WHEN TRANSLATE( ARG( 2 ) ) = 'N' THEN
RETURN host.location
WHEN TRANSLATE( ARG( 2 ) ) = 'L' THEN
RETURN host.port
WHEN TRANSLATE( ARG( 2 ) ) = 'P' THEN
RETURN host.param
WHEN TRANSLATE( ARG( 2 ) ) = 'Q' THEN
RETURN host.query
WHEN TRANSLATE( ARG( 2 ) ) = 'T' THEN
RETURN host.target
WHEN TRANSLATE( ARG( 2 ) ) = 'F' THEN
RETURN FILESPEC( 'N', host.path )
OTHERWISE
RETURN host.path
END
ToLower: PROCEDURE
RETURN TRANSLATE( ARG(1), XRANGE( 'a', 'z' )||'äöüß', XRANGE( 'A', 'Z' )||'ÄÖÜß' )
rxMeter: Procedure
progress = 76 * ARG(1) % ARG(2)
call rxOut ARG(3), left( copies( '█', progress )||copies( '░', 76 - progress ), 76 )||right( ( ( 100 * ARG(1) ) % ARG(2) )||'%', 4 )
Return 0
rxOut: procedure
if datatype( ARG(1), 'W' ) then
parse value SysCurPos( ARG(1), 0 ) with prev_row prev_col
say left( ARG(2), max( 79, min( length( ARG(2) ), 80 ) ) )
if datatype( ARG(1), 'W' ) then
call SysCurPos prev_row, prev_col
Return 0