• Welcome to OS2World OLD-STATIC-BACKUP Forum.
 

News:

This is an old OS2World backup forum for reference only. IT IS READ ONLY!!!

If you need help with OS/2 - eComStation visit http://www.os2world.com/forum

Main Menu

RSS/XML Parsing

Started by RobertM, 2009.09.07, 05:32:13

Previous topic - Next topic

RobertM

Hey all,

I've recently been playing with some datasets from Amazon.com and needed to pull some data from the xml responses. I needed something simple that pulled data from the xml doc where there may be repeated elements, so I wrote up a very simple REXX procedure.


FindXMLValue: Procedure
Parse Arg xmlString,TagFind,TagEnd,TagAfter

    /*
        xmlString   The document you are parsing
        TagFind     The tag you wish to search for
                    - Preface this with #| to select the #th tag.
                      For instance, if you want the 5th <Review> content, it would be 5|<Review>
        TagEnd      The close tag for the data you are searching for
        TagAfter    The tag to begin searching after (ie: this can be a parent node, etc)
    */


    tSkip="N"

    If Pos("|",TagFind)>0 Then
    Do
        Parse Value TagFind with TagCount"|"TagNewFind
        TagFind=TagNewFind
        tSkip="Y"
    End

    if TagAfter="" Then
    Do
        tStart=1
    End
    Else
    Do
        tStart=Pos(TagAfter,xmlString)
        if tStart=0 Then tStart=1
    End

    If tSkip="Y" Then
    Do
        tPointer=tStart-1
        Do M=1 to TagCount
            tPointer=Pos(TagFind,xmlString,tPointer+1)


        End
        tStart=tPointer

        if tStart=0 then
        Do
            Return ""

        End
    End


    fStart=Pos(TagFind,xmlString,tStart)

    If Pos(TagFind,xmlString,tStart)=0 Then
    Do
        Return ""
    End

    fValue=Substr(xmlString,fStart+Length(TagFind))

    fValue=SubStr(fValue,1,Pos(TagEnd,fValue)-1)


Return fValue



Function returns nothing if no value found. So, for instance, if you search for <Review> and there is no <Review> tag, you get a nothing value returned (ie, it returns ""). The same applies for if you search for the 5th <Review> data and there are only 4 or less.

I believe there are function packages out there that drop this stuff into stems, but I refused to use such because OREXX is absolutely horrendous with dealing with stems (adds a LOT of processing time when handling, and there is a very bad memory leak in it's stem handling, and the large datasets I am retrieving would create a LOT of stems).

Use, modify, improve, ignore, or whatever, the code above.

Enjoy!
Rob


|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


RobertM

The XML Doc should be in one variable (ie: not in an array).


Value=FindXMLValue(XMLDocument,(Tag/Value you want),(Close Tag for the value you want),[Parent Node - or simply a previous tag - can be ommitted])

And an example or three:
SmallImageH=FindXMLValue(XMLDoc,'<Height Units="pixels">',"</Height>","<SmallImage>")

Find Height for SmallImage



Description=FindXMLValue(XMLDoc,"5|<Description>","</Description>","")

Finds 5th Description value, starting at beginning of document (returns nothing if no 5th value)



Rating=FindXMLValue(NextGet,"5|<Rating>","</Rating>","<CustomerReviews>")

Finds 5th <Rating> value after first <CustomerReviews> node.


Eventually, I will expand it's capabilities so that one can select nth parent node as well, but I didnt need that functionality yet and was in a rush.


|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


jep

Hi RobertM,

Very nice way to allow a tag counter.  :)

Is it common that there is no closing tag in XML documents? e.g. <sometag /> instead of <sometag></sometag>

I found out that unbalanced/nested tags can be somewhat tricky to handle, e.g. extract the value for each <sometag> and it'll cut of at the first </sometag> after the one selected, instead of the one that it belong to. One need balanced tags alot if one tries to extract data from HTML documents found on the net.
<maintag>
  <sometag>1
    <anothertag>2
      <sometag>3
        <sometag>4</sometag>
        <sometag>5</sometag>
      </sometag>
    </anothertag>
  </sometag>
</maintag>

The parent Node approach is interesting... I have to borrow that one too.  :)

Extra values on tags cause problems if they can differ the slightest, upper case comparison help somewhat.
Hmm, need to look for similarities.

Regards,
//Jan-Erik

RobertM

Quote from: jep on 2009.09.07, 17:55:57
Hi RobertM,

Very nice way to allow a tag counter.  :)

Is it common that there is no closing tag in XML documents? e.g. <sometag /> instead of <sometag></sometag>

I have found them frequently where it's a "value assignment"... such as:
    <Argument Name="ReviewPage" Value="1"/>

Though I didnt implicitly explain how to retrieve those, it's rather simple with my script:
The first tag would be:
<Argument Name="ReviewPage" Value="
The close tag would be:
"/>

Since it looks for the first occurrence of the close tag after finding the open tag, it works for such setups. It of course will not work if tags are improperly closed and/or unbalanced.


|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


RobertM

Quote from: jep on 2009.09.07, 17:55:57
Hi RobertM,



The parent Node approach is interesting... I have to borrow that one too.  :)

All yours. Eventually I'll flesh it out more.

Quote from: jep on 2009.09.07, 17:55:57
Extra values on tags cause problems if they can differ the slightest, upper case comparison help somewhat.
Hmm, need to look for similarities.

Regards,
//Jan-Erik

Considered it, but I think technically tags should be case sensitive. I am thinking of allowing an insensitive parsing method though to allow for user/programmer error or weird xml generators that mix up case.

Best,
-R


|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


RobertM

Quote from: jep on 2009.09.07, 17:55:57
<maintag>
  <sometag>1
    <anothertag>2
      <sometag>3
        <sometag>4</sometag>
        <sometag>5</sometag>
      </sometag>
    </anothertag>
  </sometag>
</maintag>

Fortunately, even this type of mess-up can be accounted for.

AnotherTag=FindXMLValue(XMLDoc,"<anothertag>","<sometag>","")
SomeTag1=FindXMLValue(XMLDoc,"2|<sometag>","<sometag>","")
SomeTag2=FindXMLValue(XMLDoc,"3|<sometag>","</sometag>","")
SomeTag3=FindXMLValue(XMLDoc,"4|<sometag>","</sometag>","")

(note the missing "/" in the first and second one - it's not really a close tag the function is asking for... it's the next tag or whatever comes next after the value).

So as long as whatever wrote the xml is consistent, there is often (well, at least sometimes) a way to still parse out the needed values.


-R


|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


jep

Marked as: Advanced
Hi,

here's my new and improved "do-it-all" xml parser function...

Please do test it to see if it behave well or not. I've tested it quite a lot myself, but would like to have it confirmed that it doesn't barf on something and possibly can cope with faulty xml-structures as well.

It doesn't contain a feature to traverse the structure as RobertM's version, but adding a function like that shouldn't be to difficult on top of it.

/* Test XML Parsing */


        XMLTAG = '<maintag color="blue">'
XMLTAG = XMLTAG||'  <sometag style="position: absolute">1'
XMLTAG = XMLTAG||'    <anothertag border="0">2'
XMLTAG = XMLTAG||'      <notag style="Postion: static" />'
XMLTAG = XMLTAG||'      <sometag alt="" href="http://www.test.com">3'
XMLTAG = XMLTAG||'        <sometag src="red.gif">4</sometag>'
XMLTAG = XMLTAG||'        <sometag style="border: solid">5</sometag>'
XMLTAG = XMLTAG||'        <sometag colspan="2" />'
XMLTAG = XMLTAG||'        <sometag rowspan="3" />'
XMLTAG = XMLTAG||'        <sometag>6</sometag>'
XMLTAG = XMLTAG||'        <sometag border="5">7</sometag>'
XMLTAG = XMLTAG||'      </sometag>'
XMLTAG = XMLTAG||'    </anothertag>'
XMLTAG = XMLTAG||'    <sometag style="position: absolute">8</sometag>'
XMLTAG = XMLTAG||'  </sometag>'
XMLTAG = XMLTAG||'  <notag />'
XMLTAG = XMLTAG||'</maintag>'

say '#'||parse_xml_tags( XMLTAG, 'notag',, 1 )||'#'                 /* */
say '#'||parse_xml_tags( XMLTAG, 'sometag', 'style=' )||'#'         /* 1<anothertag border="0">2<sometag alt="" href="http://www.test.com">3<sometag src="red.gif">4</sometag><sometag style="border: solid">5</sometag><sometag colspan="2" /><sometag>6</sometag><sometag border="5">7</sometag></sometag></anothertag><sometag style="position: absolute">8</sometag> */
say '#'||parse_xml_tags( XMLTAG, 'sometag', 'style=', 1 )||'#'      /* 1 */ /* (The first match) */
say '#'||parse_xml_tags( XMLTAG, 'sometag', 'style=', 2 )||'#'      /* "position: absolute" */
say '#'||parse_xml_tags( XMLTAG, 'sometag', 'src=' )||'#'           /* 4 */
say '#'||parse_xml_tags( XMLTAG, 'sometag', 'href=' )||'#'          /* 3<sometag src="red.gif">4</sometag><sometag style="border: solid">5</sometag><sometag colspan="2" /><sometag>6</sometag><sometag border="5">7</sometag> */
say '#'||parse_xml_tags( XMLTAG, 'sometag', 'colspan=', 1 )||'#'    /* */
say '#'||parse_xml_tags( XMLTAG, 'anothertag', 'colspan=', 2 )||'#' /* "2"  */

retval = parse_xml_tags( XMLTAG, 'sometag', 'alt=' )

say '#'||parse_xml_tags( retval, 'sometag', 'style', 1 )||'#'       /* 5 */

RETURN 0

parse_xml_tags: PROCEDURE /* xmlString, tag <<, tag_value>, selection> */
PARSE UPPER ARG xmlString, tag, val, selection
start_tag = '<'||tag
PARSE VALUE xmlString WITH pre(start_tag)(val)extra'>'post
SELECT
    WHEN selection = 1 THEN /* tag */
    DO
        IF LENGTH( extra ) > 0 THEN
        DO
            PARSE VALUE SUBSTR( ARG(1), ABS( LENGTH( xmlString ) - LENGTH( post ) + 1 ) ) WITH pre'<'.
            RETURN pre
        END
    END
    WHEN selection = 2 THEN /* tag value */
       RETURN SUBSTR( ARG(1), ABS( LENGTH( xmlString ) - LENGTH( post ) - LENGTH( extra ) ), LENGTH( extra ) - ( RIGHT( extra, 1 ) = '/' ) )
    WHEN RIGHT( extra, 1 ) <> '/' THEN
    DO
        end_tag = '</'||tag||'>'
        next = LENGTH( xmlString ) - LENGTH( post ) + 1
        open_tag = next
        PARSE VALUE post WITH pre(end_tag)post
        close_tag = LENGTH( xmlString ) - LENGTH( post ) - LENGTH( end_tag )
        count = COUNTSTR( start_tag, xmlString, open_tag, close_tag )
        DO WHILE count > 0
            next = close_tag + LENGTH( end_tag )
            close_tag = POS( end_tag, xmlString, next )
            IF close_tag = 0 THEN
                close_tag = LENGTH( xmlString )
            post = SUBSTR( xmlString, next, close_tag - next )

            PARSE VALUE post WITH .(start_tag)extra'/>'post

            DO WHILE LENGTH( post ) > 0
                next = close_tag - LENGTH( post )
                PARSE VALUE post WITH .(start_tag)extra'/>'post
            END
            count = count + COUNTSTR( start_tag, xmlString, next, close_tag ) - 1
        END
        IF close_tag = 0 THEN close_tag = LENGTH( ARG(1) ) - open_tag
        ELSE close_tag = close_tag - open_tag
        RETURN SUBSTR( ARG(1), open_tag, MAX( 1, close_tag ) )
    END
OTHERWISE NOP
END
RETURN ''

COUNTSTR: PROCEDURE /* needle, haystack< <, startpos>, endpos> */
    IF ARG() < 2 THEN RETURN -1
    IF DATATYPE( ARG(3), 'W' ) THEN
        next = ARG(3)
    ELSE
        next = 1
    needle = ARG(1)
    haystack = ARG(2)
    IF DATATYPE( ARG(4), 'W' ) THEN
        haystack = SUBSTR( haystack, next, ABS( ARG(4) - next ) )
    next = 1
    count = 0
    DO WHILE next > 0
        next = POS( needle, haystack, next )
        IF next > 0 THEN DO
            next = next + LENGTH( needle )
            count = count + 1
        END
    END
RETURN count


//Jan-Erik