WebMaven TechSpecs: Difference between revisions
No edit summary |
mNo edit summary |
||
Line 5: | Line 5: | ||
!Term | !Term | ||
!Definition | !Definition | ||
|- | |- | ||
!URI | !URI | ||
|WebMaven uses the term URI to encompass URLs along with other associated data necessary to reference a file via the HTTP protocol. From RFC 2068: | |WebMaven uses the term URI to encompass URLs along with other associated data necessary to reference a file via the HTTP protocol. From RFC 2068: | ||
URIs have been known by many names: WWW addresses, Universal Document Identifiers, Universal Resource Identifiers , and finally the combination of Uniform Resource Locators (URL) and Names (URN). As far as HTTP is concerned, Uniform Resource Identifiers are simply formatted strings which identify -- via name, location, or any other characteristic -- a resource. | |||
|- | |- | ||
!Local path | !Local path | ||
Line 21: | Line 15: | ||
'''Note:''' [http://www.toward.com/cfsrexx/WebMaven/techspecs.htm#RemotePathsPerLocalTree Multiple remote paths] can only be specified with a the enterprise edition. | '''Note:''' [http://www.toward.com/cfsrexx/WebMaven/techspecs.htm#RemotePathsPerLocalTree Multiple remote paths] can only be specified with a the enterprise edition. | ||
|- | |- | ||
!Remote path | !Remote path | ||
Line 63: | Line 54: | ||
</nowiki></pre> | </nowiki></pre> | ||
Enterprise enabled versions of WebMaven could specify remote path values of ''Baseball/American/Red Sox'' | Enterprise enabled versions of WebMaven could specify remote path values of ''Baseball/American/Red Sox'' and ''Baseball/National/Dodgers.'' and a local path of ''MLB'' which would result in the following local tree structure on the local hard drive: | ||
<pre><nowiki> | <pre><nowiki> | ||
MLB | MLB | ||
Line 85: | Line 76: | ||
'''Note:''' [http://www.toward.com/cfsrexx/WebMaven/techspecs.htm#RemotePathsPerLocalTree Multiple remote paths] can only be specified with the enterprise edition. | '''Note:''' [http://www.toward.com/cfsrexx/WebMaven/techspecs.htm#RemotePathsPerLocalTree Multiple remote paths] can only be specified with the enterprise edition. | ||
|- | |- | ||
| nowrap="nowrap" | '''Out of tree''' | | nowrap="nowrap" | '''Out of tree''' | ||
| Out of tree designates HTML links that are not within the remote path(s). What happens with out of tree links is detailed in the [http://www.toward.com/cfsrexx/WebMaven/techspecs.htm#ProtocolProcessingTable Protocol processing] table. | | Out of tree designates HTML links that are not within the remote path(s). What happens with out of tree links is detailed in the [http://www.toward.com/cfsrexx/WebMaven/techspecs.htm#ProtocolProcessingTable Protocol processing] table. | ||
|} | |} | ||
Line 111: | Line 96: | ||
| Maximum number of download subtasks | | Maximum number of download subtasks | ||
| align="center" | 1 | | align="center" | 1 | ||
| align="center" | 1-2; | | align="center" | 1-2; <u>2</u> | ||
| align="center" | 1-9; | | align="center" | 1-9; <u>3</u> | ||
|- valign="middle" | |- valign="middle" | ||
| Maximum download byte limit | | Maximum download byte limit | ||
Line 122: | Line 107: | ||
| align="center" | full only | | align="center" | full only | ||
| align="center" | relative only | | align="center" | relative only | ||
| align="center" | <u>relative</u> | | align="center" | <u>relative</u>|full | ||
|- valign="middle" | |- valign="middle" | ||
| HTTP client version | | HTTP client version | ||
| align="center" | 1.0 only | | align="center" | 1.0 only | ||
| align="center" | <u>1.1</u> | | align="center" | <u>1.1</u>|1.0 | ||
| align="center" | <u>1.1</u> | | align="center" | <u>1.1</u>|1.0 | ||
|- valign="middle" | |- valign="middle" | ||
| HTML priority load balancing | | HTML priority load balancing | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
|- valign="middle" | |- valign="middle" | ||
| Case sensitive URIs | | Case sensitive URIs | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
|- valign="middle" | |- valign="middle" | ||
| Useable with proxy server | | Useable with proxy server | ||
Line 164: | Line 149: | ||
| align="center" | No | | align="center" | No | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
|- valign="middle" | |- valign="middle" | ||
| [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportCookie.htm Cookie Report]<br /><font size="-2">(does not affect handling cookie requests) | | [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportCookie.htm Cookie Report]<br /><font size="-2">(does not affect handling cookie requests) | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
|- valign="middle" | |- valign="middle" | ||
| [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportDNS.htm Domain Name Server Lookup Time] | | [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportDNS.htm Domain Name Server Lookup Time] | ||
| align="center" | No | | align="center" | No | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
|- valign="middle" | |- valign="middle" | ||
| [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportDomainException.htm Unresolved Domain Names] <sup>1</sup> | | [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportDomainException.htm Unresolved Domain Names] <sup>1</sup> | ||
Line 184: | Line 169: | ||
| align="center" | No | | align="center" | No | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
|- valign="middle" | |- valign="middle" | ||
| [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportMailto.htm E-mail Addresses] | | [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportMailto.htm E-mail Addresses] | ||
| align="center" | No | | align="center" | No | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
|- valign="middle" | |- valign="middle" | ||
| [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportMimeType.htm Externally Processed MIME Types] | | [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportMimeType.htm Externally Processed MIME Types] | ||
| align="center" | No | | align="center" | No | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
|- valign="middle" | |- valign="middle" | ||
| [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportSyntaxErrors.htm HTML Syntax Errors] <sup>1</sup> | | [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportSyntaxErrors.htm HTML Syntax Errors] <sup>1</sup> | ||
Line 208: | Line 193: | ||
| [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportImageException.htm Image Exception Report] | | [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportImageException.htm Image Exception Report] | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
|- valign="middle" | |- valign="middle" | ||
| [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportImageGroup.htm Image Group Report] | | [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportImageGroup.htm Image Group Report] | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
|- valign="middle" | |- valign="middle" | ||
| [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportJavaClass.htm Java Class Report]<br />(does not affect processing Java classes) | | [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportJavaClass.htm Java Class Report]<br />(does not affect processing Java classes) | ||
| align="center" | No | | align="center" | No | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
|- valign="middle" | |- valign="middle" | ||
| [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportOutOfTree.htm Out Of Tree Links] | | [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportOutOfTree.htm Out Of Tree Links] | ||
| align="center" | No | | align="center" | No | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>No</u> | | align="center" | <u>No</u>|Yes | ||
|- valign="middle" | |- valign="middle" | ||
| [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportRelocation.htm Relocated URIs] | | [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportRelocation.htm Relocated URIs] | ||
| align="center" | No | | align="center" | No | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>No</u> | | align="center" | <u>No</u>|Yes | ||
|- valign="middle" | |- valign="middle" | ||
| [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportSitemap.htm Site Map] | | [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportSitemap.htm Site Map] | ||
| align="center" | No | | align="center" | No | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>No</u> | | align="center" | <u>No</u>|Yes | ||
|- valign="middle" | |- valign="middle" | ||
| [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportSummary.htm Summary of WebMaven reports] | | [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportSummary.htm Summary of WebMaven reports] | ||
Line 244: | Line 229: | ||
| align="center" | No | | align="center" | No | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>Yes</u> | | align="center" | <u>Yes</u>|No | ||
|- valign="middle" | |- valign="middle" | ||
| [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportXref.htm File Cross Reference Table] | | [http://www.toward.com/cfsrexx/WebMaven/SampleReports/ReportXref.htm File Cross Reference Table] | ||
| align="center" | No | | align="center" | No | ||
| align="center" | No | | align="center" | No | ||
| align="center" | <u>No</u> | | align="center" | <u>No</u>|Yes | ||
|- valign="middle" | |- valign="middle" | ||
| colspan="4" | | | colspan="4" | | ||
;Note 1:The user must confirm sending the e-mail report to the Webmaster. | |||
The user must confirm sending the e-mail report to the Webmaster. | |||
|} | |} | ||
Line 259: | Line 243: | ||
|+Protocol processing: | |+Protocol processing: | ||
|- valign="bottom" | |- valign="bottom" | ||
!align="left" |Protocol | |||
!align="left" |WebMaven action | |||
|- | |- | ||
| HTTP; FTP | | HTTP; FTP | ||
Line 277: | Line 261: | ||
| Retrieves the referenced URI. | | Retrieves the referenced URI. | ||
|} | |} | ||
{|class="wikitable" | {|class="wikitable" | ||
Line 290: | Line 273: | ||
| 400+ - Not found<br /> 500+ - Server error | | 400+ - Not found<br /> 500+ - Server error | ||
| | | | ||
HTML expected: | HTML expected: | ||
Image expected: | Image expected: | ||
[ | [Example of bad image gif] | ||
|} | |} | ||
{|class="wikitable" | {|class="wikitable" | ||
Line 313: | Line 295: | ||
|- | |- | ||
| align="center" | WebMaven.LOG | | align="center" | WebMaven.LOG | ||
| | |A WebMaven.LOG file is created / extended in the local path directory each time WebMaven is run. The contents of the .LOG file vary depending upon user-selected options. | ||
|- | |- | ||
| align="center" | WebMaven.!!! | | align="center" | WebMaven.!!! | ||
| This sentinel file exists in the local path while WebMaven is running. | |This sentinel file exists in the local path while WebMaven is running. | ||
|} | |} | ||
{|class="wikitable" | {|class="wikitable" | ||
|+HTML Tag Processing | |+HTML Tag Processing | ||
!HTML Tag | !HTML Tag | ||
!Attributes processed | !Attributes processed | ||
|- | |- | ||
| <A> | |<A> | ||
| HREF | | HREF | ||
|- | |- | ||
| <APPLET> | |<APPLET> | ||
| ARCHIVE; CODE; CODEBASE; PARAM | | ARCHIVE; CODE; CODEBASE; PARAM | ||
|- | |- | ||
| <AREA> | |<AREA> | ||
| HREF | | HREF | ||
|- | |- | ||
| <BASE> | |<BASE> | ||
| HREF | | HREF | ||
|- | |- | ||
| <BGSOUND> | |<BGSOUND> | ||
| SRC | | SRC | ||
|- | |- | ||
| <BLOCKQUOTE | |<BLOCKQUOTE> | ||
| CITE | | CITE | ||
|- | |- | ||
| <BODY> | |<BODY> | ||
| BACKGROUND; STYLESRC | | BACKGROUND; STYLESRC | ||
|- | |- | ||
| <DEL | |<DEL> | ||
| CITE | | CITE | ||
|- | |- | ||
| <EMBED> | |<EMBED> | ||
| SRC | | SRC | ||
|- | |- | ||
| <FORM> | |<FORM> | ||
| ACTION; INPUT | | ACTION; INPUT | ||
|- | |- | ||
| <FRAME> | |<FRAME> | ||
| LONGDESC; SRC | | LONGDESC; SRC | ||
|- | |- | ||
| <HEAD> | |<HEAD> | ||
| PROFILE | | PROFILE | ||
|- | |- | ||
| <IFRAME> | |<IFRAME> | ||
| LONGDESC; SRC | | LONGDESC; SRC | ||
|- | |- | ||
| <IMG> | |<IMG> | ||
| LONGDESC; LOWSRC; SRC; USEMAP | | LONGDESC; LOWSRC; SRC; USEMAP | ||
|- | |- | ||
| <INPUT> | |<INPUT> | ||
| SRC; USEMAP | | SRC; USEMAP | ||
|- | |- | ||
| <INS | |<INS> | ||
| CITE | | CITE | ||
|- | |- | ||
| <INPUT> | |<INPUT> | ||
| SRC; USEMAP | | SRC; USEMAP | ||
|- | |- | ||
| <LINK> | |<LINK> | ||
| HREF | | HREF | ||
|- | |- | ||
| <META> | |<META> | ||
| HTTP-EQUIV | | HTTP-EQUIV | ||
|- | |- | ||
| <OBJECT> | |<OBJECT> | ||
| ARCHIVE; CLASSID; CODEBASE; DATA; PARAM; USEMAP | | ARCHIVE; CLASSID; CODEBASE; DATA; PARAM; USEMAP | ||
|- | |- | ||
| <OPTGROUP> | |<OPTGROUP> | ||
| VALUE | | VALUE | ||
|- | |- | ||
| <OPTION> | |<OPTION> | ||
| VALUE | | VALUE | ||
|- | |- | ||
| <PARAM> | |<PARAM> | ||
| VALUE | | VALUE | ||
|- | |- | ||
| <Q> | |<Q> | ||
| CITE | | CITE | ||
|- | |- | ||
| <SCRIPT> | |<SCRIPT> | ||
| SRC | | SRC | ||
|} | |} | ||
[[Category:Software]] | [[Category:Software]] |
Revision as of 20:17, 24 October 2017
- Definition of terms
- In order to fully understand the material below, it is necessary to specify some of the jargon used with WebMaven.
Term | Definition |
---|---|
URI | WebMaven uses the term URI to encompass URLs along with other associated data necessary to reference a file via the HTTP protocol. From RFC 2068:
URIs have been known by many names: WWW addresses, Universal Document Identifiers, Universal Resource Identifiers , and finally the combination of Uniform Resource Locators (URL) and Names (URN). As far as HTTP is concerned, Uniform Resource Identifiers are simply formatted strings which identify -- via name, location, or any other characteristic -- a resource. |
Local path | A remote site, be it a complete domain (e.g. www.cfsrexx.com) or one or more paths within a remote domain (e.g. www.cfsrexx.com/pub/ & www.cfsrexx.com/WebMaven/) are downloaded into the local path directory.
Note: Multiple remote paths can only be specified with a the enterprise edition. |
Remote path | Remote path is a domain name, with or without a path. The remote path corresponds to the local path on a one-to-one basis when only one remote path is specified.
When more than one remote path is specified, each directory in the remote path corresponds, one-for-one, with a local directory subordinate to the local path. Multiple remote paths can include the same or different domains and protocols. For example, assuming the following remote tree structure: [http://www.majorleaguebaseball.com/ Baseball] | *-- [http://www.majorleaguebaseball.com/al/ American] | | | *-- [http://www.majorleaguebaseball.com/al/bos/index.sml Red Sox] | | | | | *-- [http://www.pawsox.com/ Pawtucket] | | | | | *-- [http://www.trentonthunder.com/ Trenton] | | | *-- [http://www.majorleaguebaseball.com/al/nyy/index.sml Yankees] | | | *-- [http://www.clippersbaseball.com/ Columbus] | | | *-- [http://www.gators.com/ Norwich] | *-- [http://www.majorleaguebaseball.com/nl/ National] | *-- [http://www.majorleaguebaseball.com/nl/ny/index.sml%22 Mets] | | | *-- [http://www.piloton-line.com/tides/index.shtml Norfolk] | | | *-- [http://www.spectra.net/mall/bmets/ Binghampton] | *-- [http://www.majorleaguebaseball.com/nl/la/index.sml Dodgers] | *-- [http://www.nmia.com/~roberts/dukes.html Albuquerque] | *-- [http://www.samissions.com/ San Antonio] Enterprise enabled versions of WebMaven could specify remote path values of Baseball/American/Red Sox and Baseball/National/Dodgers. and a local path of MLB which would result in the following local tree structure on the local hard drive: MLB | *-- American | | | *-- Red Sox | | | *-- Pawtucket | | | *-- Trenton | *-- National | *-- Dodgers | *-- Albuquerque | *-- San Antonio Note: Multiple remote paths can only be specified with the enterprise edition. |
Out of tree | Out of tree designates HTML links that are not within the remote path(s). What happens with out of tree links is detailed in the Protocol processing table. |
Function | Unlicensed | Personal license | Enterprise license |
---|---|---|---|
Default file name | INDEX!.HTM | INDEX!.HTM | user specified | INDEX!.HTM | user specified |
Maximum number of download subtasks | 1 | 1-2; 2 | 1-9; 3 |
Maximum download byte limit | Yes | Yes | Yes |
Full vs. relative localized paths | full only | relative only | relative|full |
HTTP client version | 1.0 only | 1.1|1.0 | 1.1|1.0 |
HTML priority load balancing | No | Yes|No | Yes|No |
Case sensitive URIs | No | Yes|No | Yes|No |
Useable with proxy server | No | Yes | Yes |
Remote paths per local tree | 1 | 1 | unlimited |
User ID / Password capable | No | Yes | Yes |
Sample Reports (Sample reports) | |||
Report Name | Unlicensed | Personal license | Enterprise license |
Aged File List | No | No | Yes|No |
Cookie Report (does not affect handling cookie requests) |
No | Yes|No | Yes|No |
Domain Name Server Lookup Time | No | No | Yes|No |
Unresolved Domain Names 1 | Yes | Yes | Yes |
Download Time Report | No | No | Yes|No |
E-mail Addresses | No | No | Yes|No |
Externally Processed MIME Types | No | No | Yes|No |
HTML Syntax Errors 1 | Yes | Yes | Yes |
Invalid HTML links 1 | Yes | Yes | Yes |
Image Exception Report | No | Yes|No | Yes|No |
Image Group Report | No | Yes|No | Yes|No |
Java Class Report (does not affect processing Java classes) |
No | No | Yes|No |
Out Of Tree Links | No | No | No|Yes |
Relocated URIs | No | No | No|Yes |
Site Map | No | No | No|Yes |
Summary of WebMaven reports | Yes | Yes | Yes |
Titles of HTML Pages & Images | No | No | Yes|No |
File Cross Reference Table | No | No | No|Yes |
|
Protocol | WebMaven action |
---|---|
HTTP; FTP |
Content-type=text/html: Content-type=(all others) |
MAILTO | Left unchanged (this will permit deferred delivery of mail messages). |
FILE; GOPHER; HTTPS; NEWS; NNTP; TELNET; WAIS |
Creates an out of tree page. |
PNM (Real Audio) | Retrieves the referenced URI. |
HTTP Response | WebMaven action |
300+ - Redirection | Process redirected URI |
400+ - Not found 500+ - Server error |
HTML expected: Image expected: [Example of bad image gif] |
File name | Function |
---|---|
WebMaven.CHK | This is WebMaven's checkpoint / restart file. It is created in the local path if WebMaven terminates without finishing the retrieval of all files in the remote tree(s) |
WebMave?.DMP | In the event that WebMaven terminates abnormally (abends), a .DMP file is created in the WebMaven program directory. DMP files must be ZIP'ed and sent to Program Product Support when reporting the problem.
ZIP'ed .DMP files, along with other relevant material, can be e-mailed to product support or they can be anonymously uploaded to ftp.cfsrexx.com/incoming/. Be sure to use a file name that is unique to you so that it will not conflict with any files uploaded by others. |
WebMaven.LOG | A WebMaven.LOG file is created / extended in the local path directory each time WebMaven is run. The contents of the .LOG file vary depending upon user-selected options. |
WebMaven.!!! | This sentinel file exists in the local path while WebMaven is running. |
HTML Tag | Attributes processed |
---|---|
<A> | HREF |
<APPLET> | ARCHIVE; CODE; CODEBASE; PARAM |
<AREA> | HREF |
<BASE> | HREF |
<BGSOUND> | SRC |
<BLOCKQUOTE> | CITE |
<BODY> | BACKGROUND; STYLESRC |
<DEL> | CITE |
<EMBED> | SRC |
<FORM> | ACTION; INPUT |
<FRAME> | LONGDESC; SRC |
<HEAD> | PROFILE |
<IFRAME> | LONGDESC; SRC |
<IMG> | LONGDESC; LOWSRC; SRC; USEMAP |
<INPUT> | SRC; USEMAP |
<INS> | CITE |
<INPUT> | SRC; USEMAP |
<LINK> | HREF |
<META> | HTTP-EQUIV |
<OBJECT> | ARCHIVE; CLASSID; CODEBASE; DATA; PARAM; USEMAP |
<OPTGROUP> | VALUE |
<OPTION> | VALUE |
<PARAM> | VALUE |
CITE | |
<SCRIPT> | SRC |