Author Topic: SM with .SUBJECT EA  (Read 354 times)

André Heldoorn

  • Hero Member
  • *****
  • Thank You
  • -Given: 63
  • -Receive: 17
  • Posts: 571
    • View Profile
SM with .SUBJECT EA
« on: July 09, 2019, 07:48:23 pm »
What's the last, still downloadable version of SM which still saved the location of a download in the .SUBJECT EA?

The <CTRL-J> donwloads windows tends to cause a TRP file, and I need a full name.

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 10
  • -Receive: 139
  • Posts: 1990
    • View Profile
Re: SM with .SUBJECT EA
« Reply #1 on: July 10, 2019, 02:44:04 am »
https://github.com/bitwiseworks/mozilla-os2/issues/133 says Firefox 24, which would correspond to Sm 2.21.
I couldn't figure out how to re-enable it and Bitwise didn't seem interested.

The CTRL-J may be fixed by moving downloads.sqlite out of the way.
« Last Edit: July 10, 2019, 02:46:35 am by Dave Yeo »

André Heldoorn

  • Hero Member
  • *****
  • Thank You
  • -Given: 63
  • -Receive: 17
  • Posts: 571
    • View Profile
Re: SM with .SUBJECT EA
« Reply #2 on: July 10, 2019, 11:42:20 am »
I couldn't figure out how to re-enable it and Bitwise didn't seem interested.
As such it's wasn't a RFC albeit it's at least nice-to-have, but I hope a website still works with SM 2.21. I may need a mix of several old and new browsers OS/2 and/or Windows browsers anyway.

A website was changed (the usual "looks great, no more Flash required, speed and functionalities down the drain"), so I'm hoping I can find all of the download-related forms and form settings again for several WGET-based downloads. If possible. A genuine disaster. IIRC using the .SUBJECT EA data was just one of the steps.

Thanks.

Heikki Kekki

  • Newbie
  • *
  • Thank You
  • -Given: 0
  • -Receive: 0
  • Posts: 2
    • View Profile
Re: SM with .SUBJECT EA
« Reply #3 on: July 10, 2019, 07:33:37 pm »
PMDownloader v1.0 saves download url in Comments: field.

Seamonkey 2.35 saves image file download url in Subject: field, when you use MB2 menu and 'Save Image As...'

Also you can put download url manually to Subject: field.

André Heldoorn

  • Hero Member
  • *****
  • Thank You
  • -Given: 63
  • -Receive: 17
  • Posts: 571
    • View Profile
Re: SM with .SUBJECT EA
« Reply #4 on: July 10, 2019, 09:32:39 pm »
Probably too simple.

There seems to be one basic direct download, which is XLSX and fails with WGET. Error 500, IIRC. With a browser a less basic version now shows "Access restricted", or something like that.

All of the other ones are/were forms. Then you need a links to and data of that form, to set choices like Excel or CSV, the date format, and so on, and to virtually press the Download-button.

I'm not even sure if downloading automatically is still possible. Not only the URLs have changed, but apaprently data too. The date format was a bleedin' Unix millisecond time stamp, but I've also seen "2019-07-10".

Sample: https://live.euronext.com/en/products/equities/list (arrow pointing down, to the right of Help). This webpage is broken, but it should show how a form works.

About the same: https://live.euronext.com/en/products/indices/list

Or https://live.euronext.com/en/product/indices/FR0003500008-XPAR (section Historical prices, press More details, and next the download button to the left of the X to close this window.

Or https://live.euronext.com/en/product/equities/BE0974293251-XBRU (parsing data, open-high-low-last-volume, or use that Download-button which is new to me).

Or https://live.euronext.com/en/product/equities/BE0974293251-XBRU (section Central order book, download).

In the case of (the old) forms you'ld "download" the form, and post data to that form. You could open the form in a browser by using the URL of the "download"", and then could manually select the output format, and so on.

Old Rexx sample code, direct download of historical prices:

Code: [Select]
from=(Date('B',date1,'S')-719162)*86400000
to=(Date('B',date2,'S')-719162)*86400000
browser='--user-agent="Mozilla/5.0 (OS/2; U; OS/2 4.50; nl-NL; rv:14.093)"'

url='"https://www.euronext.com/nyx_eu_listings/price_chart/download_'
url=url||'historical?typefile=csv&layout=vertical&typedate=dmy&separator=com'
url=url||'ma&mic='||market||'&'||'isin='||isin||'&namefile=Price_Data_Hist'
url=url||'orical&from='||from||'&to='||to||'&adjusted=1&base=0"'

'@START "'||symbol||'" /C /B /WIN /MIN E:\TCPIP\BIN\WGET.EXE -q --no-check-certificate --timeout=10 --tries=5 --waitretry=10 -O'||symbol browser url

Old form sample (equity list):

Code: [Select]
PARSE ARG date

file=date||'.001'
browser='--user-agent="Mozilla/5.0 (OS/2; U; OS/2 4.50; nl-NL; rv:14.093)"'

postdata='--post-data "format=2&layout=2&decimal_separator=1&date_format=1&op=Go&form_build_id=form-46e2fc2b0019c90a55af0703490c9ec0&form_id=nyx_download_form"'

url='"https://www.euronext.com/nl/popup/data/download?ml=nyx_pd_stocks&cmd=default&formKey=nyx_pd_filter_values%3A6eb6d791dc71fa1308eb5a048efdcc08"'

'@E:\TCPIP\BIN\WGET.EXE -q --no-check-certificate -O'||file browser postdata url

E.g. layout=2 was probably CSV (second button of the layout group), and e.g. op=Go probaly pushed the Download-button of the form virtually.

IIRC I had to use several old OS/2 and Windows browsers to obtain (most of) the data required. You may still be able to download the new forms with a basic downloader, but you also have to be able to fill in the form to download the real file you're after. About 2500-3750 downloads per day in Europe, so you really need a powerul downloader with extras.

At the moment I don't even know if the old download HTML forms techniques still are possible, even if I would have all of the data (and certifcates, and ...) I need. IIRC the latest SM by Dave Yeo shows a lot of the form data without having to study source code.

André Heldoorn

  • Hero Member
  • *****
  • Thank You
  • -Given: 63
  • -Receive: 17
  • Posts: 571
    • View Profile
Re: SM with .SUBJECT EA
« Reply #5 on: July 11, 2019, 12:19:20 pm »
Firefox 24, which would correspond to Sm 2.21.
Which turns out to be probably too old for the "modernized" website's main page already, FWIW. A SSL cypher-related error while using SM 2.42.9ESR SUa2-compliant generic DLLs and 2 required old DLLs (FNTCFG2.DLL and FREETYPE2.DLL).

<Ctrl-J> started working again, without intentionally touching files. Basicly the window pops up, XOR it keeps TRP'ing until it somehow starts working again (a reboot or an added download doesn't (always) help). I do delete the list frequently, so let's say that a little bit of luck is required.

Presumably the full URL of a download is the full same URL as the one formerly saved as the .SUBJECT EA?