OS2 World Community Forum

OS/2, eCS & ArcaOS - Technical => Programming => Topic started by: Paul Smedley on June 27, 2020, 11:55:26 am

Title: Python 3.x?
Post by: Paul Smedley on June 27, 2020, 11:55:26 am
Hi All,

Wondering how much interest there might be in a Python 3.x port? OS/2 support was removed from the source code around Python 3.3 - so it will be a reasonable amount of work to re-instate it. However, Samba keeps marching towards Python 3.x, and I fear that soon our 2.7.x build won't be good enough to keep Samba current....

Cheers,

Paul
Title: Re: Python 3.x?
Post by: Martin Iturbide on June 27, 2020, 01:13:43 pm
Hi Paul

I can not talk from the developers point of view, I can only talk from what I read and saw around. According to some articles I had read since 2018, python popularity was growing and it is the most used language for machine learning today. Yes, it is hot and the "cool people" are using it.

From my perspective I had found some .py scripts that I want to use, but I don't realize the full potential of it yet. I guess we should try to find out what cool things we can potentially have in OS/2 if python 3.x gets ported.

Comments are welcome.

Regards

Title: Re: Python 3.x?
Post by: Martin Iturbide on June 27, 2020, 01:18:32 pm

(https://miro.medium.com/max/1400/0*agN3x-Al2UXuuZgx.)

Source: 2018 https://medium.com/@UdacityINDIA/why-use-python-for-machine-learning-e4b0b4457a77 (https://medium.com/@UdacityINDIA/why-use-python-for-machine-learning-e4b0b4457a77)
Title: Re: Python 3.x?
Post by: David McKenna on June 27, 2020, 03:15:55 pm
Paul,

  While I don't use Python directly, it seems more and more things need Python 3 and I would definitely like to see porting to OS/2 continue, so it would be great if you could do so...

Regards,
Title: Re: Python 3.x?
Post by: Dariusz Piatkowski on June 28, 2020, 12:06:27 am
Hi Paul,

Indeed, Python has a tremendous amount of functionality these days, the ML stuff in particular being part of my interest.

I would love to see 3.x stuff on our platform, mostly b/c majority of ML libraries (the stuff being actively talked about, etc.) is at that level.

Thanks!
Title: Re: Python 3.x?
Post by: Paul Smedley on June 28, 2020, 12:58:25 am
OK I'll start exploring what's required. bear in mind there is zero commitment to producing something that works. There's every chance something else shiny will pop up and distract me from this :)
Title: Re: Python 3.x?
Post by: Paul Smedley on June 28, 2020, 11:30:52 pm
Brief status update:
I've started trying to compile using Andy Mac's build files that were formerly located in pc\os2emx - mainly as I don't have the energy right now to battle hacking configure script - and porting Python.

I nearly have a python38.dll - just missing a few symbols that I'm working through. I then need to make it work :) I don't intend on applying *all* of the old patches as there is a lot of cruft there from vacpp and emx that we don't need anymore - the aim is to apply only the minimal patches required to make it work.

I'll try and update status here periodically...

Cheers,

Paul
Title: Re: Python 3.x?
Post by: Paul Smedley on June 28, 2020, 11:53:19 pm
python38.dll built; now to get an executable to call it - then start debugging - probably starting with path stuff - then to get modules building...
Title: Re: Python 3.x?
Post by: Paul Smedley on June 29, 2020, 08:26:25 am
python38.dll built; now to get an executable to call it - then start debugging - probably starting with path stuff - then to get modules building...

Got a python.exe and the associated modules - of course - it doesn't work yet - I need to start applying patches - starting with path separator stuff....
Title: Re: Python 3.x?
Post by: Paul Smedley on June 30, 2020, 12:00:14 pm
Slowly making progress....

Python path configuration:
  PYTHONHOME = '/dev/python-3.8.3'
  PYTHONPATH = (not set)
  program name = 'U:\dev\python-3.8.3\python.exe'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = 'U:\\dev\\python-3.8.3\\python.exe'
  sys.base_prefix = '/dev/python-3.8.3'
  sys.base_exec_prefix = '/dev/python-3.8.3'
  sys.executable = 'U:\\dev\\python-3.8.3\\python.exe'
  sys.prefix = '/dev/python-3.8.3'
  sys.exec_prefix = '/dev/python-3.8.3'
  sys.path = [
    '/dev/python-3.8.3/lib/python38.zip',
    '/dev/python-3.8.3/lib/python3.8/./Lib',
    '/dev/python-3.8.3/lib/python3.8/./Lib/plat-os2emx',
    '/dev/python-3.8.3/lib/python3.8/./Lib/lib-dynload',
    '/dev/python-3.8.3/lib/python3.8/./Lib/site-packages',
    '/dev/python-3.8.3/lib/python3.8/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x20036680 (most recent call first):
<no Python frame>
Title: Re: Python 3.x?
Post by: Paul Smedley on July 01, 2020, 01:34:21 am
Far from being tested.... but....

{0}[u:\dev\python-3.8.3] python.exe PC\os2emx\script.py
Goodbye, World!

{0}[u:\dev\python-3.8.3] cat PC\os2emx\script.py
print("Goodbye, World!")
Title: Re: Python 3.x?
Post by: Dariusz Piatkowski on July 01, 2020, 05:30:12 am
Far from being tested.... but....

{0}[u:\dev\python-3.8.3] python.exe PC\os2emx\script.py
Goodbye, World!

{0}[u:\dev\python-3.8.3] cat PC\os2emx\script.py
print("Goodbye, World!")

NICE!!!

Thank you for the rapid updates...I for one am following them closely Paul!
Title: Re: Python 3.x?
Post by: Valery Sedletski on July 01, 2020, 05:48:50 am
Paul, good luck with porting Python 3! I also would like to have Python 3 on OS/2, because I have some things I'd like
to have, like Electrum Bitcoin wallet, which requires a latest and greatest Python to work ok. I use it on Linux and it is good
to have on OS/2 too. Also, Web Automation tools, like Selenium, use Python 3 too. And many more programs written in
Python 3

PS: Electrum uses  PyQt for user interface, which is quite big. Though, we seem to already have PyQt4 for Python 2.x.
Title: Re: Python 3.x?
Post by: Paul Smedley on July 01, 2020, 11:15:25 am
Hi all - making progress - running Samba 4.12.3 configure with it now..... I still have a fair amount of work to do to incorporate patches.....
Title: Re: Python 3.x?
Post by: Paul Smedley on July 01, 2020, 11:18:55 am
Hi Valery,

Paul, good luck with porting Python 3! I also would like to have Python 3 on OS/2, because I have some things I'd like
to have, like Electrum Bitcoin wallet, which requires a latest and greatest Python to work ok. I use it on Linux and it is good
to have on OS/2 too. Also, Web Automation tools, like Selenium, use Python 3 too. And many more programs written in
Python 3

PS: Electrum uses  PyQt for user interface, which is quite big. Though, we seem to already have PyQt4 for Python 2.x.

I'd suggest that unless I discover some issues in the next day or two, I'll have some sort of binary package for testing over the weekend.

I don't have any real plans to build PyQt - but you never know - my real need for this is solely due to Samba....

Cheers,

Paul
Title: Re: Python 3.x?
Post by: Paul Smedley on July 01, 2020, 11:26:53 pm
Hi all - making progress - running Samba 4.12.3 configure with it now..... I still have a fair amount of work to do to incorporate patches.....

Update - Samba configure completed successfully. Trying to build Samba with Python 3.8.3 now...
Title: Re: Python 3.x?
Post by: Paul Smedley on July 02, 2020, 07:00:28 am
It's working.... getting some weird errors where:
python3 ./buildtools/bin/waf build gives errors like:

[1468/2428] Compiling bin/default/source3/modules/vfs.empty.c
gcc.exe: fatal error: cannot execute 'U:/usr/local920/bin/../libexec/gcc/i686-pc-os2-emx/9.2.0/cc1.exe': spawnv: Invalid argument
compilation terminated.

Waf: Leaving directory `U:/dev/samba-4.11.10/bin/default'
Build failed
 -> task in 'vfs' failed with exit status 1 (run with -v to display more information)

but running:
python ./buildtools/bin/waf --targets=vfs

results in a successful compile

I also get complaints about failure to convert a.out to OMF from emxomfld which similarly are fine if I build the specific target (rather than 'build')
emxomfld: a.out to omf conversion failed for 'U:\dev\samba-4.11.10\bin\default\source4\lib\cmdline\popt_common.c.4.o'.


More investigation required, but things are looking promising!
Title: Re: Python 3.x?
Post by: Paul Smedley on July 03, 2020, 11:55:51 am
Update:

Looks like I've fixed these errors:
[1468/2428] Compiling bin/default/source3/modules/vfs.empty.c
gcc.exe: fatal error: cannot execute 'U:/usr/local920/bin/../libexec/gcc/i686-pc-os2-emx/9.2.0/cc1.exe': spawnv: Invalid argument
compilation terminated.


but still getting:
emxomfld: a.out to omf conversion failed for 'U:\dev\samba-4.11.10\bin\default\source4\lib\cmdline\popt_common.c.4.o'.

<sigh> slowly getting there...
Title: Re: Python 3.x?
Post by: ivan on July 03, 2020, 04:29:50 pm
Hi Paul,

I don't know how you do it but then I'm not a programmer.

All I can say is thanks for the effort you put in for all of us.





Title: Re: Python 3.x?
Post by: Paul Smedley on July 04, 2020, 03:26:02 am
Hey Ivan,

I don't know how you do it but then I'm not a programmer.

All I can say is thanks for the effort you put in for all of us.

Thanks for the kind words! Ironically, Covid-19 has provided me the opportunity to spend some more time on stuff. I've been working from home for the past 3 months or so, so have gained an hour a day of commute time, and part of my lunch break towards working on things. Some people play golf, I try and port stuff ;)

Cheers,

Paul
Title: Re: Python 3.x?
Post by: Paul Smedley on July 07, 2020, 05:53:11 am
Samba configures and makes with Python 3.8.3 now!!!!!!!!!!

Now to get 'make install' working....
Title: Re: Python 3.x?
Post by: Paul Smedley on July 11, 2020, 07:14:58 am
Samba configures and makes with Python 3.8.3 now!!!!!!!!!!

Now to get 'make install' working....

Initial build:
http://smedley.id.au/tmp/python-3.8.3-os2-20200711.zip

Currently expects to be installed in /python38 - it should be possible to move it provided PYTHONPATH & PYTHONHOME are set appropriately.

I've used this to configure and build Samba 4.11.11 successfully.

Cheers,

Paul
Title: Re: Python 3.x?
Post by: Paul Smedley on July 11, 2020, 08:36:30 am
OK, 1 known issue....
running python3.exe to get the console results in 100% CPU usage....

edit: workaround for this is to remove readline.pyd
Title: Re: Python 3.x?
Post by: Martin Iturbide on July 13, 2020, 01:50:27 am
Thanks Paul for the port.

Open question to everyone. What will be interesting to run with Python on OS/2? What are you using python with?
I think it will be interesting to give the community ideas of what they can do with the python port.

Regards
Title: Re: Python 3.x?
Post by: Paul Smedley on July 13, 2020, 02:16:59 am
For me, it's for Samba 4. Currently for configuring and building the file server, but potentially to build the management utils that rely on python 3.x
Title: Re: Python 3.x?
Post by: Dariusz Piatkowski on July 14, 2020, 05:49:10 pm
Hi Paul,

Initial build:
http://smedley.id.au/tmp/python-3.8.3-os2-20200711.zip

Currently expects to be installed in /python38 - it should be possible to move it provided PYTHONPATH & PYTHONHOME are set appropriately...

Alright, so I set this up in a separate code\tools directory on my machine (to prevent any conflict with the RPM 2.7 stuff), here is what my environment config cmd file looks like:

Code: [Select]
@ECHO OFF
mode 80,60
SET MAIN=G:\code\tools\python38

rem *************************************************
rem Python 3.8 for OS/2 environment variable settings
rem *************************************************

REM change PATH below unless using default YUM/RPM install in \usr\bin
SET PATH=%MAIN%\bin;%PATH%

REM change LIBPATH to pull in the python3.8 specific DLLs
SET BEGINLIBPATH=%MAIN%\bin;

SET PYTHONHOME=%MAIN%
SET PYTHONPATH=%MAIN%\Lib\python3.8;%MAIN%\Lib\python3.8\plat-os2knix;%MAIN%\Lib\python3.8\lib-dynload;%MAIN%\Lib\python3.8\site-packages

if $%1$==$$ goto end
start python3 %1 %2 %3 %4 %5 %6 %7 %8 %9
exit
:end

So a couple of issues to report:

1) speedtest execution
Attempting to execute the speedtest script I get the following error msg:

Code: [Select]
  File "G:\code\tools\python38\Lib\python3.8/site.py", line 178
    file=sys.stderr)
        ^
SyntaxError: invalid syntax

So it feels like maybe the unix vs os2 path separator is getting in the way? I took a stab at this just by converting my python environment setup cmd to use '/' instead of '\', but that made no difference.

Since the cmd driven approach did not seem to work I took to just executing the speedtest dirctly by passing the script and the input params into python3.exe:

Code: [Select]
[G:\util\misc]\code\tools\python38\bin\python3 speedtest-cli --server 20916
Retrieving speedtest.net configuration...
Cannot retrieve speedtest configuration
ERROR: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
: unable to get local issuer certificate (_ssl.c:1108)>

In this case it would appear I hit some sort of certificate issue.

Any ideas Paul? I did not try anything else since this speed test is a pretty good excercise and if I could get it working with 3.8 that would be a pretty good test.

2) python3 invocation from CLI
The python3 environment seems to start up, but I absolutely can not type anything into the window itself. Can't even CTRL-C out of it, the only way is to kill it:

Code: [Select]
[G:\code\tools\python38\bin]python3
Python 3.8.3 (default, Jul 11 2020, 10:35:02)
[GCC 9.2.0] on os2knix
Type "help", "copyright", "credits" or "license" for more information.
>>>

Not sure what to make of that one.

Thanks,
-Dariusz
Title: Re: Python 3.x?
Post by: Joop on July 14, 2020, 06:56:19 pm
Thanks Paul for the port.

Open question to everyone. What will be interesting to run with Python on OS/2? What are you using python with?
I think it will be interesting to give the community ideas of what they can do with the python port.

Regards
There are a lot of programs which uses py scripts and therefore Python works in the background.
Title: Re: Python 3.x?
Post by: Paul Smedley on July 14, 2020, 11:21:17 pm
Hi Dariusz,

Initial build:
http://smedley.id.au/tmp/python-3.8.3-os2-20200711.zip

Currently expects to be installed in /python38 - it should be possible to move it provided PYTHONPATH & PYTHONHOME are set appropriately...

Alright, so I set this up in a separate code\tools directory on my machine (to prevent any conflict with the RPM 2.7 stuff), here is what my environment config cmd file looks like:

Code: [Select]
@ECHO OFF
mode 80,60
SET MAIN=G:\code\tools\python38

rem *************************************************
rem Python 3.8 for OS/2 environment variable settings
rem *************************************************

REM change PATH below unless using default YUM/RPM install in \usr\bin
SET PATH=%MAIN%\bin;%PATH%

REM change LIBPATH to pull in the python3.8 specific DLLs
SET BEGINLIBPATH=%MAIN%\bin;

SET PYTHONHOME=%MAIN%
SET PYTHONPATH=%MAIN%\Lib\python3.8;%MAIN%\Lib\python3.8\plat-os2knix;%MAIN%\Lib\python3.8\lib-dynload;%MAIN%\Lib\python3.8\site-packages

if $%1$==$$ goto end
start python3 %1 %2 %3 %4 %5 %6 %7 %8 %9
exit
:end

So a couple of issues to report:

1) speedtest execution
Attempting to execute the speedtest script I get the following error msg:

Code: [Select]
  File "G:\code\tools\python38\Lib\python3.8/site.py", line 178
    file=sys.stderr)
        ^
SyntaxError: invalid syntax

So it feels like maybe the unix vs os2 path separator is getting in the way? I took a stab at this just by converting my python environment setup cmd to use '/' instead of '\', but that made no difference.

Since the cmd driven approach did not seem to work I took to just executing the speedtest dirctly by passing the script and the input params into python3.exe:

Code: [Select]
[G:\util\misc]\code\tools\python38\bin\python3 speedtest-cli --server 20916
Retrieving speedtest.net configuration...
Cannot retrieve speedtest configuration
ERROR: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
: unable to get local issuer certificate (_ssl.c:1108)>

In this case it would appear I hit some sort of certificate issue.

Any ideas Paul? I did not try anything else since this speed test is a pretty good excercise and if I could get it working with 3.8 that would be a pretty good test.

2) python3 invocation from CLI
The python3 environment seems to start up, but I absolutely can not type anything into the window itself. Can't even CTRL-C out of it, the only way is to kill it:

Code: [Select]
[G:\code\tools\python38\bin]python3
Python 3.8.3 (default, Jul 11 2020, 10:35:02)
[GCC 9.2.0] on os2knix
Type "help", "copyright", "credits" or "license" for more information.
>>>

Not sure what to make of that one.

For 1) - I'll investigate locally, for 2) see https://www.os2world.com/forum/index.php/topic,2484.msg27230.html#msg27230

Cheers

Paul
Title: Re: Python 3.x?
Post by: Dave Yeo on July 15, 2020, 02:31:17 am
1 was a problem on Python 2 for a long time. IIRC, dmik fixed it when getting VirtualEnv working.
Title: Re: Python 3.x?
Post by: Paul Smedley on July 15, 2020, 05:25:27 am
Hi Dariusz,

1) speedtest execution
Attempting to execute the speedtest script I get the following error msg:

Code: [Select]
  File "G:\code\tools\python38\Lib\python3.8/site.py", line 178
    file=sys.stderr)
        ^
SyntaxError: invalid syntax

So it feels like maybe the unix vs os2 path separator is getting in the way? I took a stab at this just by converting my python environment setup cmd to use '/' instead of '\', but that made no difference.

Since the cmd driven approach did not seem to work I took to just executing the speedtest dirctly by passing the script and the input params into python3.exe:

Code: [Select]
[G:\util\misc]\code\tools\python38\bin\python3 speedtest-cli --server 20916
Retrieving speedtest.net configuration...
Cannot retrieve speedtest configuration
ERROR: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
: unable to get local issuer certificate (_ssl.c:1108)>

In this case it would appear I hit some sort of certificate issue.

2nd issue is definitely certificate related. If you unzip http://smedley.id.au/tmp/certs.zip in the root of the drive you're trying to run python3 from, it will find the certs.
Code: [Select]
Retrieving speedtest.net configuration...
Testing from Aussie Broadband (61.245.145.183)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Telstra (Adelaide) [0.14 km]: 20.047 ms
Testing download speed..........................................................
......................
Download: 106.66 Mbit/s
Testing upload speed............................................................
..........................................
Upload: 11.41 Mbit/s
I'll try and fix it to use the @unixroot location - I just need to work out if it's getting this path from ssl or from python somewhere...

Cheers,

Paul
Title: Re: Python 3.x?
Post by: Dariusz Piatkowski on July 15, 2020, 05:29:50 am
...for 2) see https://www.os2world.com/forum/index.php/topic,2484.msg27230.html#msg27230...

Yikes...umm, that's embarrassing that I missed the answer right there!!!

Yes, this works.
Title: Re: Python 3.x?
Post by: Paul Smedley on July 15, 2020, 10:09:32 am
New _ssl.pyd at http://smedley.id.au/tmp/_ssl.zip

Copy it over the top of the one in \python38\lib\python3.8\lib-dynload

Cheers,

Paul
Title: Re: Python 3.x?
Post by: Dariusz Piatkowski on July 15, 2020, 04:45:28 pm
Hi Paul,

New _ssl.pyd at http://smedley.id.au/tmp/_ssl.zip

Copy it over the top of the one in \python38\lib\python3.8\lib-dynload...

Yes sir, that worked.

Here is the output, there are additional messages showing (DeprecationWarning), not sure if this is a Python 2.x vs 3.x script difference, but I thought I'd share in case this is actually a Python 3.8 issue:

Code: [Select]
Retrieving speedtest.net configuration...
Testing from Managed Network Systems (98.143.64.3)...
Retrieving speedtest.net server list...
speedtest-cli:1265: DeprecationWarning: This method will be removed in future versions.  Use 'tree.iter()' or 'list(tree.iter())' instead.
  elements = root.getiterator('server')
Retrieving information for the selected server...
Hosted by MNSi Telecom (Windsor, ON) [1.76 km]: 32.353 ms
Testing download speed.speedtest-cli:1517: DeprecationWarning: isAlive() is deprecated, use is_alive() instead
  while prod_thread.isAlive():
speedtest-cli:1504: DeprecationWarning: isAlive() is deprecated, use is_alive() instead
  while thread.isAlive():
.......................................................................speedtest-cli:1519: DeprecationWarning: isAlive() is deprecated, use is_alive() instead
  while cons_thread.isAlive():
........
Download: 756.56 Mbit/s
Testing upload speed.speedtest-cli:1589: DeprecationWarning: isAlive() is deprecated, use is_alive() instead
  while thread.isAlive():
speedtest-cli:1602: DeprecationWarning: isAlive() is deprecated, use is_alive() instead
  while prod_thread.isAlive():
................................................................................................speedtest-cli:1604: DeprecationWarning: isAlive() is deprecated, use is_alive() instead
  while cons_thread.isAlive():
.....
Upload: 343.51 Mbit/s

In comparison, here is what the execution with Python 2.7 looks like:

Code: [Select]
Retrieving speedtest.net configuration...
Testing from Managed Network Systems (98.143.64.3)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by MNSi Telecom (Windsor, ON) [1.76 km]: 33.667 ms
Testing download speed................................................................................
Download: 759.68 Mbit/s
Testing upload speed................................................................................................
Upload: 381.17 Mbit/s

EDIT
====
Issue resolved, these messages pertain to the script itself. Version 2.1.2 addresses these, can be found here => https://pypi.org/project/speedtest-cli (https://pypi.org/project/speedtest-cli).
Title: Re: Python 3.x?
Post by: Paul Smedley on July 15, 2020, 11:13:39 pm
FYI I installed it using pip... Suggest you install pip using the instructions at https://pip.pypa.io/en/stable/installing/

Then you can install things with 'python3 \python38\bin\pip install speedtest-cli'
Title: Re: Python 3.x?
Post by: TeLLie on July 16, 2020, 10:51:57 am
Hi Paul,
I did like you suggested and it works with speedtest

[F:\usr\bin]python3.exe speedtest
Retrieving speedtest.net configuration...
Testing from Xs4all Internet BV (83.161.157.109)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Flexyz (Harderwijk) [11.02 km]: 5.984 ms
Testing download speed................................................................................
Download: 364.44 Mbit/s
Testing upload speed...
Upload: 90.25 Mbit/s
Title: Re: Python 3.x?
Post by: Paul Smedley on July 29, 2020, 10:26:43 am
Build updated to 3.8.5... http://smedley.id.au/tmp/python-3.8.5-os2-20200729.zip