sigh....Python3 upgrade - curl problem?

[Dariusz Piatkowski]

So this whole 'upgrade to python3' is by now a well traversed path, I did my reading, took to the CLI approach of executing: 'yum install python3 python2.7', which gave me the following results:

Code: [Select]

[G:\]yum install python3 python2.7
Loaded plugins: changelog, downloadonly, ps, replace, verify
Setting up Install Process
Resolving Dependencies
There are unfinished transactions remaining. You might consider running yum-comp
lete-transaction first to finish them.
--> Running transaction check
---> Package python2.7.pentium4 0:2.7.18-3.oc00 will be installed
--> Processing Dependency: python-rpm-macros for package: python2.7-2.7.18-3.oc00.pentium4
--> Processing Dependency: python-srpm-macros for package: python2.7-2.7.18-3.oc00.pentium4
---> Package python3.pentium4 0:3.9.17-1.oc00 will be installed
--> Processing Dependency: python3-libs = 3.9.17-1.oc00 for package: python3-3.9.17-1.oc00.pentium4
--> Processing Dependency: python39.dll for package: python3-3.9.17-1.oc00.pentium4
--> Running transaction check
---> Package python-libs.pentium4 0:2.7.6-25.oc00 will be obsoleted
--> Processing Dependency: python-libs = 2.7.6-25.oc00 for package: python-2.7.6-25.oc00.pentium4
---> Package python-rpm-macros.noarch 0:3.9-3.oc00 will be installed
---> Package python-srpm-macros.noarch 0:3.9-3.oc00 will be installed
---> Package python3-libs.pentium4 0:3.9.17-1.oc00 will be obsoleting
--> Running transaction check
---> Package python.pentium4 0:2.7.6-25.oc00 will be obsoleted
---> Package python-unversioned-command.pentium4 0:3.9.17-1.oc00 will be obsoleting
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================

 Package                      Arch       Version            Repository     Size
================================================================================

Installing:
 python-unversioned-command   pentium4   3.9.17-1.oc00      netlabs-rel   8.6 k
     replacing  python.pentium4 2.7.6-25.oc00
 python2.7                    pentium4   2.7.18-3.oc00      netlabs-rel    18 M
 python3                      pentium4   3.9.17-1.oc00      netlabs-rel    22 k
 python3-libs                 pentium4   3.9.17-1.oc00      netlabs-rel    13 M
     replacing  python-libs.pentium4 2.7.6-25.oc00
Installing for dependencies:
 python-rpm-macros            noarch     3.9-3.oc00         netlabs-rel   9.1 k
 python-srpm-macros           noarch     3.9-3.oc00         netlabs-rel    15 k

Transaction Summary
================================================================================

Install       6 Packages

Total download size: 30 M
Is this ok [y/N]:

INPUT => Y

Downloading Packages:
(1/6): python-rpm-ma | 9.1 kB  00:00
(2/6): python-srpm-m |  15 kB  00:00
(3/6): python-unvers | 8.6 kB  00:00
(4/6): python2.7-2.7 |  18 MB  00:07     ====-] 2.1 MB/s |  17 MB  00:00 ETA
(5/6): python3-3.9.1 |  22 kB  00:00
(6/6): python3-libs- |  13 MB  00:05     ==== ] 1.9 MB/s |  12 MB  00:00 ETA
--------------------------------------------------------------------------------

Total                                           2.2 MB/s |  30 MB     00:13

Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
  Installing : python-srpm-macros-3.9-3.oc00.noarch                         1/8
  Installing : python3-libs-3.9.17-1.oc00.pentium4                          2/8
  Installing : python3-3.9.17-1.oc00.pentium4                               3/8
  Installing : python-rpm-macros-3.9-3.oc00.noarch                          4/8
  Installing : python2.7-2.7.18-3.oc00.pentium4                             5/8
  Installing : python-unversioned-command-3.9.17-1.oc00.pentium4            6/8
  Erasing    : python-2.7.6-25.oc00.pentium4                                7/8
  Erasing    : python-libs-2.7.6-25.oc00.pentium4                           8/8

Rpmdb checksum is invalid: dCDPT(pkg checksums): python3-libs.pentium4 0:3.9.17-1.oc00 - u

Following this upgrade my inventory of python RPM packages installed is as follows:

Code: [Select]

Installed Packages
python-dateutil.noarch                        2.6.0-1.oc00           installed
python-pycurl.pentium4                        7.19.5.1-2.oc00        installed
python-rpm-macros.noarch                      3.9-3.oc00             installed
python-srpm-macros.noarch                     3.9-3.oc00             installed
python-unversioned-command.pentium4           3.9.17-1.oc00          installed
python2-deltarpm.pentium4                     3.6-1.oc00             installed
python2-libxml2.pentium4                      2.9.10-3.oc00          installed
python2-rpm.pentium4                          4.13.0-20.oc00         installed
python2.7.pentium4                            2.7.18-3.oc00          installed
python3.pentium4                              3.9.17-1.oc00          installed
python3-libs.pentium4                         3.9.17-1.oc00          installed

Therefore, my next logical (and maybe this was a mistake???) step was to upgrade all the pythonx stuff to python3, which I went to ANPM for:

1) python3-dateutil - this seems to have gone fine

Code: [Select]

----------[ 10 Dec 2023 10:26:12 ]----------
Executing: @python2.7 G:\UTIL\ANPM\scripts\yum_install_dep.py python3-dateutil
Enabling temporary repository arcanoae-sub
Enabling temporary repository arcanoae-exp
Enabling temporary repository arcanoae-stage
Enabling temporary repository arcanoae-arcaos
No argument pipe.
Return code: 0
----------[ 10 Dec 2023 10:26:25 ]----------
Executing: @python2.7 G:\UTIL\ANPM\scripts\yum_install.py python3-dateutil
No argument pipe.
Enabling temporary repository arcanoae-sub
Enabling temporary repository arcanoae-exp
Enabling temporary repository arcanoae-stage
Enabling temporary repository arcanoae-arcaos
Running Transaction Check
Warning: RPMDB altered outside of yum.
PackageSackError(): Rpmdb checksum is invalid: dCDPT(pkg checksums): python3-six.noarch 0:1.10.0-2.oc00 - u
Error: Rpmdb checksum is invalid: dCDPT(pkg checksums): python3-six.noarch 0:1.10.0-2.oc00 - u
PackageSackError()
Return code: 0
----------[ 10 Dec 2023 10:26:41 ]----------
Executing: @python2.7 G:\UTIL\ANPM\scripts\yum_list.py
Setting up yum
No argument pipe.
Adding temporary repositories
Enabling temporary repository arcanoae-sub
Enabling temporary repository arcanoae-exp
Enabling temporary repository arcanoae-stage
Enabling temporary repository arcanoae-arcaos
Initializing cache
Requesting package list
Return code: 0
----------[ 10 Dec 2023 10:27:05 ]----------
Executing: @python2.7 G:\UTIL\ANPM\scripts\yum_info.py python3-six
Warning: progress pipe not available.
Enabling temporary repository arcanoae-sub
Enabling temporary repository arcanoae-exp
Enabling temporary repository arcanoae-stage
Enabling temporary repository arcanoae-arcaos
No argument pipe.
Return code: 0
----------[ 10 Dec 2023 10:27:17 ]----------
Executing: @python2.7 G:\UTIL\ANPM\scripts\yum_files.py installed python3-six
Enabling temporary repository arcanoae-sub
Enabling temporary repository arcanoae-exp
Enabling temporary repository arcanoae-stage
Enabling temporary repository arcanoae-arcaos
No argument pipe.
Return code: 0
----------[ 10 Dec 2023 10:27:51 ]----------
Executing: @python2.7 G:\UTIL\ANPM\scripts\yum_files.py installed python-unversioned-command
Enabling temporary repository arcanoae-sub
Enabling temporary repository arcanoae-exp
Enabling temporary repository arcanoae-stage
Enabling temporary repository arcanoae-arcaos
No argument pipe.
Return code: 0
----------[ 10 Dec 2023 10:28:02 ]----------
Executing: @python2.7 G:\UTIL\ANPM\scripts\yum_info.py python-unversioned-command
Warning: progress pipe not available.
Enabling temporary repository arcanoae-sub
Enabling temporary repository arcanoae-exp
Enabling temporary repository arcanoae-stage
Enabling temporary repository arcanoae-arcaos
No argument pipe.
Return code: 0

2) python3-curl - this seems to have NOT gone fine

Code: [Select]

----------[ 10 Dec 2023 10:29:24 ]----------
Executing: @python2.7 G:\UTIL\ANPM\scripts\yum_install_dep.py python3-pycurl
Enabling temporary repository arcanoae-sub
Enabling temporary repository arcanoae-exp
Enabling temporary repository arcanoae-stage
Enabling temporary repository arcanoae-arcaos
No argument pipe.
Return code: 0
----------[ 10 Dec 2023 10:29:32 ]----------
Executing: @python2.7 G:\UTIL\ANPM\scripts\yum_install.py python3-pycurl
No argument pipe.
Enabling temporary repository arcanoae-sub
Enabling temporary repository arcanoae-exp
Enabling temporary repository arcanoae-stage
Enabling temporary repository arcanoae-arcaos
Running Transaction Check
Warning: RPMDB altered outside of yum.
PackageSackError(): Rpmdb checksum is invalid: dCDPT(pkg checksums): python3-pycurl.pentium4 0:7.44.1-3.oc00 - u
Error: Rpmdb checksum is invalid: dCDPT(pkg checksums): python3-pycurl.pentium4 0:7.44.1-3.oc00 - u
PackageSackError()
Return code: 0
----------[ 10 Dec 2023 10:29:58 ]----------
Executing: @python2.7 G:\UTIL\ANPM\scripts\yum_list.py
Traceback (most recent call last):
  File "G:\UTIL\ANPM\scripts\yum_list.py", line 6, in <module>
    import yum
  File "G:/usr/lib/python2.7/site-packages/yum/__init__.py", line 52, in <module>
    import config
  File "G:/usr/lib/python2.7/site-packages/yum/config.py", line 30, in <module>
    from parser import ConfigPreProcessor, varReplace
  File "G:/usr/lib/python2.7/site-packages/yum/parser.py", line 4, in <module>
    import urlgrabber
  File "G:/usr/lib/python2.7/site-packages/urlgrabber/__init__.py", line 55, in <module>
    from grabber import urlgrab, urlopen, urlread
  File "G:/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 512, in <module>
    import pycurl
ImportError: No module named pycurl
Return code: 0
----------[ 10 Dec 2023 10:30:00 ]----------
Executing: @python2.7 G:\UTIL\ANPM\scripts\yum_list.py available
Traceback (most recent call last):
  File "G:\UTIL\ANPM\scripts\yum_list.py", line 6, in <module>
    import yum
  File "G:/usr/lib/python2.7/site-packages/yum/__init__.py", line 52, in <module>
    import config
  File "G:/usr/lib/python2.7/site-packages/yum/config.py", line 30, in <module>
    from parser import ConfigPreProcessor, varReplace
  File "G:/usr/lib/python2.7/site-packages/yum/parser.py", line 4, in <module>
    import urlgrabber
  File "G:/usr/lib/python2.7/site-packages/urlgrabber/__init__.py", line 55, in <module>
    from grabber import urlgrab, urlopen, urlread
  File "G:/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 512, in <module>
    import pycurl
ImportError: No module named pycurl
Return code: 0
----------[ 10 Dec 2023 10:39:59 ]----------
Executing: @python2.7 G:\UTIL\ANPM\scripts\yum_populate.py
Traceback (most recent call last):
  File "G:\UTIL\ANPM\scripts\yum_populate.py", line 6, in <module>
    import yum
  File "G:/usr/lib/python2.7/site-packages/yum/__init__.py", line 52, in <module>
    import config
  File "G:/usr/lib/python2.7/site-packages/yum/config.py", line 30, in <module>
    from parser import ConfigPreProcessor, varReplace
  File "G:/usr/lib/python2.7/site-packages/yum/parser.py", line 4, in <module>
    import urlgrabber
  File "G:/usr/lib/python2.7/site-packages/urlgrabber/__init__.py", line 55, in <module>
    from grabber import urlgrab, urlopen, urlread
  File "G:/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 512, in <module>
    import pycurl
ImportError: No module named pycurl
Return code: 0
----------[ 10 Dec 2023 10:40:01 ]----------
Executing: @python2.7 G:\UTIL\ANPM\scripts\yum_list.py
Traceback (most recent call last):
  File "G:\UTIL\ANPM\scripts\yum_list.py", line 6, in <module>
    import yum
  File "G:/usr/lib/python2.7/site-packages/yum/__init__.py", line 52, in <module>
    import config
  File "G:/usr/lib/python2.7/site-packages/yum/config.py", line 30, in <module>
    from parser import ConfigPreProcessor, varReplace
  File "G:/usr/lib/python2.7/site-packages/yum/parser.py", line 4, in <module>
    import urlgrabber
  File "G:/usr/lib/python2.7/site-packages/urlgrabber/__init__.py", line 55, in <module>
    from grabber import urlgrab, urlopen, urlread
  File "G:/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 512, in <module>
    import pycurl
ImportError: No module named pycurl
Return code: 0

Sure enough, ANPM is no longer able to pull anything up, reports a pipe error, although from CLI I get the following:

Code: [Select]

[G:\]yum list python* |less
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   No module named pycurl

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.18 (default, Dec 27 2021, 15:11:56)
[GCC 9.2.0 20190812 (OS/2 RPM build 9.2.0-5.oc00)]

If you cannot solve this problem yourself, please go to
the yum faq at:
  http://yum.baseurl.org/wiki/Faq

So it would appear that installing python3-pycurl obsoleted python-pycurl (which I would have expected is the correct thing to do), but for some reason ANPM continues to see python 2.7.18 as the version it is executing.

From any CLI when checking for the installed Python3 version I get the following:

Code: [Select]

[G:\]python --version
Python 3.9.17

...so clearly I'm running Python3, but ANPM is still using 2.7.18, is that by design?

Either way, any ideas how I can get back to a working YUM/RPM/ANPM config?...and what (if anything) do I do about the remaining pythonx RPM packages (do I upgrade, or leave alone)?

OH, BTW: a system reboot was executed to rule out any other impacts, results are the same

[Dave Yeo]

Hmm, I never forced any Python upgrades besides the python2.7/python3 one and see I have python2-pycurl and python3-pycurl installed. Lots of things are python2 including ANPM.
If yum itself works, install python2-curl or perhaps download the zip and install the 8 files manually to get ANPM working then install the package.

[JTA]

From the perspective of a new install of ArcaOS, there seems to be a difficulty in applying ALL  outstanding ANPM updates at once, as the python issue pops up (2.x and 3.x aren't in place all the way yet?).

There are supposedly help articles on AN support pages to get you around this, but I just basically did a few specific updates by hand (in ANPM), and got it going again. IIRC, it was something like:

- update only the base 2.x python
- then update only the base 3.x python

Can't quite remember which one of the two I did first, but I think the trick was doing them separately ... after getting both of those to succeed, the last step is a mass ANPM update of anything else remaining, which also then worked for me.

I just stumbled through that, so didn't document each and every step, and assumed that what was on AN's support pages was a much better write-up.

Hope this helps ...

[Martin Iturbide]

Hello

From the perspective of a new install of ArcaOS, there seems to be a difficulty in applying ALL  outstanding ANPM updates at once, as the python issue pops up (2.x and 3.x aren't in place all the way yet?).

I wish I know more about python, like understanding more on how to run scripts.

I had an issue with ANPM on a recently installed copy of ArcaOS 5.1 that ANPM will not run an update all and will stuck on python.  Error: "0003523: Can not update to python 3"

To fix the problem was basically:
- In ANPM  went to "Available - RPM"
- Select"python2.7", "python3", "python-unversioned-command" and run "Install".

This is a package issue rather than an ANPM issue (the same problem occurs at the command line) according to what Lewis told me on the ticket.

Regards

[Dariusz Piatkowski]

Hi Dave,

Thanks for the quick response.

Hmm, I never forced any Python upgrades besides the python2.7/python3 one and see I have python2-pycurl and python3-pycurl installed. Lots of things are python2 including ANPM.
If yum itself works, install python2-curl or perhaps download the zip and install the 8 files manually to get ANPM working then install the package.

Alright...so I continued to troubleshoot the issue in the meantime, as in reviewing the scripts ANPM uses and concluded that it's a Python2 dependent implementation.

Therefore, precisely as you pointed out it made sense to bring back the pycurl RPM in order for the current YUM/ANPM to get back into working status.

To that end, I now see the following:

Code: [Select]

[G:\]rpm -i /test/python-pycurl-7.19.5.1-2.oc00.pentium4.rpm
error: Failed dependencies:
        python-pycurl < 7.44.1-3.oc00 is obsoleted by (installed) python3-pycurl-7.44.1-3.oc00.pentium4

...clearly the python-pycurl and python3-pycurl are incompatible with each other, and in hindsight it seems like I should NOT have attempted to upgrade anything beyond just the core Python itself (at which point in time this becomes a little nonsensical / counterintuitive, after all, if the Python runtime is now at 3.x and if I happen to be using other python utils, shouldn't I be updating them all to make sure I am using the matching python stuff???).

Anyways, OK, so it looks like I'm going to force RPM to install python-pycurl regardless:

Code: [Select]

[G:\]rpm -ivh --nodeps /test/python-pycurl-7.19.5.1-2.oc00.pentium4.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:python-pycurl-7.19.5.1-2.oc00    ################################# [100%]

Yup, all good now: YUM at CLI works, as does ANPM.

[Dariusz Piatkowski]

Martin & JTA,

Thank you both for your feedback!

While I was aware of what you both highlighted, I did not realize that certain python2 version packages must not be upgraded, at least my results so far indicate that be the case for the python-pycurl package.

I'm reluctant to execute any other upgrades, and will monitor whether the python3-dateutil needs to be either reverted, or the python-dateutil re-installed.

[Dariusz Piatkowski]

...let me add the following:

Given the following RPM package installs on my system today:

Code: [Select]

Installed Packages
python-pycurl.pentium4                        7.19.5.1-2.oc00        installed
python-rpm-macros.noarch                      3.9-3.oc00             installed
python-srpm-macros.noarch                     3.9-3.oc00             installed
python-unversioned-command.pentium4           3.9.17-1.oc00          installed
python2-deltarpm.pentium4                     3.6-1.oc00             installed
python2-libxml2.pentium4                      2.9.10-3.oc00          installed
python2-rpm.pentium4                          4.13.0-20.oc00         installed
python2.7.pentium4                            2.7.18-3.oc00          installed
python3.pentium4                              3.9.17-1.oc00          installed
python3-dateutil.noarch                       1:2.8.1-3.oc00         installed
python3-libs.pentium4                         3.9.17-1.oc00          installed
python3-pycurl.pentium4                       7.44.1-3.oc00          installed
python3-six.noarch                            1.10.0-2.oc00          installed

I seem to be OK.

However, looking at what's out there in the repositories, there are basically:

1) python
2) python2
3) python3

versions of most of such RPM packages, albeit not all (meaning, no one-to-one match for each package).

So how do we know what should be installed, or to be more specific: which pythonX versions are needed by which apps? Seems pretty easy to answer, after all that's what dependency checking is for...but if that was the case shouldn't ANPM have complained about my attempt to update python-pycurl??? (maybe it can't do that b/c it doesn't check the packages that are being updated against it's built-in "bare necessities" list?)

In other words: I never spotted ANPM documented as requiring Python2 for example...clearly I ran it for quite some time, up until today, using python-dateutil instead of python2-dateutil...so what are the real/known dependencies?

[Dave Yeo]

Hi Dariusz, yes, the packaging is somewhat broken, not enough hands on deck so we get things like a dependency on Python 3 in netlabs-rel.
As for python-pycurl, what you want is python2-pycurl.

[Alex Taylor]

For the record, to clear up any confusion: ANPM does use Python 2. (Blame the Python guys for deciding to make Python 3 a different i.e. incompatible scripting language from Python 2.)

Porting ANPM to Python 3 would be a large job, and since many people presumably haven't installed Python 3 yet, it seems premature to do so.

[Dariusz Piatkowski]

Hi Alex,

For the record, to clear up any confusion: ANPM does use Python 2. (Blame the Python guys for deciding to make Python 3 a different i.e. incompatible scripting language from Python 2.)...

OK, so ANPM does need Python2, perfect.

FWIW, this is not a blame assignment take on my part, rather I simply want to understand the operational criteria within which I need to contain the PythonX installs on my system.

The priority here being: ANPM & our RPM/YUM installations working perfectly, everything else becomes a secondary thing.

So taking Dave's response further: can you define what specific PythonX level of other utils we should be matching up here? Or should I consider your response above to me: pick up everything that's 'python2-xx'?

I just want to play it safe...that approach has worked for me for quite a long time, and I was holding off on this Python upgrade for a very long time primarily because of that reason.

Thanks, as always!

[Dariusz Piatkowski]

Last update on my part, maybe relevant to others who might run into this type of an issue.

I proceeded with the remaining python package updates to bring things up to the python2 release level. Subsequently, YUM/RPM/ANPM all appear to work successfully with the following combination of packages having been deployed:

Code: [Select]

Installed Packages

python-rpm-macros.noarch                      3.9-3.oc00
python-srpm-macros.noarch                     3.9-3.oc00
python-unversioned-command.pentium4           3.9.17-1.oc00

python2-dateutil.noarch                       1:2.8.1-3.oc00
python2-deltarpm.pentium4                     3.6-1.oc00
python2-libxml2.pentium4                      2.9.10-3.oc00
python2-pycurl.pentium4                       7.44.1-3.oc00
python2-rpm.pentium4                          4.13.0-20.oc00
python2-six.noarch                            1.10.0-2.oc00
python2.7.pentium4                            2.7.18-3.oc00

python3.pentium4                              3.9.17-1.oc00
python3-dateutil.noarch                       1:2.8.1-3.oc00
python3-libs.pentium4                         3.9.17-1.oc00
python3-pycurl.pentium4                       7.44.1-3.oc00
python3-six.noarch                            1.10.0-2.oc00

There are a couple more python2 packages that perhaps could be deployed (in my specific use-case), but I haven't taken that chance:

- python2-rpm-macros.noarch                     1-3.oc00
- python2.7-unversioned-command.pentium4        2.7.18-3.oc00