Hi, this is about:
https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal
We have ~300 bugz open and blocking
https://bugzilla.redhat.com/show_bug.cgi?id=PY2REMOVAL
Some of them are > 1 week old and hence anyone can go and remove the mentioned (sub)packages. I've retired the packages that were for retirement and I will retire more once they are unblocked (needs to go to compose -> portingdb before the automation recognizes it).
I open more bugs as they are closed (I don't wan to deal with 1000 bugs at once).
Retirement is easy. The problematic part is removing subpackages. I go and manually choose the packages that appear not to be just simple "py spec generates py2 and py3 subpackages" combo, but a bit more complicated, and I leave the "simple" ones alone for now.
I'm theoretically able to remove a simple py2 subpackage in about a minute, yet it is tedious and after ~20 packages I need to stop (before I go crazy).
We have something in between 900 and 2000 packages to remove (there are 900 leaves to remove and we may make another 1100 into leaves as we go).
Is anyone willing to look into automating this? Write a spec depython2izer?
This usually means:
- removing python2- and python- BRs - removing %package -n python2 and it's metadata, description & %files - removing all %py2_build, %py2_install or their older manual variants - removing python2 tests from %check - (sometimes) removing %py3dir or python3 dir pushd/popd dance - (sometimes) removing with_python3 conditionals as they make no sense
IIRC Zbyszek said at Flock that he might work on that. Is that still true? If not, shall I draft something?
On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
Is anyone willing to look into automating this? Write a spec depython2izer?
This usually means:
- removing python2- and python- BRs
- removing %package -n python2 and it's metadata, description & %files
- removing all %py2_build, %py2_install or their older manual variants
- removing python2 tests from %check
- (sometimes) removing %py3dir or python3 dir pushd/popd dance
- (sometimes) removing with_python3 conditionals as they make no sense
IIRC Zbyszek said at Flock that he might work on that. Is that still true? If not, shall I draft something?
Yeah, I wanted to work on this. When do you need it?
Zbyszek
On 18.9.2018 21:43, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
Is anyone willing to look into automating this? Write a spec depython2izer?
This usually means:
- removing python2- and python- BRs
- removing %package -n python2 and it's metadata, description & %files
- removing all %py2_build, %py2_install or their older manual variants
- removing python2 tests from %check
- (sometimes) removing %py3dir or python3 dir pushd/popd dance
- (sometimes) removing with_python3 conditionals as they make no sense
IIRC Zbyszek said at Flock that he might work on that. Is that still true? If not, shall I draft something?
Yeah, I wanted to work on this. When do you need it?
Thank You!
I could use it right now. Realistically, we should get this done by the end of 2018. Given how we start with leaves, and that there is a week waiting period at each point, I think we would really need this working by the end of October. Getting a somehow testable version of it in 2-3 weeks would be great, if it's doable.
On 18.9.2018 21:57, Miro Hrončok wrote:
On 18.9.2018 21:43, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
Is anyone willing to look into automating this? Write a spec depython2izer?
This usually means:
- removing python2- and python- BRs - removing %package -n python2 and it's metadata, description & %files - removing all %py2_build, %py2_install or their older manual variants - removing python2 tests from %check - (sometimes) removing %py3dir or python3 dir pushd/popd dance - (sometimes) removing with_python3 conditionals as they make no sense
IIRC Zbyszek said at Flock that he might work on that. Is that still true? If not, shall I draft something?
Yeah, I wanted to work on this. When do you need it?
Thank You!
I could use it right now. Realistically, we should get this done by the end of 2018. Given how we start with leaves, and that there is a week waiting period at each point, I think we would really need this working by the end of October. Getting a somehow testable version of it in 2-3 weeks would be great, if it's doable.
Just a note that all bugs for packages that can be dropped now are filled.
On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
Is anyone willing to look into automating this? Write a spec depython2izer?
Hi,
I wrote something that mostly seems to work: https://pagure.io/pyrenamer/blob/master/f/depython2ize.py
../pyrenamer/depython2ize.py -b -d python-alchimia python-gzipstream \ python-ivi python-pdfkit python-simpleparse python-walkdir \ python-XStatic-Jasmine python-yourls ...
If does:
- removing python2- and python- BRs
yes
- removing %package -n python2 and it's metadata, description & %files
yes
- removing all %py2_build, %py2_install or their older manual variants
yes
- removing python2 tests from %check
yes
- (sometimes) removing %py3dir or python3 dir pushd/popd dance
yes (by default, can be disabled with a switch)
- (sometimes) removing with_python3 conditionals as they make no sense
no
The workflow is to first calls it with just -bd for review, and then with -bwn to actually commit.
I tested it for packages from $(bugzilla query --blocked=1625773 -s NEW). For the ones that are called python-* it seems to work, and quite a lot of them even build fine. Sometimes manual fixups will be necessary (e.g. when the py2 directories or files are referenced from %build in non-trivial ways).
For packages with generic names it doesn't work because python2 subpackage detection is broken. Should be relatively easy to fix. I'll work on that tomorrow, so please just ignore them for now.
PTAL. Let me know what bits are missing or buggy.
I'd also be happy to help with pushing some of those patches to dist-git and doing the builds, but for this we need to coordinate.
Example patches: http://in.waw.pl/~zbyszek/fedora/depy2/.
Zbyszek
On 9.10.2018 22:51, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
Is anyone willing to look into automating this? Write a spec depython2izer?
Hi,
I wrote something that mostly seems to work: https://pagure.io/pyrenamer/blob/master/f/depython2ize.py
../pyrenamer/depython2ize.py -b -d python-alchimia python-gzipstream \ python-ivi python-pdfkit python-simpleparse python-walkdir \ python-XStatic-Jasmine python-yourls ...
If does:
- removing python2- and python- BRs
yes
- removing %package -n python2 and it's metadata, description & %files
yes
- removing all %py2_build, %py2_install or their older manual variants
yes
- removing python2 tests from %check
yes
- (sometimes) removing %py3dir or python3 dir pushd/popd dance
yes (by default, can be disabled with a switch)
- (sometimes) removing with_python3 conditionals as they make no sense
no
The workflow is to first calls it with just -bd for review, and then with -bwn to actually commit.
I tested it for packages from $(bugzilla query --blocked=1625773 -s NEW). For the ones that are called python-* it seems to work, and quite a lot of them even build fine. Sometimes manual fixups will be necessary (e.g. when the py2 directories or files are referenced from %build in non-trivial ways).
For packages with generic names it doesn't work because python2 subpackage detection is broken. Should be relatively easy to fix. I'll work on that tomorrow, so please just ignore them for now.
PTAL. Let me know what bits are missing or buggy.
I'd also be happy to help with pushing some of those patches to dist-git and doing the builds, but for this we need to coordinate.
Example patches: http://in.waw.pl/~zbyszek/fedora/depy2/.
Thank You, Zbyszek, this looks very good.
Before we start going wild and use this, there are 2 things we should consider:
1) We should query bugzilla for NEW bugs without comments. Those with comments can be hand-triaged later (it isn't be many). Some comments may say OK, some may say STOP.
2) There is a list of packages to remove. We should use the list and check if the script doesn't remove anything else (e.g. it may be that we want to remove python2-foo-coolfeature, but we need to keep python2-foo). The results are obtainable via [1] and I've copied my last results to [2]. For every package the script wants to remove a check should be made that pkgs[name]["verdict"] == "drop_now".
I'll try the script myself for a couple packages tomorrow.
For coordination: I was doing it in a way that I've assigned the bug to me and changed to ASSIGNED, built, CLOSED RAWHIDE. I guess we can continue doing this (individually or in bulks) and any number of provenpackagers can proceed as long as [1]/[2] is checked.
[1] https://github.com/fedora-python/portingdb#check-drops [2] https://churchyard.fedorapeople.org/results.json
On 9.10.2018 23:42, Miro Hrončok wrote:
On 9.10.2018 22:51, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
Is anyone willing to look into automating this? Write a spec depython2izer?
Hi,
I wrote something that mostly seems to work: https://pagure.io/pyrenamer/blob/master/f/depython2ize.py
../pyrenamer/depython2ize.py -b -d python-alchimia python-gzipstream \ python-ivi python-pdfkit python-simpleparse python-walkdir \ python-XStatic-Jasmine python-yourls ...
If does:
- removing python2- and python- BRs
yes
- removing %package -n python2 and it's metadata, description & %files
yes
- removing all %py2_build, %py2_install or their older manual variants
yes
- removing python2 tests from %check
yes
- (sometimes) removing %py3dir or python3 dir pushd/popd dance
yes (by default, can be disabled with a switch)
- (sometimes) removing with_python3 conditionals as they make no sense
no
The workflow is to first calls it with just -bd for review, and then with -bwn to actually commit.
I tested it for packages from $(bugzilla query --blocked=1625773 -s NEW). For the ones that are called python-* it seems to work, and quite a lot of them even build fine. Sometimes manual fixups will be necessary (e.g. when the py2 directories or files are referenced from %build in non-trivial ways).
For packages with generic names it doesn't work because python2 subpackage detection is broken. Should be relatively easy to fix. I'll work on that tomorrow, so please just ignore them for now.
PTAL. Let me know what bits are missing or buggy.
I'd also be happy to help with pushing some of those patches to dist-git and doing the builds, but for this we need to coordinate.
Example patches: http://in.waw.pl/~zbyszek/fedora/depy2/.
Thank You, Zbyszek, this looks very good.
Before we start going wild and use this, there are 2 things we should consider:
- We should query bugzilla for NEW bugs without comments. Those with
comments can be hand-triaged later (it isn't be many). Some comments may say OK, some may say STOP.
- There is a list of packages to remove. We should use the list and
check if the script doesn't remove anything else (e.g. it may be that we want to remove python2-foo-coolfeature, but we need to keep python2-foo). The results are obtainable via [1] and I've copied my last results to [2]. For every package the script wants to remove a check should be made that pkgs[name]["verdict"] == "drop_now".
I'm writing this part now.
I'll try the script myself for a couple packages tomorrow.
For coordination: I was doing it in a way that I've assigned the bug to me and changed to ASSIGNED, built, CLOSED RAWHIDE. I guess we can continue doing this (individually or in bulks) and any number of provenpackagers can proceed as long as [1]/[2] is checked.
[1] https://github.com/fedora-python/portingdb#check-drops [2] https://churchyard.fedorapeople.org/results.json
On 9.10.2018 22:51, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
Is anyone willing to look into automating this? Write a spec depython2izer?
Hi,
I wrote something that mostly seems to work: https://pagure.io/pyrenamer/blob/master/f/depython2ize.py
../pyrenamer/depython2ize.py -b -d python-alchimia python-gzipstream \ python-ivi python-pdfkit python-simpleparse python-walkdir \ python-XStatic-Jasmine python-yourls ...
If does:
- removing python2- and python- BRs
yes
- removing %package -n python2 and it's metadata, description & %files
yes
- removing all %py2_build, %py2_install or their older manual variants
yes
- removing python2 tests from %check
yes
- (sometimes) removing %py3dir or python3 dir pushd/popd dance
yes (by default, can be disabled with a switch)
- (sometimes) removing with_python3 conditionals as they make no sense
no
The workflow is to first calls it with just -bd for review, and then with -bwn to actually commit.
I tested it for packages from $(bugzilla query --blocked=1625773 -s NEW). For the ones that are called python-* it seems to work, and quite a lot of them even build fine. Sometimes manual fixups will be necessary (e.g. when the py2 directories or files are referenced from %build in non-trivial ways).
For packages with generic names it doesn't work because python2 subpackage detection is broken. Should be relatively easy to fix. I'll work on that tomorrow, so please just ignore them for now.
PTAL. Let me know what bits are missing or buggy.
python-XStatic-* packages have a lot of remaining mvs with python2_sitelib involved. e.g. python-XStatic-Jasmine, python-XStatic-Font-Awesome.
On 9.10.2018 22:51, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
Is anyone willing to look into automating this? Write a spec depython2izer?
Hi,
I wrote something that mostly seems to work: https://pagure.io/pyrenamer/blob/master/f/depython2ize.py
../pyrenamer/depython2ize.py -b -d python-alchimia python-gzipstream \ python-ivi python-pdfkit python-simpleparse python-walkdir \ python-XStatic-Jasmine python-yourls ... ...
Hi, I've open a fresh batch of bugs. Please make sure you only remove the subpackages after a week has passed.
Thanks,
python-devel@lists.fedoraproject.org
-
Miro Hrončok -
Zbigniew Jędrzejewski-Szmek