This adds zchunk support for the updates and updates-testing repositories for both rpms and modularity. We already have zchunk metadata being generated for the fedora repository. I'd like to get this in before Beta comes out so Beta users will have zchunk-enabled updates-testing repositories when Beta is released.
I am making the assumption that a zchunk-enabled createrepo_c (0.12.0-2 or later) is available on the builders (I think I'm safe making that assumption, since zchunk metadata is already being generated for some repos).
I have *not* tested this patch, because I'm not sure how I'd go about doing so. If we don't have any test builders, my suggestion would be to wait until no compose is running, and then run this play on a builder, verifying that the generated pungi configuration is valid for both f29 and f30, with no createrepo_extra_args in f29.
Signed-off-by: Jonathan Dieter jdieter@gmail.com --- roles/bodhi2/backend/templates/pungi.module.conf.j2 | 3 +++ roles/bodhi2/backend/templates/pungi.rpm.conf.j2 | 3 +++ 2 files changed, 6 insertions(+)
diff --git a/roles/bodhi2/backend/templates/pungi.module.conf.j2 b/roles/bodhi2/backend/templates/pungi.module.conf.j2 index bb021eb13..7dad35403 100644 --- a/roles/bodhi2/backend/templates/pungi.module.conf.j2 +++ b/roles/bodhi2/backend/templates/pungi.module.conf.j2 @@ -59,6 +59,9 @@ greedy_method = 'build' createrepo_c = True createrepo_checksum = 'sha256' createrepo_deltas = False +[% if release.version_int >= 30 %] +createrepo_extra_args = ['--zck', '--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]'] +[% endif %]
#jigdo create_jigdo = False diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 index 8d9e9a3f2..020736aee 100644 --- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 +++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 @@ -66,6 +66,9 @@ createrepo_deltas = [ ('^Everything$', {'*': True}) ] createrepo_database = True +[% if release.version_int >= 30 %] +createrepo_extra_args = ['--zck', '--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]'] +[% endif %]
# CHECKSUMS media_checksums = ['sha256']
On 3/11/19 12:26 AM, Jonathan Dieter wrote:
This adds zchunk support for the updates and updates-testing repositories for both rpms and modularity. We already have zchunk metadata being generated for the fedora repository. I'd like to get this in before Beta comes out so Beta users will have zchunk-enabled updates-testing repositories when Beta is released.
yeah, hopefilly not too much pain since it's been in rawhide a while now.
I am making the assumption that a zchunk-enabled createrepo_c (0.12.0-2 or later) is available on the builders (I think I'm safe making that assumption, since zchunk metadata is already being generated for some repos).
Well, bodhi-backend01 (where the updates process/pungi runs for these) has a newer one, so yes. It's all run on bodhi-backend01, not builders.
I have *not* tested this patch, because I'm not sure how I'd go about doing so. If we don't have any test builders, my suggestion would be to wait until no compose is running, and then run this play on a builder, verifying that the generated pungi configuration is valid for both f29 and f30, with no createrepo_extra_args in f29.
yeah, we can commit this, run the playbook then examine the results.
kevin
On Mon, 2019-03-11 at 11:24 -0700, Kevin Fenzi wrote:
On 3/11/19 12:26 AM, Jonathan Dieter wrote:
This adds zchunk support for the updates and updates-testing repositories for both rpms and modularity. We already have zchunk metadata being generated for the fedora repository. I'd like to get this in before Beta comes out so Beta users will have zchunk- enabled updates-testing repositories when Beta is released.
yeah, hopefilly not too much pain since it's been in rawhide a while now.
I am making the assumption that a zchunk-enabled createrepo_c (0.12.0-2 or later) is available on the builders (I think I'm safe making that assumption, since zchunk metadata is already being generated for some repos).
Well, bodhi-backend01 (where the updates process/pungi runs for these) has a newer one, so yes. It's all run on bodhi-backend01, not builders.
I have *not* tested this patch, because I'm not sure how I'd go about doing so. If we don't have any test builders, my suggestion would be to wait until no compose is running, and then run this play on a builder, verifying that the generated pungi configuration is valid for both f29 and f30, with no createrepo_extra_args in f29.
yeah, we can commit this, run the playbook then examine the results.
Great. I'm on UTC time right now, so hopefully I'll be off of work and available if there are any issues whenever we get another +1 and you run it. I do expect that it will go fine.
Jonathan
On Mon, 2019-03-11 at 20:23 +0000, Jonathan Dieter wrote:
On Mon, 2019-03-11 at 11:24 -0700, Kevin Fenzi wrote:
On 3/11/19 12:26 AM, Jonathan Dieter wrote:
This adds zchunk support for the updates and updates-testing repositories for both rpms and modularity. We already have zchunk metadata being generated for the fedora repository. I'd like to get this in before Beta comes out so Beta users will have zchunk- enabled updates-testing repositories when Beta is released.
yeah, hopefilly not too much pain since it's been in rawhide a while now.
I am making the assumption that a zchunk-enabled createrepo_c (0.12.0-2 or later) is available on the builders (I think I'm safe making that assumption, since zchunk metadata is already being generated for some repos).
Well, bodhi-backend01 (where the updates process/pungi runs for these) has a newer one, so yes. It's all run on bodhi-backend01, not builders.
I have *not* tested this patch, because I'm not sure how I'd go about doing so. If we don't have any test builders, my suggestion would be to wait until no compose is running, and then run this play on a builder, verifying that the generated pungi configuration is valid for both f29 and f30, with no createrepo_extra_args in f29.
yeah, we can commit this, run the playbook then examine the results.
Great. I'm on UTC time right now, so hopefully I'll be off of work and available if there are any issues whenever we get another +1 and you run it. I do expect that it will go fine.
Since we never got the extra +1 to get this in before Beta, are we at a point where we can turn this on now?
Thanks Jonathan
On 3/29/19 1:33 PM, Jonathan Dieter wrote:
On Mon, 2019-03-11 at 20:23 +0000, Jonathan Dieter wrote:
On Mon, 2019-03-11 at 11:24 -0700, Kevin Fenzi wrote:
On 3/11/19 12:26 AM, Jonathan Dieter wrote:
This adds zchunk support for the updates and updates-testing repositories for both rpms and modularity. We already have zchunk metadata being generated for the fedora repository. I'd like to get this in before Beta comes out so Beta users will have zchunk- enabled updates-testing repositories when Beta is released.
yeah, hopefilly not too much pain since it's been in rawhide a while now.
I am making the assumption that a zchunk-enabled createrepo_c (0.12.0-2 or later) is available on the builders (I think I'm safe making that assumption, since zchunk metadata is already being generated for some repos).
Well, bodhi-backend01 (where the updates process/pungi runs for these) has a newer one, so yes. It's all run on bodhi-backend01, not builders.
I have *not* tested this patch, because I'm not sure how I'd go about doing so. If we don't have any test builders, my suggestion would be to wait until no compose is running, and then run this play on a builder, verifying that the generated pungi configuration is valid for both f29 and f30, with no createrepo_extra_args in f29.
yeah, we can commit this, run the playbook then examine the results.
Great. I'm on UTC time right now, so hopefully I'll be off of work and available if there are any issues whenever we get another +1 and you run it. I do expect that it will go fine.
Since we never got the extra +1 to get this in before Beta, are we at a point where we can turn this on now?
Nope, we are still frozen until the day after beta release. ;(
But I will try and scare up another +1
kevin
+1
On Sat, 30 Mar 2019 at 13:53, Kevin Fenzi kevin@scrye.com wrote:
On 3/29/19 1:33 PM, Jonathan Dieter wrote:
On Mon, 2019-03-11 at 20:23 +0000, Jonathan Dieter wrote:
On Mon, 2019-03-11 at 11:24 -0700, Kevin Fenzi wrote:
On 3/11/19 12:26 AM, Jonathan Dieter wrote:
This adds zchunk support for the updates and updates-testing repositories for both rpms and modularity. We already have zchunk metadata being generated for the fedora repository. I'd like to get this in before Beta comes out so Beta users will have zchunk- enabled updates-testing repositories when Beta is released.
yeah, hopefilly not too much pain since it's been in rawhide a while now.
I am making the assumption that a zchunk-enabled createrepo_c (0.12.0-2 or later) is available on the builders (I think I'm safe making that assumption, since zchunk metadata is already being generated for some repos).
Well, bodhi-backend01 (where the updates process/pungi runs for these) has a newer one, so yes. It's all run on bodhi-backend01, not builders.
I have *not* tested this patch, because I'm not sure how I'd go about doing so. If we don't have any test builders, my suggestion would be to wait until no compose is running, and then run this play on a builder, verifying that the generated pungi configuration is valid for both f29 and f30, with no createrepo_extra_args in f29.
yeah, we can commit this, run the playbook then examine the results.
Great. I'm on UTC time right now, so hopefully I'll be off of work and available if there are any issues whenever we get another +1 and you run it. I do expect that it will go fine.
Since we never got the extra +1 to get this in before Beta, are we at a point where we can turn this on now?
Nope, we are still frozen until the day after beta release. ;(
But I will try and scare up another +1
kevin
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Sat, 2019-03-30 at 14:05 -0400, Stephen John Smoogen wrote:
+1
On Sat, 30 Mar 2019 at 13:53, Kevin Fenzi kevin@scrye.com wrote:
On 3/29/19 1:33 PM, Jonathan Dieter wrote:
On Mon, 2019-03-11 at 20:23 +0000, Jonathan Dieter wrote:
On Mon, 2019-03-11 at 11:24 -0700, Kevin Fenzi wrote:
On 3/11/19 12:26 AM, Jonathan Dieter wrote:
This adds zchunk support for the updates and updates-testing repositories for both rpms and modularity. We already have zchunk metadata being generated for the fedora repository. I'd like to get this in before Beta comes out so Beta users will have zchunk- enabled updates-testing repositories when Beta is released.
yeah, hopefilly not too much pain since it's been in rawhide a while now.
I am making the assumption that a zchunk-enabled createrepo_c (0.12.0-2 or later) is available on the builders (I think I'm safe making that assumption, since zchunk metadata is already being generated for some repos).
Well, bodhi-backend01 (where the updates process/pungi runs for these) has a newer one, so yes. It's all run on bodhi-backend01, not builders.
I have *not* tested this patch, because I'm not sure how I'd go about doing so. If we don't have any test builders, my suggestion would be to wait until no compose is running, and then run this play on a builder, verifying that the generated pungi configuration is valid for both f29 and f30, with no createrepo_extra_args in f29.
yeah, we can commit this, run the playbook then examine the results.
Great. I'm on UTC time right now, so hopefully I'll be off of work and available if there are any issues whenever we get another +1 and you run it. I do expect that it will go fine.
Since we never got the extra +1 to get this in before Beta, are we at a point where we can turn this on now?
Nope, we are still frozen until the day after beta release. ;(
But I will try and scare up another +1
kevin
-- Stephen J Smoogen.
Stephen and Kevin, thanks so much!
Jonathan
Rebased patch against master
This adds zchunk support for the updates and updates-testing repositories for both rpms and modularity
Signed-off-by: Jonathan Dieter jdieter@gmail.com --- roles/bodhi2/backend/templates/pungi.module.conf.j2 | 3 +++ roles/bodhi2/backend/templates/pungi.rpm.conf.j2 | 3 +++ 2 files changed, 6 insertions(+)
diff --git a/roles/bodhi2/backend/templates/pungi.module.conf.j2 b/roles/bodhi2/backend/templates/pungi.module.conf.j2 index 43c6a7e5f..b5bb0c1fb 100644 --- a/roles/bodhi2/backend/templates/pungi.module.conf.j2 +++ b/roles/bodhi2/backend/templates/pungi.module.conf.j2 @@ -61,6 +61,9 @@ greedy_method = 'build' createrepo_c = True createrepo_checksum = 'sha256' createrepo_deltas = False +[% if release.version_int >= 30 %] +createrepo_extra_args = ['--zck', '--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]'] +[% endif %]
#jigdo create_jigdo = False diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 index 8d9e9a3f2..020736aee 100644 --- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 +++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 @@ -66,6 +66,9 @@ createrepo_deltas = [ ('^Everything$', {'*': True}) ] createrepo_database = True +[% if release.version_int >= 30 %] +createrepo_extra_args = ['--zck', '--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]'] +[% endif %]
# CHECKSUMS media_checksums = ['sha256']
On Sat, 2019-03-30 at 15:13 -0700, Kevin Fenzi wrote:
On 3/30/19 11:35 AM, Jonathan Dieter wrote:
Stephen and Kevin, thanks so much!
Can you rebase and attach the patch?
It's not applying cleanly for me... if not I can try and manually poke it later.
kevin
I've just rebased and posted the updated patch. There were no conflicts when I rebased it against master, so please let me know if I should be rebasing against a different branch.
Jonathan
On 3/30/19 3:32 PM, Jonathan Dieter wrote:
On Sat, 2019-03-30 at 15:13 -0700, Kevin Fenzi wrote:
On 3/30/19 11:35 AM, Jonathan Dieter wrote:
Stephen and Kevin, thanks so much!
Can you rebase and attach the patch?
It's not applying cleanly for me... if not I can try and manually poke it later.
kevin
I've just rebased and posted the updated patch. There were no conflicts when I rebased it against master, so please let me know if I should be rebasing against a different branch.
Not sure why it was complaining, but its applied and pushed now.
kevin
On Sat, 2019-03-30 at 16:00 -0700, Kevin Fenzi wrote:
On 3/30/19 3:32 PM, Jonathan Dieter wrote:
On Sat, 2019-03-30 at 15:13 -0700, Kevin Fenzi wrote:
On 3/30/19 11:35 AM, Jonathan Dieter wrote:
Stephen and Kevin, thanks so much!
Can you rebase and attach the patch?
It's not applying cleanly for me... if not I can try and manually poke it later.
kevin
I've just rebased and posted the updated patch. There were no conflicts when I rebased it against master, so please let me know if I should be rebasing against a different branch.
Not sure why it was complaining, but its applied and pushed now.
kevin
Great, thanks! I'll be keeping an eye on the composes to see if there are any issues.
Jonathan
On Sat, Mar 30, 2019 at 11:10 PM Jonathan Dieter jdieter@gmail.com wrote:
On Sat, 2019-03-30 at 16:00 -0700, Kevin Fenzi wrote:
On 3/30/19 3:32 PM, Jonathan Dieter wrote:
On Sat, 2019-03-30 at 15:13 -0700, Kevin Fenzi wrote:
On 3/30/19 11:35 AM, Jonathan Dieter wrote:
Stephen and Kevin, thanks so much!
Can you rebase and attach the patch?
It's not applying cleanly for me... if not I can try and manually poke it later.
kevin
I've just rebased and posted the updated patch. There were no conflicts when I rebased it against master, so please let me know if I should be rebasing against a different branch.
Not sure why it was complaining, but its applied and pushed now.
kevin
Great, thanks! I'll be keeping an eye on the composes to see if there are any issues.
Wasn't this disabled in the main Fedora branched compose? If so why would we want to enable it only on updates?
On 3/30/19 9:50 PM, Peter Robinson wrote:
Great, thanks! I'll be keeping an eye on the composes to see if there are any issues.
Wasn't this disabled in the main Fedora branched compose? If so why would we want to enable it only on updates?
There's no updates in f30 indeed, but updates-testing should be there and available for testing. Nearer release we will enable updates and if we didn't enable this for them now we might well not remember to do so, so it seemed like a good idea to just do them both.
kevin
On Sun, Mar 31, 2019 at 6:01 AM Kevin Fenzi kevin@scrye.com wrote:
On 3/30/19 9:50 PM, Peter Robinson wrote:
Great, thanks! I'll be keeping an eye on the composes to see if there are any issues.
Wasn't this disabled in the main Fedora branched compose? If so why would we want to enable it only on updates?
There's no updates in f30 indeed, but updates-testing should be there and available for testing. Nearer release we will enable updates and if we didn't enable this for them now we might well not remember to do so, so it seemed like a good idea to just do them both.
I was referring to commits 6c392f16 and 96adf9a in pungi-fedora, if it's disabled in the base fedora repo why enable it in updates/testing?
On Sun, 2019-03-31 at 05:13 +0000, Peter Robinson wrote:
On Sun, Mar 31, 2019 at 6:01 AM Kevin Fenzi kevin@scrye.com wrote:
On 3/30/19 9:50 PM, Peter Robinson wrote:
Great, thanks! I'll be keeping an eye on the composes to see if there are any issues.
Wasn't this disabled in the main Fedora branched compose? If so why would we want to enable it only on updates?
There's no updates in f30 indeed, but updates-testing should be there and available for testing. Nearer release we will enable updates and if we didn't enable this for them now we might well not remember to do so, so it seemed like a good idea to just do them both.
I was referring to commits 6c392f16 and 96adf9a in pungi-fedora, if it's disabled in the base fedora repo why enable it in updates/testing?
Hey Peter, the zchunk metadata generation was disabled in the base repo because of a bug that popped up in a combination that the compose process happened to hit: using a single baseurl and downloading a zchunk file with tens of thousands of chunks on a slow processor.
The bug has been fixed with updates to both zchunk and libcurl (see https://bugzilla.redhat.com/show_bug.cgi?id=1690971) and it shouldn't affect beta users because the number of chunks in updates and updates- testing is a magnitude lower than the base repo.
*However*
Due to an unrelated *major* bug in the latest librepo update ( https://bugzilla.redhat.com/show_bug.cgi?id=1694411), I'd like to request that we disable zchunk metadata generation in updates and updates-testing until it's fixed.
Jonathan
On Sun, 2019-03-31 at 09:09 +0100, Jonathan Dieter wrote:
Due to an unrelated *major* bug in the latest librepo update ( https://bugzilla.redhat.com/show_bug.cgi?id=1694411), I'd like to request that we disable zchunk metadata generation in updates and updates-testing until it's fixed.
Just to be clear, until either: * We get a new updates compose out without zchunk metadata, or * The user sets zchunk=False in /etc/dnf/dnf.conf
dnf update is broken for anybody using F30
Should I send an email to -devel explaining the above?
Jonathan
On 3/31/19 1:56 AM, Jonathan Dieter wrote:
On Sun, 2019-03-31 at 09:09 +0100, Jonathan Dieter wrote:
Due to an unrelated *major* bug in the latest librepo update ( https://bugzilla.redhat.com/show_bug.cgi?id=1694411), I'd like to request that we disable zchunk metadata generation in updates and updates-testing until it's fixed.
Just to be clear, until either:
- We get a new updates compose out without zchunk metadata, or
- The user sets zchunk=False in /etc/dnf/dnf.conf
dnf update is broken for anybody using F30
Should I send an email to -devel explaining the above?
Please do, perhaps devel-announce ?
I have reverted things and am working on a new f30-updates-testing push. There was a failed f29-updates-testing last night so I have to finish that first, but hopefully we will have it out in a few hours.
kevin
On Sun, 2019-03-31 at 10:28 -0700, Kevin Fenzi wrote:
On 3/31/19 1:56 AM, Jonathan Dieter wrote:
On Sun, 2019-03-31 at 09:09 +0100, Jonathan Dieter wrote:
Due to an unrelated *major* bug in the latest librepo update ( https://bugzilla.redhat.com/show_bug.cgi?id=1694411), I'd like to request that we disable zchunk metadata generation in updates and updates-testing until it's fixed.
Just to be clear, until either:
- We get a new updates compose out without zchunk metadata, or
- The user sets zchunk=False in /etc/dnf/dnf.conf
dnf update is broken for anybody using F30
Should I send an email to -devel explaining the above?
Please do, perhaps devel-announce ?
I have reverted things and am working on a new f30-updates-testing push. There was a failed f29-updates-testing last night so I have to finish that first, but hopefully we will have it out in a few hours.
I've sent it out to devel-announce, but it was rejected as I'm not in the right group. Will I send it to you and let you forward it?
Jonathan
On 3/31/19 10:35 AM, Jonathan Dieter wrote:
On Sun, 2019-03-31 at 10:28 -0700, Kevin Fenzi wrote:
On 3/31/19 1:56 AM, Jonathan Dieter wrote:
On Sun, 2019-03-31 at 09:09 +0100, Jonathan Dieter wrote:
Due to an unrelated *major* bug in the latest librepo update ( https://bugzilla.redhat.com/show_bug.cgi?id=1694411), I'd like to request that we disable zchunk metadata generation in updates and updates-testing until it's fixed.
Just to be clear, until either:
- We get a new updates compose out without zchunk metadata, or
- The user sets zchunk=False in /etc/dnf/dnf.conf
dnf update is broken for anybody using F30
Should I send an email to -devel explaining the above?
Please do, perhaps devel-announce ?
I have reverted things and am working on a new f30-updates-testing push. There was a failed f29-updates-testing last night so I have to finish that first, but hopefully we will have it out in a few hours.
I've sent it out to devel-announce, but it was rejected as I'm not in the right group. Will I send it to you and let you forward it?
Yeah, you have to be subscribed to devel-announce to post there... if you just subscribe and resend it should go to moderation and I can pass it.
Or if you want, just send it my way and I can post it...
kevin
On Sun, 2019-03-31 at 10:37 -0700, Kevin Fenzi wrote:
On 3/31/19 10:35 AM, Jonathan Dieter wrote:
On Sun, 2019-03-31 at 10:28 -0700, Kevin Fenzi wrote:
On 3/31/19 1:56 AM, Jonathan Dieter wrote:
On Sun, 2019-03-31 at 09:09 +0100, Jonathan Dieter wrote:
Due to an unrelated *major* bug in the latest librepo update ( https://bugzilla.redhat.com/show_bug.cgi?id=1694411), I'd like to request that we disable zchunk metadata generation in updates and updates-testing until it's fixed.
Just to be clear, until either:
- We get a new updates compose out without zchunk metadata, or
- The user sets zchunk=False in /etc/dnf/dnf.conf
dnf update is broken for anybody using F30
Should I send an email to -devel explaining the above?
Please do, perhaps devel-announce ?
I have reverted things and am working on a new f30-updates-testing push. There was a failed f29-updates-testing last night so I have to finish that first, but hopefully we will have it out in a few hours.
I've sent it out to devel-announce, but it was rejected as I'm not in the right group. Will I send it to you and let you forward it?
Yeah, you have to be subscribed to devel-announce to post there... if you just subscribe and resend it should go to moderation and I can pass it.
Or if you want, just send it my way and I can post it...
Ok, I've subscribed, sent the message, and it's awaiting moderation.
Jonathan
infrastructure@lists.fedoraproject.org
-
Jonathan Dieter -
Kevin Fenzi -
Peter Robinson -
Stephen John Smoogen