Le 2020-03-02 14:45, clime a écrit :
On Mon, 2 Mar 2020 at 12:05, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2020-03-01 02:31, clime a écrit :
On Sat, 29 Feb 2020 at 21:50, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 20:30 +0100, clime a écrit :
Putting %{dynrel} reconciliation in the rpmbuild -bs stage using detached file state means that fedpkg local (or checking out git state and building in mock or copr or OBS or via plain rpmbuild -bs) will give you the same result as launching fedpkg build.
Well, I believe it doesn't. You run:
- fedpkg local -> produces <somepkg>-1.0-1.fc32 (1 in release because
you haven't built that package before) 2) fedpkg build
At that point state is undefined till the build succeeds or not. If the build succeeds, the buildsystem will write back a new state. Let’s assume it succeeds.
It's undefined if you add more elements to the equation than necessary (i.e. build system), otherwise it would be well-defined in this case.
- vim <somepkg>.spec and do some change in %description
- fedpkg commit -m "description improvement"
- fedpkg local -> produces <somepkg>-1.0-1.fc32
- fedpkg push -> error because build system pushed meanwhile
Yes, here the packager notices something else has been going on, and he needs to merge or rebase. He’d have the same effect if another packager had been doing stuff on the package, or a mass rebuild had been going on. That’s the distributed decentralized aspect of git, except here the packager collided with himself by starting two work threads in parallel (one buildsys-side, one local).
The problem is that you launched some process and you need to wait until it finishes, normally you don't need to anything like that when working with git.
But fedpkg build is a release to production process. It’s not a dev staging process. In release processes, actually doing the release is not an inconvenient optional check.
You're asked to autobump a “Release” field. You’re asked to automate a changelog which is effectively a release changelog, not dev changelog (the dev changelog lives in git, people want a good changelog in rpm proper to track release not git history).
The *single* *only* reason good release numbers and good changelog matter is because the result will be released to third parties. koshei, or scratch builds, or whatever, do not care about those.
A normal git flow would lock the branch at release time to let the release manager do his releasing (merge window closed in Linus’s terms). That’s option 1 I gave you.
Alternatively, you can use option 2, the “I feel lucky” approach, let everything open, and abort the release if merge-back failed. A packager that changes his package before the production build is finished will probably want it canceled anyway (what is the point of risking breaking builds of other packages by pushing an unfinished package to the shared koji buildroot)?
Either option does not stop the packager to do anything he wants in his local branches. They just require him to merge/rebase when syncing with the centralized master Fedora state. That’s a core element of git architecture, be it in Fedora or elsewhere.
You can not avoid a buildsys merge-back when doing production builds. The merge-back can be explicit and automatic (as I proposed), or you can rely on informal meatware for it, but it will exist. Since the whole point of the request is to lower dependence on meatware, why on hell would you want to keep meatware as part of the whole process?
The only “cost” for the packager is to wait for his build to finish before continuing to change the package (or merge/rebase). That’s not an horrific cost. No one forced the packager to start a build before he was ready. No one forced the packager to perform a production build instead of a scratch build if he just wanted to check partial changes. If shit happens, and he realizes before the build end something is not ok, aborting the build on failed merge back is helping the packager.
Or am I missing something? “normally you don't need to anything like that when working with git” is not a clear technical point.
Regards,
On Tue, 3 Mar 2020 at 09:22, Nicolas Mailhot nicolas.mailhot@laposte.net wrote:
Le 2020-03-02 14:45, clime a écrit :
On Mon, 2 Mar 2020 at 12:05, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2020-03-01 02:31, clime a écrit :
On Sat, 29 Feb 2020 at 21:50, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 20:30 +0100, clime a écrit :
Putting %{dynrel} reconciliation in the rpmbuild -bs stage using detached file state means that fedpkg local (or checking out git state and building in mock or copr or OBS or via plain rpmbuild -bs) will give you the same result as launching fedpkg build.
Well, I believe it doesn't. You run:
- fedpkg local -> produces <somepkg>-1.0-1.fc32 (1 in release because
you haven't built that package before) 2) fedpkg build
At that point state is undefined till the build succeeds or not. If the build succeeds, the buildsystem will write back a new state. Let’s assume it succeeds.
It's undefined if you add more elements to the equation than necessary (i.e. build system), otherwise it would be well-defined in this case.
- vim <somepkg>.spec and do some change in %description
- fedpkg commit -m "description improvement"
- fedpkg local -> produces <somepkg>-1.0-1.fc32
- fedpkg push -> error because build system pushed meanwhile
Yes, here the packager notices something else has been going on, and he needs to merge or rebase. He’d have the same effect if another packager had been doing stuff on the package, or a mass rebuild had been going on. That’s the distributed decentralized aspect of git, except here the packager collided with himself by starting two work threads in parallel (one buildsys-side, one local).
The problem is that you launched some process and you need to wait until it finishes, normally you don't need to anything like that when working with git.
But fedpkg build is a release to production process. It’s not a dev staging process. In release processes, actually doing the release is not an inconvenient optional check.
Well, you could have a situation when somebody wants to immediately continue working after making a release. Probably rare but could happen and if build time of a package is long (libreOffice/firefox), this could hit you as an inconvenience. Or do you want to push-back immediately after srpm build when you don't yet know whether the build will succeed? That wasn't clear to me before.
You're asked to autobump a “Release” field. You’re asked to automate a changelog which is effectively a release changelog, not dev changelog (the dev changelog lives in git, people want a good changelog in rpm proper to track release not git history).
I am not suggesting to use raw git commit messages for the changelog but instead content of annotated tags which can be initially prepopulated by commit messages but can be then edited to any extent and even the way the edit window for annotated tag is prepopulated might be configurable (to .e.g. load content from a file in which case you would probably skip the editing part). And I see the annotated tags as the actual releases.
From https://git-scm.com/docs/git-tag: Annotated tags are meant for release while lightweight tags are meant for private or temporary object labels.
They were designed to represent releases and because we are using git I don't see why we should avoid using annotated tags for releases.
On src.fp.o, there is also already support for this: e.g. https://src.fedoraproject.org/rpms/2ping/releases
The release needs building (koji), and distribution (bodhi + mirrors) but that's stuff that needs to be done with the release. Scratch builds can be used to make sure that the package builds and in the worst case, a new release needs to be done that accounts for changed build environment. But I don't think we need to throw the whole concept of annotated tags = releases because of it.
Of course, everything here can be made opt-in. As an approach somebody likes and somebody doesn't.
I think this is where our approaches really differ, you see release as something that happens somewhere else than in git, but I see the release as something that happens in git. And afterwards we need to get that released content to users.
I think we will never be able to reconciliate our views because they are different from the very base. In the end, what matters is implementation complexity and ease of use. I can present to you the things that need to be done for this to work in a detailed way (although it would take a bit of time to put into a document) but I also know it is a doable and it is actually fairly simple when not accounting for time spending on pull requests and getting them accepted. Your approach seems to be intuitively difficult to implement, either locking implementation (i am sure there would be quite unpleasant cases when the repo just stays locked even if it should be already unlocked, at least initially) or the fail after repo change - koji would need to listen for changes in dist-git and then abort builds based on that, which is doable but then you need authentication between the two systems (i.e koji needs to have write access to dist-git which is probably not the case today because the releng script to do mass rebuilds works separately and only invokes builds in koji) and you need some job that gets parsed changelog and NVR and does the pushing back to dist-git, which needs to be strictly separated from srpm/rpm builds due to security. Again, quite fairly complex to introduce something like that into a system like koji. Maybe you would like to bypass koji and create some external service that listens for dist-git and koji events and it is doing what you describe but introduction of this service is again quite a big task especially if one needs access to write to dist-git. And i think I would be able to find out more smaller or bigger obstacles when actually really trying to evaluate how difficult this would be.
The thing is that the I am not that much of a fan of "push into dist-git can make build fail" - it seems to me like something nobody expects so I don't want to be spending time figuring this out so that I can give you arguments that would be acceptable for you. If, however, you will spend time on "what piece of code need to be inserted where" in the infrastructure, I'll spend time on it as well for what I am proposing and we can evaluate what would be easier.
I am basically suggesting a small preprocessing step before sprm build. Cloning dist-git repo is already somewhere there in Koji, there just needs to be one more command invocation before everything continues the same way as before. No write access do dist-git needed which is already a big relief.
The *single* *only* reason good release numbers and good changelog matter is because the result will be released to third parties. koshei, or scratch builds, or whatever, do not care about those.
I agree they matter.
A normal git flow would lock the branch at release time to let the release manager do his releasing (merge window closed in Linus’s terms). That’s option 1 I gave you.
Alternatively, you can use option 2, the “I feel lucky” approach, let everything open, and abort the release if merge-back failed. A packager that changes his package before the production build is finished will probably want it canceled anyway (what is the point of risking breaking builds of other packages by pushing an unfinished package to the shared koji buildroot)?
Either option does not stop the packager to do anything he wants in his local branches. They just require him to merge/rebase when syncing with the centralized master Fedora state. That’s a core element of git architecture, be it in Fedora or elsewhere.
You can not avoid a buildsys merge-back when doing production builds.
Merge backs by build system can be avoided, however. Why do you think they can't be avoided?
There is the case of mass rebuilds but this is pretty much a one-shot event for each branch and it's not a merge-back we are talking about here....I would need to see the implementation today to be sure but in F12, it used be a separate script doing this: https://fedoraproject.org/wiki/Fedora_12_Mass_Rebuild
The merge-back can be explicit and automatic (as I proposed), or you can rely on informal meatware for it, but it will exist. Since the whole point of the request is to lower dependence on meatware, why on hell would you want to keep meatware as part of the whole process?
To me the whole point of the request is "make packager's life easier". If it goes hand-to-hand with automation (which it does by definition), then even better.
The only “cost” for the packager is to wait for his build to finish before continuing to change the package (or merge/rebase). That’s not an horrific cost. No one forced the packager to start a build before he was ready. No one forced the packager to perform a production build instead of a scratch build if he just wanted to check partial changes. If shit happens, and he realizes before the build end something is not ok, aborting the build on failed merge back is helping the packager.
Or am I missing something? “normally you don't need to anything like that when working with git” is not a clear technical point.
It's rather a point about what people are used to and what should be taken into account as well. If we want more projects on src.fp.o (wouldn't be bad, right?), then we shouldn't introduce an unexpected complexity that people would need to learn about in probably not that pleasant manner (unless they read dist-git documentation or there is some banner somewhere saying "don't push while building").
Best regards! clime
Regards,
-- Nicolas Mailhot _______________________________________________ packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject....
Le 2020-03-03 15:14, clime a écrit :
On Tue, 3 Mar 2020 at 09:22, Nicolas Mailhot nicolas.mailhot@laposte.net wrote:
Le 2020-03-02 14:45, clime a écrit :
On Mon, 2 Mar 2020 at 12:05, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2020-03-01 02:31, clime a écrit :
On Sat, 29 Feb 2020 at 21:50, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 20:30 +0100, clime a écrit :
Putting %{dynrel} reconciliation in the rpmbuild -bs stage using detached file state means that fedpkg local (or checking out git state and building in mock or copr or OBS or via plain rpmbuild -bs) will give you the same result as launching fedpkg build.
Well, I believe it doesn't. You run:
- fedpkg local -> produces <somepkg>-1.0-1.fc32 (1 in release because
you haven't built that package before) 2) fedpkg build
At that point state is undefined till the build succeeds or not. If the build succeeds, the buildsystem will write back a new state. Let’s assume it succeeds.
It's undefined if you add more elements to the equation than necessary (i.e. build system), otherwise it would be well-defined in this case.
- vim <somepkg>.spec and do some change in %description
- fedpkg commit -m "description improvement"
- fedpkg local -> produces <somepkg>-1.0-1.fc32
- fedpkg push -> error because build system pushed meanwhile
Yes, here the packager notices something else has been going on, and he needs to merge or rebase. He’d have the same effect if another packager had been doing stuff on the package, or a mass rebuild had been going on. That’s the distributed decentralized aspect of git, except here the packager collided with himself by starting two work threads in parallel (one buildsys-side, one local).
The problem is that you launched some process and you need to wait until it finishes, normally you don't need to anything like that when working with git.
But fedpkg build is a release to production process. It’s not a dev staging process. In release processes, actually doing the release is not an inconvenient optional check.
Well, you could have a situation when somebody wants to immediately continue working after making a release. Probably rare but could happen and if build time of a package is long (libreOffice/firefox), this could hit you as an inconvenience. Or do you want to push-back immediately after srpm build when you don't yet know whether the build will succeed? That wasn't clear to me before.
Either you lock the centralized branch while the build is ongoing or you fail the build if someone pushes to the centralized branch before the end of the build.
That does not stop packagers from preparing the next stage in a local branch, only changing the state of the centralized branch at the same time the build process is changing it (two conflicting release state changes, option 1: buildsystem wins, option 2, packager wins, can’t have an heisenstate where things are both released and not released).
I am not suggesting to use raw git commit messages for the changelog but instead content of annotated tags which can be initially prepopulated by commit messages but can be then edited to any extent and even the way the edit window for annotated tag is prepopulated might be configurable (to .e.g. load content from a file in which case you would probably skip the editing part). And I see the annotated tags as the actual releases.
The only actual release is the package built in koji. So, no matter what mechanism you use git side: – you must represent the real state of package builds and not a git-only fiction → that means a form of buildsys write-back because builds can and will fail – you must produce something packagers can also build in mock or copr or OBS or whatever → that means checkout-ing changelog data in a form that can be read by rpmbuild -bs → ie a file not git metadata → with the associated conflict risk if multiple changes occur between checkouts
You can not avoid a buildsys merge-back when doing production builds.
Merge backs by build system can be avoided, however. Why do you think they can't be avoided?
Because they’re the real state rpm changelogs records. Because that’s what people use rpm changelogs for. Something broke, what is the affected packaged, what where the released package states since last time it worked.
There is the case of mass rebuilds but this is pretty much a one-shot event
Mass rebuilds are not an exception, they’re becoming the norm. Every SIG that deals with modern software released as a huge number of interlinked components needs to perform SIG-level mass rebuilds all the time (directly in rawhide, in a side tag, whatever). koshei can now autobuild all dependent packages when one of their requirements changed. That’s where the project is going and has been going for several years.
All the modularity efforts in the past years have been justified by the need to find a way to mass build things.
Regards,
packaging@lists.fedoraproject.org