We voted today on
"Build scripts of packages (%prep, %build, %install and %check) may only alter files (create, modify, delete) under %{buildroot}, %{_builddir} and valid temporary locations like /tmp, /var/tmp (or $TMPDIR or %{_tmppath} as set by the rpmbuild process).
Further clarification: That should hold true irrespective of the builder's uid"
But after thinking about it I'm not quite happy now. Since we go into details naming what the build scripts are, we should make clear that they are not equal in what they may or may not do. For example %{buildroot} should only be modified by %install, not %prep/%build and %check.
How about extending the rule and having a root/non-root rule, too, e.g.
o Package builds should yield the same results irrespective of the packaging process' uid/gid, for example builds under root and non-root users should behave the same.
o Build scripts of packages (%prep, %build, %install and %check) may only alter files (create, modify, delete) under %{buildroot}, %{_builddir} and valid temporary locations like /tmp, /var/tmp (or $TMPDIR or %{_tmppath} as set by the rpmbuild process). %{buildroot} should only be allowed to be altered in %install scripts.
Note I: The first part of this rule is automatically fulfilled for typical non-user build process uids but the packager should not rely on that, since users may rebuild the src.rpm under root
Note II: As a consequence $HOME and similar parts of the filesystem are not to be used directly. Of course some of the allowed write spaces like the builddir, buildroot or $TMPDIR may have been placed under $HOME, so indirectly a user may be writing under $HOME, but direct access to parts under $HOME are strictly forbidden.
Note III: Cheating with relative paths (".." escapes) grants you a ticket to packaging hell.
On Thu, 2006-10-12 at 20:20 +0200, Axel Thimm wrote:
How about extending the rule and having a root/non-root rule, too, e.g.
Well, keep in mind that the buildsystem is building everything as non-root, so any "root" specific rules are not terribly valuable to Fedora (but are almost certainly good practices for packaging in general).
o Package builds should yield the same results irrespective of the packaging process' uid/gid, for example builds under root and non-root users should behave the same.
While this is accurate, if we add it, it should be a SHOULD, and not a MUST. It is a SHOULD in Axel's wording, but I just want to emphasize the point. :)
o Build scripts of packages (%prep, %build, %install and %check) may only alter files (create, modify, delete) under %{buildroot}, %{_builddir} and valid temporary locations like /tmp, /var/tmp (or $TMPDIR or %{_tmppath} as set by the rpmbuild process). %{buildroot} should only be allowed to be altered in %install scripts.
Note I: The first part of this rule is automatically fulfilled for typical non-user build process uids but the packager should not rely on that, since users may rebuild the src.rpm under root
But we really don't want them too. Heck, I would wholeheartedly support patching rpm to complain when rpmbuild is being run by UID 0. :) Generally, people rebuilding packages as root don't know what they're doing. These are the same sort of people who are liable to download SRPMS willy-nilly, from any possible source (and for any possible distribution) and rebuild them. So even if Fedora sets the bar high for quality here, we still don't want people to think that rebuilding SRPMS as root is a recommended or safe idea.
Note II: As a consequence $HOME and similar parts of the filesystem are not to be used directly. Of course some of the allowed write spaces like the builddir, buildroot or $TMPDIR may have been placed under $HOME, so indirectly a user may be writing under $HOME, but direct access to parts under $HOME are strictly forbidden.
Agreed. If an end-user wants to set their variables to brain-dead locations, then they deserve whatever pain they get.
Note III: Cheating with relative paths (".." escapes) grants you a ticket to packaging hell.
This really ought to be a MUST. Using relative paths ("..") in anything involving a write operation should be actively discouraged. Perhaps even using it at all should be discouraged.
Example:
Version 1 of Package foo leaves some junk in a toplevel directory. When we run make install, we get this junk as part of a glob.
To keep the install clean (and minimize the amount of bleeding from the eyes of dealing with autotools), the packager does this:
cd .. rm -rf foo*
But, in version 2 of the Package foo, they get rid of these files for us, and incorporate the same command set in their build process, and leave us in that toplevel directory. Suddenly, we've dropped down a level and we're deleting all sorts of unexpected things. Hopefully, a packager would catch this before he sent it to the buildsystem, and yes, mock insulates us for this case (worst case, he kills himself and the build fails). But for the end-user, not rebuilding in mock, they may do more damage. Multiple relative paths ("../..") only make the damage potential worse.
Just food for thought at 35,000 feet over the Pacific.
~spot
On Thu, 2006-10-12 at 20:20 +0200, Axel Thimm wrote:
We voted today on
"Build scripts of packages (%prep, %build, %install and %check) may only alter files (create, modify, delete) under %{buildroot}, %{_builddir} and valid temporary locations like /tmp, /var/tmp (or $TMPDIR or %{_tmppath} as set by the rpmbuild process).
Further clarification: That should hold true irrespective of the builder's uid"
But after thinking about it I'm not quite happy now. Since we go into details naming what the build scripts are, we should make clear that they are not equal in what they may or may not do. For example %{buildroot} should only be modified by %install, not %prep/%build and %check.
Though I agree that the formulation could have been better, I do not agree upon your conclusion.
Remember, the intent behind all this was to say: "building an rpm must be free of side-effects on the hosting system.". We had tried to narrow this to file system operations ("alter files") to make this more understandable/handy to "Joe Occasional Builder".
I don't think we should try to further narrow this to "what to do when, and when is rpm allowed to do what". IMO, this is a completely different question and beyond the scope of the problem we had wanted to address.
How about extending the rule and having a root/non-root rule, too, e.g.
o Package builds should yield the same results irrespective of the packaging process' uid/gid, for example builds under root and non-root users should behave the same.
o Build scripts of packages (%prep, %build, %install and %check) may only alter files (create, modify, delete) under %{buildroot}, %{_builddir} and valid temporary locations like /tmp, /var/tmp (or $TMPDIR or %{_tmppath} as set by the rpmbuild process). %{buildroot} should only be allowed to be altered in %install scripts.
Technically, in some (rare) occasions, this last sentence is not applicable.
E.g. there exist packages, which want/need to be built "multi-staged", with %build containing (often: temporary) installs to %{buildroot}. In some (very rare) occasions, packages even require "building" inside of %buildroot.
The constraint you're adding above, would (IMO: unnecessarily) close out these packages from being packaged, or force packagers to resort to move "building" to %install.
Ralf
On Fri, Oct 13, 2006 at 06:06:11AM +0200, Ralf Corsepius wrote:
E.g. there exist packages, which want/need to be built "multi-staged", with %build containing (often: temporary) installs to %{buildroot}. In some (very rare) occasions, packages even require "building" inside of %buildroot.
These are exactly the broken packages that I want to cater with the proposed changes!
If you want to do staged installs during build time you *HAVE* to do so in builddir, not buildroot.
The constraint you're adding above, would (IMO: unnecessarily) close out these packages from being packaged, or force packagers to resort to move "building" to %install.
Nope, both ways are a sloppy way of packaging. It should be forbidden. Stage your builds in %build/%{builddir}, don't build in %install and don't touch %{buildroot} in %prep/%build. This should be carved in stone.
On Fri, 2006-10-13 at 09:33 +0200, Axel Thimm wrote:
On Fri, Oct 13, 2006 at 06:06:11AM +0200, Ralf Corsepius wrote:
E.g. there exist packages, which want/need to be built "multi-staged", with %build containing (often: temporary) installs to %{buildroot}. In some (very rare) occasions, packages even require "building" inside of %buildroot.
These are exactly the broken packages that I want to cater with the proposed changes!
There ain't anything broken with these packages ;)
They do not leave files around, nor do they do anything harmful. They simply do not fit into the constraints you are trying to set up.
If you want to do staged installs during build time you *HAVE* to do so in builddir, not buildroot.
Why? rpmbuild and the spec have full access to both directories and can read/write to both.
Whether a package does a
%build make something make install-something DESTDIR=$RPM_BUILD_ROOT export LD_LIBRARY_PATH=$RPM_BUILD_ROOT%{_lib} make all %install rm -rf $RPM_BUILD_ROOT make install DESTDIR=$RPM_BUILD_ROOT
or a
%build make something make install-something DESTDIR=$(PWD)/tmp export LD_LIBRARY_PATH=(PWD)/tmp%{_lib} make all %install rm -rf $(PWD)/tmp rm -rf $RPM_BUILD_ROOT make install DESTDIR=$RPM_BUILD_ROOT
doesn't make a real difference.
To the contrary, the later is sightly more error-prone.
The constraint you're adding above, would (IMO: unnecessarily) close out these packages from being packaged, or force packagers to resort to move "building" to %install.
Nope, both ways are a sloppy way of packaging. It should be forbidden. Stage your builds in %build/%{builddir}, don't build in %install and don't touch %{buildroot} in %prep/%build. This should be carved in stone.
<sigh/> IMO, you are trying to overengineer something.
Ralf
On Fri, Oct 13, 2006 at 10:25:36AM +0200, Ralf Corsepius wrote:
On Fri, 2006-10-13 at 09:33 +0200, Axel Thimm wrote:
On Fri, Oct 13, 2006 at 06:06:11AM +0200, Ralf Corsepius wrote:
E.g. there exist packages, which want/need to be built "multi-staged", with %build containing (often: temporary) installs to %{buildroot}. In some (very rare) occasions, packages even require "building" inside of %buildroot.
These are exactly the broken packages that I want to cater with the proposed changes!
There ain't anything broken with these packages ;)
You're entitled to think so, yes. But they are broken nonetheless. :)
They do not leave files around, nor do they do anything harmful. They simply do not fit into the constraints you are trying to set up.
Same is true for a package doing everything in %prep. Does that justify it just the same? And these are not constraints *I* am trying to set up, this is how rpm was designed to be used, the constraints are just to make sure it is used that way.
If you want to do staged installs during build time you *HAVE* to do so in builddir, not buildroot.
Why? rpmbuild and the spec have full access to both directories and can read/write to both.
rpmbuild has access to a lot of other directories, the guidelines are there to restrict this access to the sane set of directories.
Nope, both ways are a sloppy way of packaging. It should be forbidden. Stage your builds in %build/%{builddir}, don't build in %install and don't touch %{buildroot} in %prep/%build. This should be carved in stone.
<sigh/> IMO, you are trying to overengineer something.
Well, in your opinion. That's the nice thing about a democratic institution, everybody may have an opinion and need not agree with the other. Still if enough opinions are gathered we'll have a functional guideline.
Le Ven 13 octobre 2006 10:33, Axel Thimm a écrit :
On Fri, Oct 13, 2006 at 10:25:36AM +0200, Ralf Corsepius wrote:
They do not leave files around, nor do they do anything harmful. They simply do not fit into the constraints you are trying to set up.
Same is true for a package doing everything in %prep. Does that justify it just the same? And these are not constraints *I* am trying to set up, this is how rpm was designed to be used, the constraints are just to make sure it is used that way.
While some packagers may manipulate directories outside of the intended rpm section safely, most people who do so just do not understand the risks and usually end up with broken packages. (installing stuff before a script changes it, installing a mix of production and build files and so on).
Also a spec which does not respect common style is more difficult to review or salvage later if the maintainer went awol.
I'm all for Axel's proposal, except I'd also allow other constructs provided the corresponding commented template is posted in the wiki and approved by the board.
On Fri, 2006-10-13 at 10:33 +0200, Axel Thimm wrote:
On Fri, Oct 13, 2006 at 10:25:36AM +0200, Ralf Corsepius wrote:
On Fri, 2006-10-13 at 09:33 +0200, Axel Thimm wrote:
On Fri, Oct 13, 2006 at 06:06:11AM +0200, Ralf Corsepius wrote:
E.g. there exist packages, which want/need to be built "multi-staged", with %build containing (often: temporary) installs to %{buildroot}. In some (very rare) occasions, packages even require "building" inside of %buildroot.
These are exactly the broken packages that I want to cater with the proposed changes!
There ain't anything broken with these packages ;)
Then I can't avoid replying a bit clearer:
* The issue you are trying to address is not related at all to our original problem ("free of side-effects")
* Your proposal does not solve an actual technical problem, to the contrary, it artificially introduces new ones.
Ralf
On Fri, Oct 13, 2006 at 10:56:39AM +0200, Ralf Corsepius wrote:
On Fri, 2006-10-13 at 10:33 +0200, Axel Thimm wrote:
On Fri, Oct 13, 2006 at 10:25:36AM +0200, Ralf Corsepius wrote:
On Fri, 2006-10-13 at 09:33 +0200, Axel Thimm wrote:
On Fri, Oct 13, 2006 at 06:06:11AM +0200, Ralf Corsepius wrote:
E.g. there exist packages, which want/need to be built "multi-staged", with %build containing (often: temporary) installs to %{buildroot}. In some (very rare) occasions, packages even require "building" inside of %buildroot.
These are exactly the broken packages that I want to cater with the proposed changes!
There ain't anything broken with these packages ;)
Then I can't avoid replying a bit clearer:
- The issue you are trying to address is not related at all to our
original problem ("free of side-effects")
So? Is that the only problem we are interested in solving in this group? What kind of childish argument is that? Let's become constructive again.
- Your proposal does not solve an actual technical problem, to the
contrary, it artificially introduces new ones.
As said, you're entitled to your opinion. As well as others like myself are entitled to the opinion that packages writing into %{buildroot} at any other stage that %install are broken.
If you continue to stiffly argue on technical grounds we'll end up doing all in %prep. There is no technical reason not to do everything there, right? No side-effects, the binary results are the same and so on. You'll probably start removing comments next, since they is no technically needed. Greetings from homo faber.
Just going into hyperboles to make the point since IMO it is a very trivial point that shouldn't even need discussion. There are reasons building and installing software have been seperated in all build mechanisms, make, rpm, ant, cmake, etc, and arguing that it's technically not neccessary is nonsense.
On Fri, 2006-10-13 at 11:14 +0200, Axel Thimm wrote:
On Fri, Oct 13, 2006 at 10:56:39AM +0200, Ralf Corsepius wrote:
On Fri, 2006-10-13 at 10:33 +0200, Axel Thimm wrote:
On Fri, Oct 13, 2006 at 10:25:36AM +0200, Ralf Corsepius wrote:
On Fri, 2006-10-13 at 09:33 +0200, Axel Thimm wrote:
On Fri, Oct 13, 2006 at 06:06:11AM +0200, Ralf Corsepius wrote:
E.g. there exist packages, which want/need to be built "multi-staged", with %build containing (often: temporary) installs to %{buildroot}. In some (very rare) occasions, packages even require "building" inside of %buildroot.
These are exactly the broken packages that I want to cater with the proposed changes!
There ain't anything broken with these packages ;)
Then I can't avoid replying a bit clearer:
- The issue you are trying to address is not related at all to our
original problem ("free of side-effects")
So? Is that the only problem we are interested in solving in this group? What kind of childish argument is that? Let's become constructive again.
Pardon, in which world are you living? Being opposed to a proposal is considered to be destructive?
I am saying your extension to the proposal doesn't solve any problem, it introduces NEW problems, that's why I am opposed to it.
- Your proposal does not solve an actual technical problem, to the
contrary, it artificially introduces new ones.
As said, you're entitled to your opinion. As well as others like myself are entitled to the opinion that packages writing into %{buildroot} at any other stage that %install are broken.
If you continue to stiffly argue on technical grounds we'll end up doing all in %prep. There is no technical reason not to do everything there, right? No side-effects, the binary results are the same and so on. You'll probably start removing comments next, since they is no technically needed.
Right!
The %prep/%build/%install separation is an artificial separation based on the assumption that all packages have a build model supporting this 3-staged building model.
Reality is: Most GNU packages do, but this doesn't apply in general. Not all packages have a "natural separation" fitting into this model.
There are packages for which %prep/%build/%install can be collapsed into one step (e.g. by simply untarring into %RPM_BUILD_ROOT[1], instead of copying them around through the different stage), there are others for which %build consists of several stages with installation of subpackages into temporary locations.
Ralf
[1] Very handy and effective for packages consisting of several 10MBs.
On Fri, 13 Oct 2006 10:25:36 +0200, Ralf Corsepius wrote:
Whether a package does a
%build make something make install-something DESTDIR=$RPM_BUILD_ROOT export LD_LIBRARY_PATH=$RPM_BUILD_ROOT%{_lib} make all %install rm -rf $RPM_BUILD_ROOT make install DESTDIR=$RPM_BUILD_ROOT
or a
%build make something make install-something DESTDIR=$(PWD)/tmp export LD_LIBRARY_PATH=(PWD)/tmp%{_lib} make all %install rm -rf $(PWD)/tmp rm -rf $RPM_BUILD_ROOT make install DESTDIR=$RPM_BUILD_ROOT
doesn't make a real difference.
It does. ;)
The former installs into $RPM_BUILD_ROOT without doing "rm -rf $RPM_BUILD_ROOT" first. It would need to do this at the beginning of %build (!).
Rather than $RPM_BUILD_DIR I would also choose $RPM_BUILD_ROOT as a temporary working directory, because it is emptied at the beginning of %install again anyway and is specific to the current package.
$RPM_BUILD_DIR or %_builddir is not. It is %{_topdir}/BUILD by default and hence is located above the extracted source tarballs. Usually, you would prefer $(pwd) which expands to $RPM_BUILD_ROOT or below. Or append a unique string to %_builddir before working in there.
We want rpmbuild --short-circuit builds to work for both %build and %install, regardless of how that is achieved.
If it is necessary to create tmp files in %check, don't impose too many restrictions on the packagers.
On Fri, 2006-10-13 at 14:11 +0200, Michael Schwendt wrote:
On Fri, 13 Oct 2006 10:25:36 +0200, Ralf Corsepius wrote:
Whether a package does a
%build make something make install-something DESTDIR=$RPM_BUILD_ROOT export LD_LIBRARY_PATH=$RPM_BUILD_ROOT%{_lib} make all %install rm -rf $RPM_BUILD_ROOT make install DESTDIR=$RPM_BUILD_ROOT
or a
%build make something make install-something DESTDIR=$(PWD)/tmp export LD_LIBRARY_PATH=(PWD)/tmp%{_lib} make all %install rm -rf $(PWD)/tmp rm -rf $RPM_BUILD_ROOT make install DESTDIR=$RPM_BUILD_ROOT
doesn't make a real difference.
It does. ;)
The former installs into $RPM_BUILD_ROOT without doing "rm -rf $RPM_BUILD_ROOT" first. It would need to do this at the beginning of %build (!).
hastly off-head scribbled pseudo-code", you know?
The real world case in FE, I am facing this issue with, does have an rm -rf $RPM_BUILD_ROOT in %build ;)
Ralf
On Fri, Oct 13, 2006 at 02:27:51PM +0200, Ralf Corsepius wrote:
On Fri, 2006-10-13 at 14:11 +0200, Michael Schwendt wrote:
doesn't make a real difference.
It does. ;)
The former installs into $RPM_BUILD_ROOT without doing "rm -rf $RPM_BUILD_ROOT" first. It would need to do this at the beginning of %build (!).
hastly off-head scribbled pseudo-code", you know?
The real world case in FE, I am facing this issue with, does have an rm -rf $RPM_BUILD_ROOT in %build ;)
And you consider that proper? This package should not be allowed to pass the review!
*Shudder*
On Fri, 2006-10-13 at 15:58 +0200, Axel Thimm wrote:
On Fri, Oct 13, 2006 at 02:27:51PM +0200, Ralf Corsepius wrote:
On Fri, 2006-10-13 at 14:11 +0200, Michael Schwendt wrote:
doesn't make a real difference.
It does. ;)
The former installs into $RPM_BUILD_ROOT without doing "rm -rf $RPM_BUILD_ROOT" first. It would need to do this at the beginning of %build (!).
hastly off-head scribbled pseudo-code", you know?
The real world case in FE, I am facing this issue with, does have an rm -rf $RPM_BUILD_ROOT in %build ;)
And you consider that proper? This package should not be allowed to pass the review!
Why? What is the problem?
Ralf
packaging@lists.fedoraproject.org
-
Axel Thimm -
Michael Schwendt -
Nicolas Mailhot -
Ralf Corsepius -
Tom Callaway