Hello,
an OpenSUSE packager made me aware of a race condition problem with a lot of fedora specs:
http://lists.opensuse.org/opensuse-packaging/2007-02/msg00005.html
A lot of specs and the template specs from rpmdev-newspec use following:
%install rm -rf $RPM_BUILD_ROOT # racing time mkdir -p $RPM_BUILD_ROOT/bla/foo
The solution is to make the following mandatory: %install rm -rf $RPM_BUILD_ROOT mkdir $RPM_BUILD_ROOT # this fails when $RPM_BUILD_ROOT already exists
Regards, Till
Till Maas wrote:
Hello,
an OpenSUSE packager made me aware of a race condition problem with a lot of fedora specs:
http://lists.opensuse.org/opensuse-packaging/2007-02/msg00005.html
A lot of specs and the template specs from rpmdev-newspec use following:
%install rm -rf $RPM_BUILD_ROOT # racing time
How is that a race exactly? rm doesn't exit/return until it is done, afaik.
-- Rex
On Mon, 2007-03-12 at 16:30 -0400, Bill Nottingham wrote:
Rex Dieter (rdieter@math.unl.edu) said:
How is that a race exactly? rm doesn't exit/return until it is done, afaik.
Someone could pre-make the build root in between the rm and mkdir calls.
Erm, ok. In the buildsystem, this should never happen (hooray mock), but when building on a multi-user system, I can see the remote possibility. However, we're talking about someone performing an operation in a very tiny gap. It's just as likely that they would manually replace files at any point in the process, or to argue that someone might rm -rf $RPM_BUILD_ROOT behind my back.
Basically, what I'm saying is that this "race" is so unlikely, I don't think we need to bother to go out of our way to prevent it.
It would be far easier for an attacker to leverage wildcarding in %files while a package is building, wait for it to perform make install, then slide in their malicious bits.
~spot
On Mo März 12 2007, Tom 'spot' Callaway wrote:
However, we're talking about someone performing an operation in a very tiny gap. It's just as likely that they would manually replace files at
This can be easily automated,
any point in the process, or to argue that someone might rm -rf $RPM_BUILD_ROOT behind my back.
while this normally is not possible with correct/normal file permissions.
Basically, what I'm saying is that this "race" is so unlikely, I don't think we need to bother to go out of our way to prevent it.
The fix is very easy, just add one line with mkdir.
It would be far easier for an attacker to leverage wildcarding in %files while a package is building, wait for it to perform make install, then slide in their malicious bits.
This is also not possible with normal file permissions,
Regards, Till
On Mon, Mar 12, 2007 at 04:05:58PM -0500, Tom 'spot' Callaway wrote:
On Mon, 2007-03-12 at 16:30 -0400, Bill Nottingham wrote:
Rex Dieter (rdieter@math.unl.edu) said:
How is that a race exactly? rm doesn't exit/return until it is done, afaik.
Someone could pre-make the build root in between the rm and mkdir calls.
Erm, ok. In the buildsystem, this should never happen (hooray mock), but when building on a multi-user system, I can see the remote possibility.
Hey, our new preferred buildroot makes it even harder to guess the Buildroot name, hooray2 ;)
It would be far easier for an attacker to leverage wildcarding in %files while a package is building, wait for it to perform make install, then slide in their malicious bits.
How would the attacker do that if the buildroot belongs to another user?
On Mo März 12 2007, Axel Thimm wrote:
On Mon, Mar 12, 2007 at 04:05:58PM -0500, Tom 'spot' Callaway wrote:
Erm, ok. In the buildsystem, this should never happen (hooray mock), but when building on a multi-user system, I can see the remote possibility.
Hey, our new preferred buildroot makes it even harder to guess the Buildroot name, hooray2 ;)
Does "grep RPM_BUILD_ROOT= /var/tmp/rpm-tmp.*" not work with the preferred buildroot?
Regards, Till
On Tue, Mar 13, 2007 at 09:35:16PM +0100, Till Maas wrote:
On Mo März 12 2007, Axel Thimm wrote:
On Mon, Mar 12, 2007 at 04:05:58PM -0500, Tom 'spot' Callaway wrote:
Erm, ok. In the buildsystem, this should never happen (hooray mock), but when building on a multi-user system, I can see the remote possibility.
Hey, our new preferred buildroot makes it even harder to guess the Buildroot name, hooray2 ;)
Does "grep RPM_BUILD_ROOT= /var/tmp/rpm-tmp.*" not work with the preferred buildroot?
The race between two rm/mkdir are about 50%. If you add a grep into one of them the balance will be strongly shifted in our favour, just try it.
The true solution is the mkdir w/o -p method, but it didn't pass the vote. :(
On Tue, Mar 13, 2007 at 10:48:55PM +0100, Till Maas wrote:
On Di März 13 2007, Axel Thimm wrote:
The race between two rm/mkdir are about 50%. If you add a grep into one of them the balance will be strongly shifted in our favour, just try it.
The grep needs only to be performed once before the race to "guess" the buildroot.
Yes, once, but in the right time window, which is when between when the scriplet is written to disk and being executed. So the attacker has to win two races, not only one, and the grep itself and subsequent text parsing takes more time than the script's rm/mkdir.
But this is all academic, try an attack and check the success rates, I'm sure they will be very low in the mktemp BuildRoot, even if you write the grep/sed stuff in C.
But they will be zero if we handle the race in the specfile, I'm not trying to play the true issue down.
On Mi März 14 2007, Axel Thimm wrote:
Yes, once, but in the right time window, which is when between when the scriplet is written to disk and being executed. So the attacker has to win two races, not only one, and the grep itself and subsequent text parsing takes more time than the script's rm/mkdir.
In the rpm-tmp files I have on my system, there is not only the install part in the file, but also the build part. So I assume that after the file is created and the attackers knows the buildroot, he has all the time until %build is finished, to prepare the race betwenn rm/mkdir in %install.
Regards, Till
On Wed, Mar 14, 2007 at 03:07:30PM +0100, Till Maas wrote:
On Mi März 14 2007, Axel Thimm wrote:
Yes, once, but in the right time window, which is when between when the scriplet is written to disk and being executed. So the attacker has to win two races, not only one, and the grep itself and subsequent text parsing takes more time than the script's rm/mkdir.
In the rpm-tmp files I have on my system, there is not only the install part in the file, but also the build part. So I assume that after the file is created and the attackers knows the buildroot, he has all the time until %build is finished, to prepare the race betwenn rm/mkdir in %install.
Ouch! Another reason why %build even knowing where %{buildroot} is, is bad.
So, indeed we need to fix this somehow (e.g. the rm/mkdir suggestion). Very nice thinking!
Axel.Thimm@ATrpms.net (Axel Thimm) writes:
But this is all academic, try an attack and check the success rates, I'm sure they will be very low in the mktemp BuildRoot,
I do not think that probabilities should be considered in security related discussions; "A little bit insecure" is like "a little bit pregnant" ;)
Attacker could start 100 of programs which are trying to exploit the race. This has the effect of:
* increasing the load which in turn enlarges the timeslot for the race * increasing the probability that the whole attack succeeds
Therefore: either let us fix this issue completely ('mkdir $RPM_BUILD_ROOT'), perhaps with reducing functionality in some use cases (e.g. %_tmpdir/%username/%name-... buildroot).
Moving the (secure) RPM_BUILD_ROOT initialization into the automatically generated rpm script (afair, recent (jbj-)rpm does already 'rm -rf $RPM_BUILD_ROOT' by default) should free packager from this task but will require changes to all spec files.
Or, assume/require that %_tmpdir is at a secure location. I think, nearly nobody reviews buildprocesses for security, but assumes that they happen in a trusted environment. Builds itself have to happen as a separate user; I do not want a broken Makefile which wipes my $HOME...
even if you write the grep/sed stuff in C.
The 'mktemp(1) - rm(1)' sequence is done in the shell, which calls dynamic linked binaries, which are doing lot of stuff before the first 'unlink(2)/rmdir(2)' is called.
In the meantime, attacker's C program has only to react on the inotify(7) event (e.g. return from select(2)) and has to readdir(2) /var/tmp. I think this is much faster than the 'mktemp(1) - rm(1)' sequence.
Enrico
On Wed, Mar 14, 2007 at 07:53:15PM +0100, Enrico Scholz wrote:
Therefore: either let us fix this issue completely ('mkdir $RPM_BUILD_ROOT'), perhaps with reducing functionality in some use cases (e.g. %_tmpdir/%username/%name-... buildroot).
Make no mistake: I'm all for doing so, we even voted yesterday, but the vote didn't went well.
On Wed, 2007-03-14 at 20:55 +0100, Axel Thimm wrote:
On Wed, Mar 14, 2007 at 07:53:15PM +0100, Enrico Scholz wrote:
Therefore: either let us fix this issue completely ('mkdir $RPM_BUILD_ROOT'), perhaps with reducing functionality in some use cases (e.g. %_tmpdir/%username/%name-... buildroot).
Make no mistake: I'm all for doing so, we even voted yesterday, but the vote didn't went well.
I voted against it, because a) all this is playing with symptoms. The real cause is inside of rpm and therefore should be fixed inside of rpm. b) this is all is not a real issue in practice.
Ralf
On Wed, Mar 14, 2007 at 10:35:51PM +0100, Ralf Corsepius wrote:
On Wed, 2007-03-14 at 20:55 +0100, Axel Thimm wrote:
On Wed, Mar 14, 2007 at 07:53:15PM +0100, Enrico Scholz wrote:
Therefore: either let us fix this issue completely ('mkdir $RPM_BUILD_ROOT'), perhaps with reducing functionality in some use cases (e.g. %_tmpdir/%username/%name-... buildroot).
Make no mistake: I'm all for doing so, we even voted yesterday, but the vote didn't went well.
I voted against it, because a) all this is playing with symptoms. The real cause is inside of rpm and therefore should be fixed inside of rpm.
That's as helpful as "The real cause of buffer overflows is that C doesn't handle them, so we should stop fixing them until #C++ double-O fixes them."
b) this is all is not a real issue in practice.
Well, you need to pick a side. On odd days in the calendar you require corner-case buildroot setup that allows mulitple users to builde xactly the same package with the same evr. On even days the same users will take a vow to never mess with each other's buildroots.
"Tom 'spot' Callaway" tcallawa@redhat.com writes:
Someone could pre-make the build root in between the rm and mkdir calls.
Erm, ok. In the buildsystem, this should never happen (hooray mock), but when building on a multi-user system, I can see the remote possibility. However, we're talking about someone performing an operation in a very tiny gap.
No; should be trivial to exploit with:
$ create-big-load & $ d=/var/tmp/foo-package-root-512 $ while test ! -e "$d"/bin/prog; do rm -rf "$d"; mkdir -m0777 -p "$d"/bin; done; \ rm -f "$d/bin/prog"; cp -a my-backdoored-prog "$d/bin/prog"
[ the while-loop should be implemented in C ]
Enrico
Till Maas opensource@till.name writes:
an OpenSUSE packager made me aware of a race condition problem with a lot of fedora specs:
http://lists.opensuse.org/opensuse-packaging/2007-02/msg00005.html
Was brought up already when $RPM_BUILD_ROOT was discussed....
The solution is to make the following mandatory: %install rm -rf $RPM_BUILD_ROOT mkdir $RPM_BUILD_ROOT # this fails when $RPM_BUILD_ROOT already exists
Will work; another solution would be to use a secure %_tmpdir (which is write and perhaps readable by the packager only). Packagers have to setup %_topdir already so this should happen for %_tmpdir too.
Enrico
On Tuesday 13 March 2007, Enrico Scholz wrote:
The solution is to make the following mandatory: %install rm -rf $RPM_BUILD_ROOT mkdir $RPM_BUILD_ROOT # this fails when $RPM_BUILD_ROOT already exists
Will work;
...but will break in setups where some subdirs of $RPM_BUILD_ROOT are missing before %install. This wouldn't suffer from that drawback:
%install rm -rf $RPM_BUILD_ROOT mkdir -p $(dirname $RPM_BUILD_ROOT) ; mkdir $RPM_BUILD_ROOT
ville.skytta@iki.fi (Ville Skyttä) writes:
%install rm -rf $RPM_BUILD_ROOT mkdir $RPM_BUILD_ROOT # this fails when $RPM_BUILD_ROOT already exists
Will work;
...but will break in setups where some subdirs of $RPM_BUILD_ROOT are missing before %install. This wouldn't suffer from that drawback:
%install rm -rf $RPM_BUILD_ROOT mkdir -p $(dirname $RPM_BUILD_ROOT) ; mkdir $RPM_BUILD_ROOT
... but opens a new attack vector because attacker could do
| mkdir -m777 -p $(dirname $RPM_BUILD_ROOT) | ... wait until victim executes the first 2 %install lines | mv $RPM_BUILD_ROOT $(dirname $RPM_BUILD_ROOT)/old-buildroot | mkdir $RPM_BUILD_ROOT
(easy to automate by some inotify in $(dirname $RPM_BUILD_ROOT))
Enrico
On Wed, Mar 14, 2007 at 01:41:40AM +0100, Enrico Scholz wrote:
ville.skytta@iki.fi (Ville Skyttä) writes:
%install rm -rf $RPM_BUILD_ROOT mkdir $RPM_BUILD_ROOT # this fails when $RPM_BUILD_ROOT already exists
Will work;
...but will break in setups where some subdirs of $RPM_BUILD_ROOT are missing before %install. This wouldn't suffer from that drawback:
%install rm -rf $RPM_BUILD_ROOT mkdir -p $(dirname $RPM_BUILD_ROOT) ; mkdir $RPM_BUILD_ROOT
... but opens a new attack vector because attacker could do
| mkdir -m777 -p $(dirname $RPM_BUILD_ROOT) | ... wait until victim executes the first 2 %install lines | mv $RPM_BUILD_ROOT $(dirname $RPM_BUILD_ROOT)/old-buildroot | mkdir $RPM_BUILD_ROOT
(easy to automate by some inotify in $(dirname $RPM_BUILD_ROOT))
Nice catch. I agree with Enrico, if we start trying to fix that, too, we end up with a loop of mkdir's (w/o -p) from outer to inner with testing ownerships/permissions and so on. This would then bloat to take over most of the %install section. We already have resistance to adding a single mkdir line. :/
Instead the plain mkdir solution *will* fail, making the user rethink about his setup. If the user wants to build all his stuff under /var/tmp/<user>/... (which is a legitimate setup, of course), he needs to first create the basic sceleton with proper permissions, and the failure will make him do that. Otherwise we create scenarios like Enrico describes.
E.g. The buildroot setting should assume that the parent folders are all properly set up beforehand, including existance, ownership and permissions. Then we only need an rm/mkdir pair.
Till Maas wrote :
A lot of specs and the template specs from rpmdev-newspec use following:
%install rm -rf $RPM_BUILD_ROOT # racing time mkdir -p $RPM_BUILD_ROOT/bla/foo
The solution is to make the following mandatory: %install rm -rf $RPM_BUILD_ROOT mkdir $RPM_BUILD_ROOT # this fails when $RPM_BUILD_ROOT already exists
Other solutions : - Use mach or mock - Use a _tmppath inside your $HOME (thus buildroot will be too)
We all do one or the other, or even both, right? :-)
Matthias
packaging@lists.fedoraproject.org
-
Axel Thimm -
Bill Nottingham -
Enrico Scholz -
Matthias Saou -
Ralf Corsepius -
Rex Dieter -
Till Maas -
Tom Callaway -
Ville Skyttä