Hi all,
I've been looking at how to change mock so that the build chroot's can be placed at arbitrary locations (as opposed to the current hardcoded /var/lib/mock). The main reason I want to do this is that I build target root filesystems for embedded targets in our compile farm and we use a big NFS filesystem as our main storage. I can access the build directory and our cross-toolchains from any system in our farm and nothing that's specific to a build is on any farm system (I can kickstart it at will and not lose anything). Yes, we've successfully redirected /var/lib/mock to the NFS storage with a mount --bind but I just find that kinda scary. I'm just not very confident in remounting an NFS filesystem to a local filesystem. The bottom line is that I'd like to remove dependencies on the build system's local filesystem and move it all to the NFS storage.
So, the problem seems to be how to modify mock-helper so that it allows the move, yet doesn't become a handy-dandy-attack-point for some cracker wannabe.
Can we say that the following is true?
"The mock-helper program always begins execution *outside* of a chroot and is the only mechanism mock uses to *enter* a chroot."
If so, then we can check to insure that the intended chroot directory:
1. Is not '/' 2. Is not a symlink to '/' 3. Is not in a list of special directories (/bin, /sbin, /lib, etc.)
Are we "safe" if we insure that whatever command is executed by mock-helper is executed in a chroot directory that does not contain a system critical component?
Clark
On Fri, 2006-02-24 at 17:02 -0600, Clark Williams wrote:
Hi all,
I've been looking at how to change mock so that the build chroot's can be placed at arbitrary locations (as opposed to the current hardcoded /var/lib/mock). The main reason I want to do this is that I build target root filesystems for embedded targets in our compile farm and we use a big NFS filesystem as our main storage. I can access the build directory and our cross-toolchains from any system in our farm and nothing that's specific to a build is on any farm system (I can kickstart it at will and not lose anything). Yes, we've successfully redirected /var/lib/mock to the NFS storage with a mount --bind but I just find that kinda scary. I'm just not very confident in remounting an NFS filesystem to a local filesystem. The bottom line is that I'd like to remove dependencies on the build system's local filesystem and move it all to the NFS storage.
I can't speak for Seth, but I'm fairly sure the answer here is "use bind mounts". Seth is very big on the FHS, and in the FHS, that's where the buildroots and other stuff for mock are supposed to live. I've asked about this long ago, when starting to use mock for the build system, and that was the answer. Seriously, bind mounts are neither complicated, nor problematic, though I haven't done a ton of testing with NFS.
The other question is why you want the buildroots on NFS in the first place. Unless you have blazingly fast storage, it's likely that local SCSI disk will be quite a bit faster than NFS. Since creating buildroot and doing the actual builds involves _quite_ a lot of IO, the same issue applies; local storage is likely better unless you've got really a fast path to the shared storage.
Dan
On Fri, 2006-02-24 at 18:10 -0500, Dan Williams wrote:
I can't speak for Seth, but I'm fairly sure the answer here is "use bind mounts". Seth is very big on the FHS, and in the FHS, that's where the buildroots and other stuff for mock are supposed to live. I've asked about this long ago, when starting to use mock for the build system, and that was the answer. Seriously, bind mounts are neither complicated, nor problematic, though I haven't done a ton of testing with NFS.
I didn't mean to imply that I'm not confident of my ability to *use* bind mounts, just that I'm suspicious of remounting a network filesystem. Fifteen or so years of wrestling with NFS have made me leery of layering anything on top of it.
The other question is why you want the buildroots on NFS in the first place. Unless you have blazingly fast storage, it's likely that local SCSI disk will be quite a bit faster than NFS. Since creating buildroot and doing the actual builds involves _quite_ a lot of IO, the same issue applies; local storage is likely better unless you've got really a fast path to the shared storage.
The main reason I'd like to have an arbitrary location for buildroots is that in our farm system, multiple builds may be occurring on a compile host at the same time (e.g. one person building for a MIPS and another building for ARM). Having them both collide in /var/lib/mock wouldn't be fun. Since each architecture has it's own space on the NFS server (which is connected via gigabit ethernet to the compile systems), if you stay in your own sandbox, you don't collide with anyone else.
I'm not sure I want to use mock in the same way it's used by Plague (i.e. clean the chroot, populate it, build one package). Since we currently use a custom set of rpmmacros, rpmrc and a few command line definitions to cross-build our RPMs, I'd rather use mock to setup one chroot for an architecture and then build multiple packages out of it. I'm not certain of that now, but that's the way I'm leaning.
Clark
On Fri, 2006-02-24 at 22:38 -0600, Clark Williams wrote:
On Fri, 2006-02-24 at 18:10 -0500, Dan Williams wrote:
I can't speak for Seth, but I'm fairly sure the answer here is "use bind mounts". Seth is very big on the FHS, and in the FHS, that's where the buildroots and other stuff for mock are supposed to live. I've asked about this long ago, when starting to use mock for the build system, and that was the answer. Seriously, bind mounts are neither complicated, nor problematic, though I haven't done a ton of testing with NFS.
I didn't mean to imply that I'm not confident of my ability to *use* bind mounts, just that I'm suspicious of remounting a network filesystem. Fifteen or so years of wrestling with NFS have made me leery of layering anything on top of it.
The other question is why you want the buildroots on NFS in the first place. Unless you have blazingly fast storage, it's likely that local SCSI disk will be quite a bit faster than NFS. Since creating buildroot and doing the actual builds involves _quite_ a lot of IO, the same issue applies; local storage is likely better unless you've got really a fast path to the shared storage.
The main reason I'd like to have an arbitrary location for buildroots is that in our farm system, multiple builds may be occurring on a compile host at the same time (e.g. one person building for a MIPS and another building for ARM). Having them both collide in /var/lib/mock wouldn't be fun. Since each architecture has it's own space on the NFS server (which is connected via gigabit ethernet to the compile systems), if you stay in your own sandbox, you don't collide with anyone else.
Ok; if you want different directories for different architectures, there are two ways to do it:
1) in the mock config file for the target, change the 'root' option to include your architecture. Then pass '-r <target>' on the mock command line
2) use the --uniqueext=<unique extension> option on the mock command line to futher unique-ify your buildroot
Dan
The main reason I'd like to have an arbitrary location for buildroots is that in our farm system, multiple builds may be occurring on a compile host at the same time (e.g. one person building for a MIPS and another building for ARM). Having them both collide in /var/lib/mock wouldn't be fun. Since each architecture has it's own space on the NFS server (which is connected via gigabit ethernet to the compile systems), if you stay in your own sandbox, you don't collide with anyone else.
will the rpmdb's get created properly on nfs? I thought they went a bit nutty?
I'm not sure I want to use mock in the same way it's used by Plague (i.e. clean the chroot, populate it, build one package). Since we currently use a custom set of rpmmacros, rpmrc and a few command line definitions to cross-build our RPMs, I'd rather use mock to setup one chroot for an architecture and then build multiple packages out of it. I'm not certain of that now, but that's the way I'm leaning.
you should look at the mock config file - you can put your rpmmacros and rpmrc specifications in to the config file and it will be pushed into the chroot automatically.
-sv
On Sun, 2006-02-26 at 11:01 -0500, seth vidal wrote:
The main reason I'd like to have an arbitrary location for buildroots is that in our farm system, multiple builds may be occurring on a compile host at the same time (e.g. one person building for a MIPS and another building for ARM). Having them both collide in /var/lib/mock wouldn't be fun. Since each architecture has it's own space on the NFS server (which is connected via gigabit ethernet to the compile systems), if you stay in your own sandbox, you don't collide with anyone else.
will the rpmdb's get created properly on nfs? I thought they went a bit nutty?
I work with Clark so I'll throw in a bit here.
Storing a rpmdb on NFS does OK for awhile, then goes wonky (hope that's not too technical for you ;-). Since our target rpmdb's were fairly transitory, it wasn't a huge deal, just annoying. It finally got *too* annoying one day and I found a workaround (with the help of every programmer's friend - google). I added the following to our rpmmacros file:
# We're adding the private token. This should help when storing the db # on a NFS file system. %__dbi_other %{?_tmppath:tmpdir=%{_tmppath}} %{?__dbi_cdb} \ private
Adding 'private' to %__dbi_other causes rpm to use the DB_PRIVATE flag. Here's a description of DB_PRIVATE from http://www.sleepycat.com/docs/ref/env/region.html:
================================= If the DB_PRIVATE flag is specified to the DB_ENV->open method, regions are created in per-process heap memory; that is, memory returned by malloc(3).
This flag should not be specified if more than a single process is accessing the environment because it is likely to cause database corruption and unpredictable behavior. For example, if both a server application and Berkeley DB utilities (for example, db_archive, db_checkpoint or db_stat) are expected to access the environment, the DB_PRIVATE flag should not be specified.
...
If no memory-related flags are specified to DB_ENV->open, memory backed by the filesystem is used to store the regions. On UNIX systems, the Berkeley DB library will use the POSIX mmap interface. If mmap is not available, the UNIX shmget interfaces may be used instead, if they are available. =================================
After adding "private", our use of rpmdbs on NFS has been rock-solid. (However, your mileage may vary.)
I'm not sure I want to use mock in the same way it's used by Plague (i.e. clean the chroot, populate it, build one package). Since we currently use a custom set of rpmmacros, rpmrc and a few command line definitions to cross-build our RPMs, I'd rather use mock to setup one chroot for an architecture and then build multiple packages out of it. I'm not certain of that now, but that's the way I'm leaning.
you should look at the mock config file - you can put your rpmmacros and rpmrc specifications in to the config file and it will be pushed into the chroot automatically.
-sv
Hmm, didn't know you could specify different rpmmacros and rpmrc files in the config file. I'll look into that. The only hold up for us might be that we do both native and cross compiles during our full builds and we normally use different rpmmacros/rpmrc files based on what we're compiling for.
I can't speak for Seth, but I'm fairly sure the answer here is "use bind mounts". Seth is very big on the FHS, and in the FHS, that's where the buildroots and other stuff for mock are supposed to live. I've asked about this long ago, when starting to use mock for the build system, and that was the answer. Seriously, bind mounts are neither complicated, nor problematic, though I haven't done a ton of testing with NFS.
The fhs is our friend. :) However I don't think we're suggesting changing the default location for mock just making it possible for someone else to do something like that. And if we can do that securely then, sure.
-sv
On Sun, 2006-02-26 at 11:03 -0500, seth vidal wrote:
The fhs is our friend. :) However I don't think we're suggesting changing the default location for mock just making it possible for someone else to do something like that. And if we can do that securely then, sure.
You've hit the nail on the head. I realize that some folks think badly of NFS and don't want to use it. That's fine (yeah, I'm not in love with NFS, but I still want to use it), but what if someone's using FibreChannel or InfiniBand attached storage? My argument that we should be able to move the root is still valid.
So, back to my original question: if we *exclude* certain directories as candidates for chroot'ing, can we securely move the root? I'm thinking of something like the attached patch (minus the #ifdefs).
Clark
On Mon, 2006-02-27 at 09:18 -0600, Clark Williams wrote:
So, back to my original question: if we *exclude* certain directories as candidates for chroot'ing, can we securely move the root? I'm thinking of something like the attached patch (minus the #ifdefs).
Grrr. That's what I get for doing something in a hurry. I sent the wrong patch and I didn't inline it. Sigh.
Here's the patch I *meant* to send: Index: mock-helper.c =================================================================== RCS file: /cvs/fedora/mock/src/mock-helper.c,v retrieving revision 1.7 diff -u -r1.7 mock-helper.c --- mock-helper.c 14 Jul 2005 18:00:26 +++ mock-helper.c 27 Feb 2006 15:54:09 @@ -55,6 +55,12 @@ exit (1); }
+#ifdef ARBITRARY_CHROOT +const char *disallowed[] = {"/bin", "/sbin/", "/usr", "/lib", + "/boot", "/dev", "/etc", "/var" +}; +#endif + /* * perform checks on the given dir * - is the given dir under the allowed hierarchy ? @@ -68,9 +74,21 @@ char last; int retval;
+#ifdef ARBITRARY_CHROOT + int i; + + if (strncmp(given, "/", 1) != 0) + error("can't chroot to '/'"); + + for (i=0; i < sizeof(disallowed) / sizeof(char *); i++) { + if (strncmp(given, disallowed[i], strlen(given)) != 0) + error("%s: chroot not allowed\n", disallowed[i]); + } +#else /* does given start with allowed ? */ if (strncmp (given, allowed, strlen (allowed)) != 0) error ("%s: not under allowed directory (%s)", given, allowed); +#endif
/* does it try to fool us by using .. ? */ if (strstr (given, "..") != 0)
My main thought here is, what does putting your buildroots on NFS gain you? Do you realize that one of the main reasons beehive had failures in past years was due to NFS problems?
Warren Togami wtogami@redhat.com
On Sun, 2006-02-26 at 15:38 -0500, Warren Togami wrote:
My main thought here is, what does putting your buildroots on NFS gain you? Do you realize that one of the main reasons beehive had failures in past years was due to NFS problems?
Warren Togami wtogami@redhat.com
Warren,
Perhaps I should better explain what Clark and I are really doing, and what we hope to gain by using mock so you can see where we are coming from (and where we want to go). And no, I didn't realize beehive had NFS problems since I've never used beehive...
Our group does embedded Linux ports, cross-compiling the specific architecture on an x86 system. We don't do native compiles because we typically don't create native compilers and the embedded systems are typically quite slow.
For sometime we used a home-grown system of Makefiles and source downloaded off the net. This system was quite fragile in several ways, both at build time and at run time. We got frustrated with that system and looked around for something else. I really wanted to use Fedora SRPMS as a base to start with. I looked around and found an old system that H.J. Lu had written to cross-compile part of RHL7.3 for mips. Clark and I took that system and modified and extended it.
Here's how it currently works. We start with unmodified FCx SRPMs. Each SRPM has a directory where we store our modified spec file (and patches if necessary) to enable cross-compiling. We start with an empty target arch buildroot and cross compile and install packages. As you can imagine build order is quite important at first (for instance glibc is one of the first packages we build). We build and install the packages other packages need, then build the remaining packages. We currently have about 120 or so packages in our distribution (typically taking 4-6 hours to do a complete build).
Warren, you asked what does putting our buildroots on NFS gain us. I guess the main answer is machine independence. We have several build farm systems all mounting the same NFS shares. If a build system becomes slow or goes down, I can just switch machines and restart the build at the SRPM that died. Note that we don't support building multiple RPMs simultaneously on the same buildroot (either on the same machine or on multiple machines).
So, what do we hope to gain by using mock? We hope to make the cross- compile porting process easier and we hope to gain even more machine independence. Since our customers build on RHEL, part of getting the packages to cross-compile entails getting the FCx packages to build on RHELy. For instance, we build FC3 packages on RHEL3, FC4 packages on RHEL4. Unfortunately, when we're done, we've typically mangled the packages to only build on that version of RHEL (because of autoconf/automake/etc. changes). If instead we built the FCx SRPM in a mock FCx chroot, that part of the port effort wouldn't be needed. In addition, if we're building in a mock FCx chroot, the host OS doesn't really matter.
On Sun, 2006-02-26 at 15:38 -0500, Warren Togami wrote:
My main thought here is, what does putting your buildroots on NFS gain you? Do you realize that one of the main reasons beehive had failures in past years was due to NFS problems?
Which turned out to be completely unrelated to NFS, hence the running joke :)
Best, -- Elliot
buildsys@lists.fedoraproject.org
-
Clark Williams -
Dan Williams -
David Smith -
Elliot Lee -
seth vidal -
Warren Togami