skvidal@phy.duke.edu (seth vidal) writes:
The speedup gained by caching is not (primarily) related to the network connection but to the rpm installation and disk-IO. On the fedora.us server we used caching also and the initial buildroot was created in <1min (afair).
Local tests at that time showed, that the initial buildroot can be created in 20 seconds when a tmpfs will be used instead of a real disk with ext3.
So the numbers I'm curious about that you may have:
- remote repos - but lan-like connection times
- local-disk repos - chroot creation - all on ext3
- tmpfs targets for the chroot creation and repos
do you have anything like that?
ok, a little bit late, but I implemented caching now so I can provide some real numbers:
Traditional 'mock init' on a tmpfs takes
| Init buildroot... | | real 1m37.445s | user 1m15.329s | sys 0m8.009s
tar'ing this filesystem, and extracting it takes
| Extracting cached buildroot... | | real 0m1.266s | user 0m0.188s | sys 0m0.944s
Both operations are resulting into the same root-fs; the executed code was:
| mock=( env http_proxy= mock --resultdir="$results" -r "$MOCK_ROOT" ) | mock_yum=( env http_proxy= /usr/sbin/mock-helper yum --installroot "$BUILD_DIR" ) | | function initBuildroot() { | "${mock[@]}" clean >/dev/null | | if test -e "$CACHE_NAME"; then | echo "Extracting cached buildroot..." | time /usr/sbin/mock-helper uncache "$MOCK_ROOT" | | echo "Updating metadata..." | "${mock_yum[@]}" check-update || : | else | lock | echo "Init buildroot..." | time "${mock[@]}" init >/dev/null | unlock | | echo "Cleaning up root..." | "${mock_yum[@]}" clean packages headers | echo "Creating cache..." | /usr/sbin/mock-helper cache "$MOCK_ROOT" | fi | }
The 'cache' and 'uncache' operations of 'mock-helper' were added by me and are simple 'tar' wrappers. The resulting tarball is uncompressed and 280MB sized, both the rootfs and the tarball are located in the tmpfs.
Environment: mock-machine: Intel Celeron 2.80GHz, 1500MB RAM repository-server: PII 266MHz, 196MB RAM, http-access to repo (Apache2) 100Mb/s LAN; 'wget' download rate between repo and mock machine appr. 7-8MB/s
Enrico
ok, a little bit late, but I implemented caching now so I can provide some real numbers:
Traditional 'mock init' on a tmpfs takes
| Init buildroot... | | real 1m37.445s | user 1m15.329s | sys 0m8.009s
tar'ing this filesystem, and extracting it takes
| Extracting cached buildroot... | | real 0m1.266s | user 0m0.188s | sys 0m0.944s
Both operations are resulting into the same root-fs; the executed code was:
| mock=( env http_proxy= mock --resultdir="$results" -r "$MOCK_ROOT" ) | mock_yum=( env http_proxy= /usr/sbin/mock-helper yum --installroot "$BUILD_DIR" ) | | function initBuildroot() { | "${mock[@]}" clean >/dev/null | | if test -e "$CACHE_NAME"; then | echo "Extracting cached buildroot..." | time /usr/sbin/mock-helper uncache "$MOCK_ROOT" | | echo "Updating metadata..." | "${mock_yum[@]}" check-update || : | else | lock | echo "Init buildroot..." | time "${mock[@]}" init >/dev/null | unlock | | echo "Cleaning up root..." | "${mock_yum[@]}" clean packages headers | echo "Creating cache..." | /usr/sbin/mock-helper cache "$MOCK_ROOT" | fi | }
The 'cache' and 'uncache' operations of 'mock-helper' were added by me and are simple 'tar' wrappers. The resulting tarball is uncompressed and 280MB sized, both the rootfs and the tarball are located in the tmpfs.
Environment: mock-machine: Intel Celeron 2.80GHz, 1500MB RAM repository-server: PII 266MHz, 196MB RAM, http-access to repo (Apache2) 100Mb/s LAN; 'wget' download rate between repo and mock machine appr. 7-8MB/s
okay - that looks great. could I talk you into applying those patches to the mock package in fedora cvs and testing it out? If it works and people are happy with it then I don't see any reason not to use it.
fair enough?
-sv
buildsys@lists.fedoraproject.org
-
Enrico Scholz -
seth vidal