Hi,
We have several forms of non-Koji internal builds:
1) COPR (the most visible) 2) http://alt.fedoraproject.org/pub/alt/rawhide-kernel-nodebug/ (this could be a COPR) 3) rpm-ostree: http://rpm-ostree.cloud.fedoraproject.org/#/
Now, I recently was allocated a new atomic01.qa.fedoraproject.org server dedicated to rpm-ostree composes, and I'm happy to announce I now have it composing trees:
fedora-atomic/rawhide/x86_64/server/docker-host => 175510ab77ef39ec6000db2745c70444d55957e38c5ff28f890237e1f012ea5e fedora-atomic/rawhide/x86_64/workstation/gnome/core => b6f3f3b53ef957c71e8d8bdc36c35519888d4177bf6a582ba37e33416ff84216
But it's on the QA network, and so I can't just serve the content directly from it. Nor would I want to have the compose server doing double duty as a static webserver anyways.
I could just rsync this content to the current rpm-ostree.cloud.fedoraproject.org, but there are a few issues with that:
1) It's a one off instance in the fedora cloud with no backup/monitoring 2) No redundancy 3) No tuning for static webserving 4) Most critically, no TLS
COPR only relatively recently gained TLS, which is fundamentally necessary for retrieving arbitrary executable code that runs as root.
But COPR also (from what I understand from <puiterwijk>) is just one machine.
What do you think about investing in having e.g. altcdn.fedoraproject.org be a load-balanced static webserver farm protected by TLS that could be written to by a trusted subset of non-Koji build/compose tooling?
"altcdn" is the first thing that came to mind offhand for a name, feel free to pick others =)
We'd need to determine the writing process; could be rsync-over-ssh access to specific subdirectories gated by ssh key? Something else?
On Sat, 21 Jun 2014 07:25:47 -0700 Colin Walters walters@verbum.org wrote:
Hi,
We have several forms of non-Koji internal builds:
- COPR (the most visible)
- http://alt.fedoraproject.org/pub/alt/rawhide-kernel-nodebug/ (this
could be a COPR) 3) rpm-ostree: http://rpm-ostree.cloud.fedoraproject.org/#/
Now, I recently was allocated a new atomic01.qa.fedoraproject.org server dedicated to rpm-ostree composes, and I'm happy to announce I now have it composing trees:
fedora-atomic/rawhide/x86_64/server/docker-host => 175510ab77ef39ec6000db2745c70444d55957e38c5ff28f890237e1f012ea5e fedora-atomic/rawhide/x86_64/workstation/gnome/core => b6f3f3b53ef957c71e8d8bdc36c35519888d4177bf6a582ba37e33416ff84216
But it's on the QA network, and so I can't just serve the content directly from it. Nor would I want to have the compose server doing double duty as a static webserver anyways.
Yeah, we can actually map an external IP into it if required, but I agree it isn't great to have the build host also serving external content.
I could just rsync this content to the current rpm-ostree.cloud.fedoraproject.org, but there are a few issues with that:
- It's a one off instance in the fedora cloud with no
backup/monitoring 2) No redundancy 3) No tuning for static webserving 4) Most critically, no TLS
Yep.
COPR only relatively recently gained TLS, which is fundamentally necessary for retrieving arbitrary executable code that runs as root.
But COPR also (from what I understand from <puiterwijk>) is just one machine.
Correct.
What do you think about investing in having e.g. altcdn.fedoraproject.org be a load-balanced static webserver farm protected by TLS that could be written to by a trusted subset of non-Koji build/compose tooling?
"altcdn" is the first thing that came to mind offhand for a name, feel free to pick others =)
We'd need to determine the writing process; could be rsync-over-ssh access to specific subdirectories gated by ssh key? Something else?
I think this is pretty much already in place with our /pub/alt/ space (that the kernel nodebug uses above).
This space is on a netapp. It is mirrored, but only by a few larger mirrors, most don't. It's served by our 5 download servers in phx2 and 3 on I2 in rdu2.
So, we could sync the ostree stuff there as well... its a large number of small files? Or can you expand on the size and content it will have over time and how often updated?
kevin
On Sat, Jun 21, 2014, at 10:35 AM, Kevin Fenzi wrote:
Yeah, we can actually map an external IP into it if required, but I agree it isn't great to have the build host also serving external content.
Though, separate from the actual content, it would be nice if it had a webserver to output live compose status. That would be very low priority requests.
I think this is pretty much already in place with our /pub/alt/ space (that the kernel nodebug uses above).
This space is on a netapp. It is mirrored, but only by a few larger mirrors, most don't. It's served by our 5 download servers in phx2 and 3 on I2 in rdu2.
Cool. Would it make sense to sync COPR there as well?
So, we could sync the ostree stuff there as well... its a large number of small files? Or can you expand on the size and content it will have over time and how often updated?
Yep, lots of small (almost all immutable) files. At least until https://bugzilla.gnome.org/show_bug.cgi?id=721799 lands.
Here's some quick stats on the old repository:
# du -shc repo.old 46G repo.old srv/rpm-ostree <uid=0> # find repo.old | wc -l 560164
This however includes binary-level history over months for many packages. The base/core tree is 92 commits, from 2014-02-22 23:27:03 +0000 to 2014-06-04 18:48:16 +0000.
A major source of bloat is having to regenerate the initramfs each time: https://bugzilla.redhat.com/show_bug.cgi?id=1098457 92 commits * 20 MiB is almost 2G per tree, with 6 trees the initramfs images account for 25% of the size.
gnome-continuous has an optimization for this, if you look at https://git.gnome.org/browse/gnome-continuous/log/manifest.json you'll see the "initramfs-depends": true, we only regenerate it for that.
The *new* repository I just composed has just 2 commits for each tree so far:
# du -sh repo 3.5G repo # find repo | wc -l 168744
Also, the way I'd like this to work is to have separate "release" versus "integration" repositories. The release repository wouldn't have the full binary history, it gets recomposed
The amount of space used though is flexible because we can also prune/rebase the history.
The goal for how often updated would come down to the rate of change of the RPM repositories it tracks; the goal is when an RPM is updated, the tree regenerates.
On Sat, 21 Jun 2014 13:53:12 -0700 Colin Walters walters@verbum.org wrote:
On Sat, Jun 21, 2014, at 10:35 AM, Kevin Fenzi wrote:
Yeah, we can actually map an external IP into it if required, but I agree it isn't great to have the build host also serving external content.
Though, separate from the actual content, it would be nice if it had a webserver to output live compose status. That would be very low priority requests.
Well, we do have: https://apps.fedoraproject.org/releng-dash/ which is a simple page that pulls info from our fedmsg database.
If we get your composes emitting fedmsgs we can query for and watch for those too.
I think this is pretty much already in place with our /pub/alt/ space (that the kernel nodebug uses above).
This space is on a netapp. It is mirrored, but only by a few larger mirrors, most don't. It's served by our 5 download servers in phx2 and 3 on I2 in rdu2.
Cool. Would it make sense to sync COPR there as well?
Well, we have talked about it... but all of copr is a pretty large thing and only getting larger over time, so syncing could lead to slow propagation.
So, we have some dedicated storage coming in for copr. (dell equalogics unit). Which we can look at making a bit more HA with multiple frontends, etc.
So, we could sync the ostree stuff there as well... its a large number of small files? Or can you expand on the size and content it will have over time and how often updated?
Yep, lots of small (almost all immutable) files. At least until https://bugzilla.gnome.org/show_bug.cgi?id=721799 lands.
Here's some quick stats on the old repository:
# du -shc repo.old 46G repo.old srv/rpm-ostree <uid=0> # find repo.old | wc -l 560164
This however includes binary-level history over months for many packages. The base/core tree is 92 commits, from 2014-02-22 23:27:03 +0000 to 2014-06-04 18:48:16 +0000.
A major source of bloat is having to regenerate the initramfs each time: https://bugzilla.redhat.com/show_bug.cgi?id=1098457 92 commits * 20 MiB is almost 2G per tree, with 6 trees the initramfs images account for 25% of the size.
gnome-continuous has an optimization for this, if you look at https://git.gnome.org/browse/gnome-continuous/log/manifest.json you'll see the "initramfs-depends": true, we only regenerate it for that.
The *new* repository I just composed has just 2 commits for each tree so far:
# du -sh repo 3.5G repo # find repo | wc -l 168744
Also, the way I'd like this to work is to have separate "release" versus "integration" repositories. The release repository wouldn't have the full binary history, it gets recomposed
The amount of space used though is flexible because we can also prune/rebase the history.
The goal for how often updated would come down to the rate of change of the RPM repositories it tracks; the goal is when an RPM is updated, the tree regenerates.
Yeah, my only worry here is that we have too much churn, but it sounds like there's some mitigation there and you have already been working on reducing that ;)
kevin
On Thu, Jun 26, 2014, at 12:52 PM, Kevin Fenzi wrote:
Yeah, my only worry here is that we have too much churn, but it sounds like there's some mitigation there and you have already been working on reducing that ;)
I'm getting complaints about the download speed from the one-off rpm-ostree.cloud instance, and there's still the outstanding concerns about no TLS protection.
Is there anything we can do to move this forward?
On Thu, 17 Jul 2014 16:01:56 -0400 Colin Walters walters@verbum.org wrote:
On Thu, Jun 26, 2014, at 12:52 PM, Kevin Fenzi wrote:
Yeah, my only worry here is that we have too much churn, but it sounds like there's some mitigation there and you have already been working on reducing that ;)
I'm getting complaints about the download speed from the one-off rpm-ostree.cloud instance, and there's still the outstanding concerns about no TLS protection.
Is there anything we can do to move this forward?
Sorry for the delay here...
I can add you some space right now. How do you normally sync the content? Manual rsync, or is there scripting?
Ideally we want some setup with no private keys stored on machines...
We just need to come up with a setup that we think is best and implement it. Hit me on irc if you want to discuss ideas.
kevin
infrastructure@lists.fedoraproject.org
-
Colin Walters -
Kevin Fenzi