Hi, Discussion today over a few issues with fedorapeople.org and what we need to do with it.
Goals: - be able to backup fedorapeople.org much much faster - expand the amount of diskspace available on it - offer faster upload to fedorapeople.org from places where most of our userbase are. - offer 'project' space for groups to have unquota'd space available from fedorapeople.org for various non-hosted projects: examples: test spins from anaconda team abrt test-suite results
Two projects today both needed more space - but we were getting hung up b/c of how the quota'ing is laid out with the existing fedorapeople.org. I suggested just adding space and making it a new disk but Kevin mentioned the concerns about upload speed to fedorapeople.org and the issue of backing it up quickly.
After a bit of looking around ibiblio03.fedoraproject.org was seen as a likely good place to put a new fedorapeople.org.
If we put it there we could give it: 1. more ram 2. more processors 3. considerably more space 4. back it up faster (backup02 is also on ibiblio03 so backing it up should be very fast) 5. being on i2 and in a fast-colo at ibiblio would give us better upload rates for many many of our users.
So the plan would be: - setup a new people## box over at ibiblio03 - setup two disks - the normal /srv layout we have now - a new disk for project space that is outside of the quota'd area.
Kevin suggested possibly replicating this people## over to the old location using rsync so, in the event of unplanned calamity, we could continue to offer the service.
Concerns? thoughts?
-sv
On 20 February 2012 13:43, seth vidal skvidal@fedoraproject.org wrote:
Hi, Discussion today over a few issues with fedorapeople.org and what we need to do with it.
Goals:
- be able to backup fedorapeople.org much much faster
- expand the amount of diskspace available on it
- offer faster upload to fedorapeople.org from places where most of our
userbase are.
- offer 'project' space for groups to have unquota'd space available
from fedorapeople.org for various non-hosted projects: examples: test spins from anaconda team abrt test-suite results
Two projects today both needed more space - but we were getting hung up b/c of how the quota'ing is laid out with the existing fedorapeople.org. I suggested just adding space and making it a new disk but Kevin mentioned the concerns about upload speed to fedorapeople.org and the issue of backing it up quickly.
After a bit of looking around ibiblio03.fedoraproject.org was seen as a likely good place to put a new fedorapeople.org.
If we put it there we could give it: 1. more ram 2. more processors 3. considerably more space 4. back it up faster (backup02 is also on ibiblio03 so backing it up should be very fast) 5. being on i2 and in a fast-colo at ibiblio would give us better upload rates for many many of our users.
My one concern is that both backup02 and fedorapeople can have a large IO needs and we can have the starvation issues we had last year with backup02... where it would mysteriously crash when it was doing lvm items and the other ibiblio boxes were having a load.
How do the other ibiblio boxes look like for space or reallocation of disks?
So the plan would be: - setup a new people## box over at ibiblio03 - setup two disks - the normal /srv layout we have now - a new disk for project space that is outside of the quota'd area.
Kevin suggested possibly replicating this people## over to the old location using rsync so, in the event of unplanned calamity, we could continue to offer the service.
That sounds good.. though slooooow.
Concerns? thoughts?
-sv _______________________________________________ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
On Mon, 20 Feb 2012 14:34:15 -0700 Stephen John Smoogen smooge@gmail.com wrote:
My one concern is that both backup02 and fedorapeople can have a large IO needs and we can have the starvation issues we had last year with backup02... where it would mysteriously crash when it was doing lvm items and the other ibiblio boxes were having a load.
How do the other ibiblio boxes look like for space or reallocation of disks?
Not fantastic, I'm afraid.
Do we know that lvm issue was related to disk io bandwidth starvation?
I only ask b/c processor and mem are not really constricted here.
Kevin suggested possibly replicating this people## over to the old location using rsync so, in the event of unplanned calamity, we could continue to offer the service.
That sounds good.. though slooooow.
Sounds like change from day to day on people is only1-3gb. So not ridiculous to rsync
-sv
On Mon, 20 Feb 2012 14:34:15 -0700 Stephen John Smoogen smooge@gmail.com wrote:
My one concern is that both backup02 and fedorapeople can have a large IO needs and we can have the starvation issues we had last year with backup02... where it would mysteriously crash when it was doing lvm items and the other ibiblio boxes were having a load.
Well, I think that issue was a kernel one... which has since been fixed. We didn't see any issues with the old backup02 after a kernel update, so I think it wasn't related to the host in the end.
How do the other ibiblio boxes look like for space or reallocation of disks?
01 has not much space left (has download-ib01) 02 has torrent02 and a bunch of smaller guests on it.
I think 03 is the best fit.
kevin
On 20 February 2012 14:47, Kevin Fenzi kevin@scrye.com wrote:
On Mon, 20 Feb 2012 14:34:15 -0700 Stephen John Smoogen smooge@gmail.com wrote:
My one concern is that both backup02 and fedorapeople can have a large IO needs and we can have the starvation issues we had last year with backup02... where it would mysteriously crash when it was doing lvm items and the other ibiblio boxes were having a load.
Well, I think that issue was a kernel one... which has since been fixed. We didn't see any issues with the old backup02 after a kernel update, so I think it wasn't related to the host in the end.
Ok cool.
How do the other ibiblio boxes look like for space or reallocation of disks?
01 has not much space left (has download-ib01) 02 has torrent02 and a bunch of smaller guests on it.
I think 03 is the best fit.
Ok sounds like a good idea. If things came to push or shove.. one could see how much download-ib01 gets us over having download-ix01 :)
kevin
infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
On Mon, 20 Feb 2012 15:43:06 -0500 seth vidal skvidal@fedoraproject.org wrote:
Hi, Discussion today over a few issues with fedorapeople.org and what we need to do with it.
...snip...
Kevin suggested possibly replicating this people## over to the old location using rsync so, in the event of unplanned calamity, we could continue to offer the service.
Concerns? thoughts?
I've been pondering on this a while, and I think the above move to ibiblio03 could work fine.
Backups on backup02 currently run from 04:00UTC to between 5:30 and 6UTC. That should not be a time when backups of people02 run (those are usually after 22UTC, and hopefully would be done before 4UTC.
I don't know how slow sync between ibiblio03 and people02 is, but if it's not too bad we could keep the current instance synced up. If it is too bad, we could just retire people02 in favor of the new one.
kevin
infrastructure@lists.fedoraproject.org