We've got a lot of prep work to do before Fedora 7 launches. I'd like to compile a list so if anything is missing let me know:
1) Static page content: Work with duffy, karsten, ricky and the websites team to create a nice looking static page. Even better would be getting this page into plone. Here's where it is at now: http://people.redhat.com/duffy/fedora/web/static-page.html Ultimately this page will be distributed to the mirrors in hopes we'll not need to use them. The idea is that a GET request followed by a REDIRECT is the easiest, most efficient operation we can do if our webservers get overloaded like they did last release.
2) Mirrors.fedoraproject.org: Looking good so far except for the memory leak which steps have been taken to negate. We will also need to deploy this on an additional box. I have an idea where to put it and hope to have it up fairly soon.
3) Proxy server upgrades. Right now our proxy servers are running stock RHEL4. We've been meaning to upgrade them to RHEL5 for a while now but they are on a different network segment then the rest of our hardware and as such we cannot easily pxe boot them (it would involve a request to the SOC). I'm going to put a plan together to do this and minimize any risk that may come up. The main benefit being mod_proxy_balancer. I'm still hoping we can acquire some hardware balancers but this will help us limp along for this release :)
4) PPC Builder: Is being delivered right now. Still needs to be installed, built out put into the koji mix.
What am I missing?
-Mike
Mike,
Is it possible to get 1GB extra ram for each Proxy and APP server ? This would double the amount of hits that we can sustain, in the webserver point of view.
Paulo
On 5/11/07, Mike McGrath mmcgrath@redhat.com wrote:
We've got a lot of prep work to do before Fedora 7 launches. I'd like to compile a list so if anything is missing let me know:
- Static page content: Work with duffy, karsten, ricky and the websites
team to create a nice looking static page. Even better would be getting this page into plone. Here's where it is at now: http://people.redhat.com/duffy/fedora/web/static-page.html Ultimately this page will be distributed to the mirrors in hopes we'll not need to use them. The idea is that a GET request followed by a REDIRECT is the easiest, most efficient operation we can do if our webservers get overloaded like they did last release.
- Mirrors.fedoraproject.org: Looking good so far except for the memory
leak which steps have been taken to negate. We will also need to deploy this on an additional box. I have an idea where to put it and hope to have it up fairly soon.
- Proxy server upgrades. Right now our proxy servers are running stock
RHEL4. We've been meaning to upgrade them to RHEL5 for a while now but they are on a different network segment then the rest of our hardware and as such we cannot easily pxe boot them (it would involve a request to the SOC). I'm going to put a plan together to do this and minimize any risk that may come up. The main benefit being mod_proxy_balancer. I'm still hoping we can acquire some hardware balancers but this will help us limp along for this release :)
- PPC Builder: Is being delivered right now. Still needs to be
installed, built out put into the koji mix.
What am I missing?
-Mike
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Paulo Santos wrote:
Mike,
Is it possible to get 1GB extra ram for each Proxy and APP server ? This would double the amount of hits that we can sustain, in the webserver point of view.
Possibly, I'll look into it. The issue is I don't know how some of these servers are already configured so I'm not sure what slots are free. I'll take a look.
-Mike
The operating system is detecting 1GB only in each server, sometimes starts swaping which is really not good. The theory behind Apache is really simple, more memory = more hits/sec without queueing requests
Paulo
On 5/11/07, Mike McGrath mmcgrath@redhat.com wrote:
Paulo Santos wrote:
Mike,
Is it possible to get 1GB extra ram for each Proxy and APP server ? This would double the amount of hits that we can sustain, in the webserver point of view.
Possibly, I'll look into it. The issue is I don't know how some of these servers are already configured so I'm not sure what slots are free. I'll take a look.
-Mike
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Hey Mike,
- Proxy server upgrades. Right now our proxy servers are running stock
RHEL4. We've been meaning to upgrade them to RHEL5 for a while now but they are on a different network segment then the rest of our hardware and as such we cannot easily pxe boot them (it would involve a request to the SOC). I'm going to put a plan together to do this and minimize any risk that may come up. The main benefit being mod_proxy_balancer. I'm still hoping we can acquire some hardware balancers but this will help us limp along for this release :)
Have you looked at using iptables for load balancing? Nth module could help with this.
On 11/05/07, Mike McGrath mmcgrath@redhat.com wrote:
We've got a lot of prep work to do before Fedora 7 launches. I'd like to compile a list so if anything is missing let me know:
- Static page content: Work with duffy, karsten, ricky and the websites
team to create a nice looking static page. Even better would be getting this page into plone. Here's where it is at now: http://people.redhat.com/duffy/fedora/web/static-page.html Ultimately this page will be distributed to the mirrors in hopes we'll not need to use them. The idea is that a GET request followed by a REDIRECT is the easiest, most efficient operation we can do if our webservers get overloaded like they did last release.
- Mirrors.fedoraproject.org: Looking good so far except for the memory
leak which steps have been taken to negate. We will also need to deploy this on an additional box. I have an idea where to put it and hope to have it up fairly soon.
- Proxy server upgrades. Right now our proxy servers are running stock
RHEL4. We've been meaning to upgrade them to RHEL5 for a while now but they are on a different network segment then the rest of our hardware and as such we cannot easily pxe boot them (it would involve a request to the SOC). I'm going to put a plan together to do this and minimize any risk that may come up. The main benefit being mod_proxy_balancer. I'm still hoping we can acquire some hardware balancers but this will help us limp along for this release :)
- PPC Builder: Is being delivered right now. Still needs to be
installed, built out put into the koji mix.
What am I missing?
-Mike
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Damian Myerscough wrote:
Hey Mike,
- Proxy server upgrades. Right now our proxy servers are running stock
RHEL4. We've been meaning to upgrade them to RHEL5 for a while now but they are on a different network segment then the rest of our hardware and as such we cannot easily pxe boot them (it would involve a request to the SOC). I'm going to put a plan together to do this and minimize any risk that may come up. The main benefit being mod_proxy_balancer. I'm still hoping we can acquire some hardware balancers but this will help us limp along for this release :)
Have you looked at using iptables for load balancing? Nth module could help with this.
Honestly I didn't. I thought about using some of the RH clustering suite but it will add a bit more overhead then we want for our current environment. Do you have a link to some good documentation for iptables based load balancing?
-Mike
The RedHat Load balancing clustering tool (piranha) might looks a bit complex, but in reality, it's just a fancy wrapper around ipvsadm which is the user-space tool for the kernel traffic director. I havent played with Nth module, but ipvsadm is probably more tested. So, we don't have a hardware balancer ? are we searching for a software balancing solution ?
On 5/11/07, Mike McGrath mmcgrath@redhat.com wrote:
Damian Myerscough wrote:
Hey Mike,
- Proxy server upgrades. Right now our proxy servers are running
stock
RHEL4. We've been meaning to upgrade them to RHEL5 for a while now but they are on a different network segment then the rest of our hardware and as such we cannot easily pxe boot them (it would involve a request to the SOC). I'm going to put a plan together to do this and minimize any risk that may come up. The main benefit being mod_proxy_balancer. I'm still hoping we can acquire some hardware balancers but this will help us limp along for this release :)
Have you looked at using iptables for load balancing? Nth module could help with this.
Honestly I didn't. I thought about using some of the RH clustering suite but it will add a bit more overhead then we want for our current environment. Do you have a link to some good documentation for iptables based load balancing?
-Mike
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Ahmed Kamal wrote:
The RedHat Load balancing clustering tool (piranha) might looks a bit complex, but in reality, it's just a fancy wrapper around ipvsadm which is the user-space tool for the kernel traffic director. I havent played with Nth module, but ipvsadm is probably more tested. So, we don't have a hardware balancer ? are we searching for a software balancing solution ?
I've used piranha in the past with great success but it requires additional boxes that we just don't have yet. We have one hardware balancer we use for the proxy servers. I'd like to stick one between the proxy servers and the application servers.
-Mike
Hey Mike,
I have seen:
if you want to balance the load to the 3 addresses 10.0.0.5, 10.0.0.6 and 10.0.0.7, then you can do as follows :
# iptables -t nat -A POSTROUTING -o eth0 -m nth --counter 7 --every 3 --packet 0 -j SNAT --to-source 10.0.0.5 # iptables -t nat -A POSTROUTING -o eth0 -m nth --counter 7 --every 3 --packet 1 -j SNAT --to-source 10.0.0.6 # iptables -t nat -A POSTROUTING -o eth0 -m nth --counter 7 --every 3 --packet 2 -j SNAT --to-source 10.0.0.7
Do you have a box that can act as a dedicated balance loader while FC 7 is being released
On 11/05/07, Mike McGrath mmcgrath@redhat.com wrote:
Damian Myerscough wrote:
Hey Mike,
- Proxy server upgrades. Right now our proxy servers are running stock
RHEL4. We've been meaning to upgrade them to RHEL5 for a while now but they are on a different network segment then the rest of our hardware and as such we cannot easily pxe boot them (it would involve a request to the SOC). I'm going to put a plan together to do this and minimize any risk that may come up. The main benefit being mod_proxy_balancer. I'm still hoping we can acquire some hardware balancers but this will help us limp along for this release :)
Have you looked at using iptables for load balancing? Nth module could help with this.
Honestly I didn't. I thought about using some of the RH clustering suite but it will add a bit more overhead then we want for our current environment. Do you have a link to some good documentation for iptables based load balancing?
-Mike
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Damian Myerscough wrote:
Hey Mike,
I have seen:
if you want to balance the load to the 3 addresses 10.0.0.5, 10.0.0.6 and 10.0.0.7, then you can do as follows :
# iptables -t nat -A POSTROUTING -o eth0 -m nth --counter 7 --every 3 --packet 0 -j SNAT --to-source 10.0.0.5 # iptables -t nat -A POSTROUTING -o eth0 -m nth --counter 7 --every 3 --packet 1 -j SNAT --to-source 10.0.0.6 # iptables -t nat -A POSTROUTING -o eth0 -m nth --counter 7 --every 3 --packet 2 -j SNAT --to-source 10.0.0.7
Do you have a box that can act as a dedicated balance loader while FC 7 is being released
How well does this work when 10.0.0.6 goes down?
-Mike
Hmmmm, I am not 100% sure when a box goes down. I believe it would still route the traffic to the dead box however if you set the --every 1 you wouldn't notice too much.
It would give us enough time I reckon to get the box back online or we could just remove it from the iptables until the box came back up.
On 11/05/07, Mike McGrath mmcgrath@redhat.com wrote:
Damian Myerscough wrote:
Hey Mike,
I have seen:
if you want to balance the load to the 3 addresses 10.0.0.5, 10.0.0.6 and 10.0.0.7, then you can do as follows :
# iptables -t nat -A POSTROUTING -o eth0 -m nth --counter 7 --every 3 --packet 0 -j SNAT --to-source 10.0.0.5 # iptables -t nat -A POSTROUTING -o eth0 -m nth --counter 7 --every 3 --packet 1 -j SNAT --to-source 10.0.0.6 # iptables -t nat -A POSTROUTING -o eth0 -m nth --counter 7 --every 3 --packet 2 -j SNAT --to-source 10.0.0.7
Do you have a box that can act as a dedicated balance loader while FC 7 is being released
How well does this work when 10.0.0.6 goes down?
-Mike
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
On Fri, May 11, 2007 at 09:51:42AM -0500, Mike McGrath wrote:
We've got a lot of prep work to do before Fedora 7 launches. I'd like to compile a list so if anything is missing let me know:
[...]
What am I missing?
Bodhi.
I have a test instance running on publictest2[0] that people can play around with at the moment. Here is what is on my list of priority tasks that need to get done before F7:
o Deployment. Get bodhi configs into puppet, and make sure ssl/static files/etc are working from admin.fp.org/updates
o Make sure all EPEL guidelines[1] are met
o Multilib for F7. For previous releases there was a huge biarch-list-of-DOOM, which I have since imported into bodhi's database. However, there is no such list for F7, so we will most likely need to do things differently. notting mentioned somehow doing it on the fly, or by having bodhi tag things in koji and then using mash to build the entire updates repo.
o Repo cleaner. We need something like RepoPrune.py/fedora-updates-clean to run occasionally and clean up our tree.
I'll make sure our Bodhi Timeline[2] is up to date today. I'd like to get 1.0 deployed this weekend, and ideally hit 1.1 before F7.
Comments/suggestions/help welcome :)
luke
[0]: http://publictest2.fedora.redhat.com [1]: http://fedoraproject.org/wiki/EPEL/GuidelinesAndPolicies/PackageMaintenanceA... [2]: https://hosted.fedoraproject.org/projects/bodhi/roadmap
On Fri, 11 May 2007 14:21:43 -0400, Luke Macken wrote:
o Repo cleaner. We need something like RepoPrune.py/fedora-updates-clean to run occasionally and clean up our tree.
There's also repomanage, ready to use:
$ rpm -qf $(which repomanage) yum-utils-1.0.3-1.fc6
It has been used for a long time for Extras, but only solves one part of repository maintenance. Another part, that remains, can get tiresome and error-prone, especially when no package db is available (one reason why repoprune took over and also one reason why I like kojira).
Everywhere repoprune has been mentioned or derived so far it has been misunderstood. It does NOT do the same as repomanage. It is not designed to do exactly the same. The reason it is approximately four times faster is that it does something else, something more rigorous and more helpful.
So, what technique/tool to use is a repository-design decision. If for every binary rpm in a repository there MUST be a source rpm in a corresponding repository, repoprune can do its job really well and prune those repositories. Where the requirement isn't met, repoprune cannot be used. For Fedora Extras, the requirement is met.
In the interface, repoprune needs to know about:
1) root path to source rpms directory 2) root path to corresponding binary rpms directory 3) how many different builds of each package to keep 4) a white-list of source package %{name}s to ignore
Repoprune then deletes old source rpms in pass one. In pass two it deletes any binary rpm that refers to a non-existant source rpm. If necessary, I could think of making 3) a map, which would make it possible to configure the number of builds per %name.
On May 11, 2007, at 7:51 AM, Mike McGrath wrote:
- Proxy server upgrades.
One easy upgrade would be to use perlbal instead of apache/ mod_proxy. It will use less memory, be able to sustain more connections, do error handling and better balance the load between the backend servers.
I'll be happy to help with this ...
- ask
infrastructure@lists.fedoraproject.org
-
Ahmed Kamal -
Ask Bjørn Hansen -
Damian Myerscough -
Luke Macken -
Michael Schwendt -
Mike McGrath -
Paulo Santos