infrastructure June 2012

infrastructure@lists.fedoraproject.org

32 participants
27 discussions

Introduction
by Onalenna Junior Makhura 09 Oct '12

09 Oct '12

Hi, I have been a member of this list for a few years now but due to some constraints I have not been able to actively participate. I would now like to start be active in Fedora development. I have been using fedora as my 1st choice OS since Fedora Core 6. I would be very glad to have someone who can show me the ropes. Regards, Onalenna Junior Makhura

2 7

Web application frameworks and the future
by Toshio Kuratomi 27 Jul '12

27 Jul '12

""" In the beginning there was cgi. And everything was slow but simple. And lo, one day we began to crave faster speeds, MVC, and other features that plain cgi did not provide. And thus we entered the age of web frameworks.... """ At last week's infrastructure meeting, I brought up the fact that we seem to have a proliferation of web application frameworks for the new apps that we are creating. In some ways this is good as it lets us experiment with new technologies as a group and lets us fit the needs of a specific application or programmer's style with the framework. However, it has downsides as well; mostly in the realm of ongoing maintenance of the apps. We need to take a moment to figure out where we want to go with this. == Some issues == * Retaining group knowledge of many different application frameworks even when the original author stops being an active contributor * Maintaining the packages in EPEL and Infrastructure for these * Maintaining some knowledge of the frameworks' code and involvement with their upstreams to fix bugs in the frameworks themselves. * Deployment of multiple frameworks that may have conflicting deps. * Deployment of multiple frameworks taking up more memory on the servers. We think to some extent we currently have ways to manage the deployment problems: * Separate app servers for individual apps. As long as we have an inflow of hardware resources we can continue to separate out applications onto different machines instead of running them all on app* as our first generation of apps was. This would be an ongoing expense. We should continue to allocate at least two servers to each application so that we can do things like reboots and updates transparently to the users. * Openshift. Hosting applications on a cloud service like openshift allows us to separate out applications and parcel out memory as a resource differently than if we're managing multiple apps on a single host. While these factors do change the game as far as hardware allocation is concerned, it doesn't help our manpower resources. As we spin up more hosts for each web application, we need sysadmin time to spin those hosts up. As we deploy to openshift we need to figure out how we're going to integrate configuration and deployment to those hosts into our existing puppet configurations (I don't think that any of our current openshift deployed services are puppet managed) and how we're going to manage load balancing and failover. == Where are we now? == .. note:: I would like this section to be an inventory of everything that we're deploying and writing but I don't have a complete picture. If you have more things, feel free to update this on the wiki page: https://fedoraproject.org/wiki/Infrastructure_Services_Survey TG1 => Turbogears1, SQLAlchemy and genshi/mako Old TG1 => TurboGears1, SQLObject and kid TG2 => TurboGears2 Pyramid => Curent successor to TG2 but a break from the current TG1 style; may have a new layer built on top of it at a later date that is more TG-ish. Flask => Easy to get started with and wrap your head around. Great for small projects. Not a huge stack of deps. Application Host Framework Notes ----------- ---- --------- ----- bodhi app* old TG1 has a pyramid branch bodhi releng* old TG1 has a pyramid branch busmon ? TG2/moksha Not yet deployed copr(2) ? flask not yet deployed. Loosely, "buildsys for fedorapeople repos" datagrepper ? flask? Not yet deployed dataviewer ? flask? Not yet deployed dpsearch ? perl/C Not yet deployed testing on search01-dev elections app* TG1 has a TG2 branch and ianweller trying a flask branch for comparison fas fas* TG1 fedorabadges ? pyramid Not yet deployed fedoracommunity app07? TG2/moksha Only runs on RHEL5. We're retiring this pending on datanommer being deployed or we get tired of keeping app07. (Is the version of moksha here old as well?) fedorahosted-reg openshift? flask not yet deployed freemedia app* php In Puppet. Looks like it would be very simple to port to something lightweight like Flask if we wanted to get away from PHP. fudcon-reg openshift flask registration application for fudcon. Not currently configured in puppet, load balanced, etc. koji koji* custom was mod_python. plans to move to mod_wsgi. (Current status?) mirrorlist-server app* custom lightweight, mod_wsgi process. No real framework mirrormanager app* old TG1 has an older TG2 branch packagedb app* TG1 packages packages* TG2 pager app*, noc* CGI raffle app* TG2 Disposable -- no promises to keep maintaining have been made smolt value* TG1 We're planning to get rid of this in favor of census on openshift (Are we still running the process on app* even though it isn't actively serving pages?) tagger packages* TG2 We deploy but do not code for: Application Host Framework Notes ----------- ---- --------- ----- askbot ask* django Uses openid login darkserver darkserver django insight insight* drupal/php I'm not sure the level of coding that we do on this. gitweb(-caching) pkgs* cgi? thinking of replacing with cgit hosted* hg? hosted* cgi? loggerhead hosted* mod_wsgi mailman webui hosted* python cgi mailman web frontend for collab* lists.fp.o and lists.fh.o mediawiki app* php reviewboard hosted* django we've talked about moving this to openshift and/or app servers trac hosted* mod_wsgi genshi templates Deployed but only for our sysadmins: collectd, nagios, awstats == Some analysis == Right now we're deploying against the following frameworks for applications in our critical path: * TG1 * mod_wsgi/mod_python We also have a few additional applications that are not currently critical to creating Fedora but are value adds that we've worked hard on. These applications are written against * TG1 * TG2 * flask The new applications that we're writing seem to be written against: * TG2 * flask * pyramid == Some thoughts == === Openshift === Although openshift is attractive from a hardware-provisioning perspective, we haven't figured out how to manage configs for it for any of our currently deployed services. So, for instance, if there was evidence that one of our openshift instances had been compromised we wouldn't have the benefit of configs checked into puppet to refer to and to help us reconstruct that instance. We probably also don't have these hosts as part of our backups (don't know if openshift manages backups for us). We should figure out disaster recovery for these hosts before we go too much further here. We also don't currently have any openshift hosts working in a load balanced fashion so, for instance, doing an update of an app could require user visible downtime. If we're going to use openshift for deploying production apps, we should come up with answers for these tasks. === Getting rid of TG1 === At some point I want to get rid of the TG1 stack. Upstream is in maintenance-only mode for it. And increasingly, they are moving to the somewhat incompatible TG-1.5.x stack for their maintenance while simultaneously pushing people to write their apps for TG2 or pyramid. While TG1.1 "just works" for us right now, we're eventually going to run up against things that upstream isn't handling (whether bugfixes in the TG-1.1.x branch, security fixes, or porting of the stack to new versions of dependent libraries). While the maintenance burden of the TG1.1 stack is low at this time, it's just going to get higher over time. In order to port away from the TG1 stack, I want to figure out what we should be porting to. Last year we thought that should be TG2 because moksha was intrinsically linked to TG2 and we were deploying on fedoracommunity which needed moksha. Now, neither of those is true. (moksha can now run on other frameworks besides TG2. fedoracommunity is going away in the future.) However, there's no clear successor. === Plethora of frameworks === We're writing and deploying apps written against an ever expanding number of frameworks. I am a bit afraid of this. While it is nice to know that we have exactly the right tool for the job among the many choices of framework, I think that maintaining apps written in a variety of frameworks is going to cause us pain as frameworks die off or change radically and current contributors move on to other things. With that in mind I think we should commit to using only a few frameworks in our coding for infrastructure and those frameworks will serve to be where we concentrate on gathering our experience, what we write new apps against, what we design our infrastructure to support, and what we port our apps to as time goes on. From browsing the list of frameworks we're currently deploying: Django has a good track record of making new releases with clear porting guides for making changes in your old code on run on the new versions. However, it is conceptually something of an application server (like JBoss), not a pure framework like Turbogears. At the least, this would require some thought on our part on how to deploy and code for it. Flask seems to be lighter weight in terms of its deps and in terms of its learning curve. It's pretty easy to run a flask app in openshift. If we were to choose just two frameworks, it might make sense to choose flask as an entry level framework for smaller applications and one other framework with lots of bells and whistles for things that need those features. TurboGears2 is still developed upstream. Some of the main developers have moved on to work on pyramid but others are continuing to work on TG2. Upstream has committed to doing the necessary work to port TG2 to python3 but much of the TG2 underlying stack is in maintenance mode so the TG2 devs have had to do some of that work themselves. Pyramid is a merging of certain segments of the zope community and the pylons community. If pylons has a successor, this is it. Since TG2 was built on pylons, pyramid might be the next logical step (or a web framework built on top of pyramid). == Final thoughts == My primary goal is to decide what framework to port our old TG1 code to so that we can stop maintaining the TG1 stack before upstream stops working on it at all. My secondary concern is that we stop growing the other stacks that we're maintaining and concentrate on one or two which will make mainenance easier. Can we choose two frameworks right now that will suit our needs? It seems that flask can serve a niche and maybe should be one of them. What should our bells and whistles framework be? TG2 or pyramid or something else entirely? -Toshio

6 8

builders of the future!!!!!
by Seth Vidal 24 Jul '12

24 Jul '12

The discussion on devel list about ARM and my work last week on reinstalling builders quickly and commonly has raised a number of issues with how we manage our builders and how we should manage them in the future. It is apparent that if we add arm builders they will be lots of physical systems (probably in a very small space) but physical, none-the-less. So we need a sensible way to manage and reinstall these hosts commonly and quickly. Additionally, we need to consider what the introduction of a largish number of arm builders (and other arm infrastructure) would do to our existing puppet setup. Specifically overloading it pretty badly and making it not-very-manageable. I'm making certain assumptions here and I'd like to be clear about what those are: 1. the builders need to be kept pristine 2. that currently our builders are not freshly installed frequently enough. 3. that the builders are relatively static in their configuration and most changes are done with pkg additions 4. that builder setups require at least two manual-ish steps of a koji admin who can disable/enable/register the builder with the kojihub. 5. that the builders are fairly different networking and setup-wise to the rest of our systems. So I am proposing that we consider the following as a general process for maintaining our builders: 1. disable the builder in koji 2. make sure all jobs are finished 3. add installer entries into grub (or run the undefine, reinstall process if the builder is virt-based) 4. reinstall the system 5. monitor for ssh to return 6. connect in and force our post-install configuration: identification, network, mount-point setup, ssl certs/keys for koji, etc 7. reboot 8. re-enable host in koji We would do this with frequency and regularity. Perhaps even having some percentage of our builders doing this at all times. Ie: 1/10th of the boxes reinstalling at any given moment so in a certain time frame*10 all of them are reinstalled. Additionally, this would mean these systems would NOT have a puppet management piece at all. Package updates would still be handled by pushes as we do now, if things were security critical, but barring the need for significant changes we could rely on the boxes simply being refreshed frequently enough that it wouldn't need to be pushed. What do folks think about this idea? It would dramatically reduce the node entries in our puppet config, it would drop the number of hosts connecting to puppet, too. It will mean more systems being reinstalled and more often. It will also require some work to make the steps I mention above be automated. I think I can achieve that without too much difficulty, actually. I think, in general, it will increase our ability to scale up to more and more builders. I'd like input, constructive, please. Thanks, -sv

5 9

mulling the idea of a Infrastructure Security FAD (fedora activity day)
by Kevin Fenzi 21 Jun '12

21 Jun '12

Greetings. I've been toying with the idea of a Fedora Infrastructure FAD (Fedora Activity Day) around getting our security tasks further along/mapped out, or just done. We can do all these things remotely, but sitting down with less distractions and getting things done or deciding on roadmaps may work faster/better in person. More information on FAD's: http://fedoraproject.org/wiki/Fedora_Activity_Day_-_FAD Some possible Goals: * Put in place our 2 factor authentication solution. - Enable globally for sudo. - Come up with plan/roadmap for applications 2 factor authentication. - enable more 2nd factors if we only have one working. (yubikey, google authenticator, others?) * Revamp firewall rules to further restrict traffic between machines. * Come up with a better plan for signing servers - In puppet or out of puppet? - On demand vs always on - ssh access, console, 2factor? * Hash out a roadmap or plans around git commit signing. - See if this is something we want to do * Work on FAS security enhancements - backup email address? - security questions? - better gpg integration? - handling for 2 factor auth * Setup a simple IDS of some kind? - Notice non standard traffic in our internal nets * Finish up keys.fedoraproject.org and announce it. * Clean up selinux AVCs and move more things to enforcing. * Your brilliant Fedora Infrastructure security related idea here. Possible dates: last week of Aug, First week of Sept? (This puts us between the Alpha and Beta freezes, and is possibly enough notice to get better airfair/etc rates). somewhere in 2012-08-27 to 2012-09-10 First 2 weeks in Nov? (After F18 is released, before thanksgiving) somewhere in 2012-11-05 to 2012-11-16 Right before next Fudcon? 2013-01-15 to 2013-01-17? Your exciting better dates here. Possible locations: Red Hat HQ in RDU? pros: can probably get a room/network and pull in other RH folks Westford, MA pros: could probably get a room/network and pull in other RH engr folks. Other location here: must be cheap to fly to/stay at, and have a facility we could meet at and use. So, this is more a 'is there enough interest in this to peruse it' type of email. How many folks would be interested in going to something like this? What dates or places would you prefer? Is there another topic that would be a better thing to do than Security? I can think of several more topics if we would prefer something else (Fixing our application logging could be it's own FAD by itself). Thoughts? kevin

10 14

Plan for tomorrow's Fedora Infrastructure meeting (2012-06-21)
by Kevin Fenzi 20 Jun '12

20 Jun '12

The infrastructure team will be having it's weekly meeting tomorrow, 2012-06-21 at 18:00 UTC in #fedora-meeting on the freenode network. Note that I will be gone for this meeting and the one following. Smooge will run the meeting. Suggested topics: #topic New folks introductions and Apprentice tasks. If any new folks want to give a quick one line bio or any apprentices would like to ask general questions, they can do so here. #topic Applications status / discussion Check in on status of our applications: pkgdb, fas, bodhi, koji, community, voting, tagger, packager, dpsearch, etc. If there's new releases, bugs we need to work around or things to note. #topic FAD ? https://fedoraproject.org/wiki/FAD_Infrastructure_Security_2012 #topic Upcoming Tasks/Items #info 2012-06-21 to 2012-07-04 Kevin is off on trains and boats. #info 2012-06-26 Fedora 15 end of life. #info 2012-06-28 Seth at jury duty. #info 2012-07-02 remove people with pkgdb bugzilla issues. #info 2012-07-05 nag fi-apprentices #info 2012-07-12 drop inactive apprentices. #info 2012-08-07 to 2012-08-21 F18 Alpha Freeze #info 2012-08-21 F18 Alpha release. #topic Open Floor Submit your agenda items, as tickets in the trac instance and send a note replying to this thread. More info here: https://fedoraproject.org/wiki/Infrastructure/Meetings#Meetings Thanks kevin

1 0

Error email Alias
by ekoikhyar 20 Jun '12

20 Jun '12

Dears infra team I try to help my friend to email alias fp.o problems but I am having problems. when sending an email to the address contained an error, as below, please help him on this issue fas information: suba(a)fedoraproject.org ----- Delivery to the following recipient failed permanently: suba(a)fedoraproject.org Technical details of permanent failure: Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 550 550 5.1.1 <suba(a)fedoraproject.org>... < suba(a)fedoraproject.org>: Recipient address rejected: User unknown in local recipient table (state 13). -- Regards, ekoikhyar* ***PGP : 4EA60FFA< http://pgp.surfnet.nl:11371/pks/lookup?op=index&search=0x4EA60FFA> t: @FedoraIndonesia | w: http://id.fedoracommunity.org/*

2 1

Dynamic Nagios Configuration
by Christos Triantafyllidis 19 Jun '12

19 Jun '12

Hi all, long ago i volunteered myself to start the work on a tool that will create dynamic Nagios configuration based on external information (i.e. infra-hosts). You can find my work on this at: git://fedorapeople.org/~ctria/DynamicNagiosConfig.git The idea is that there is a main configuration file: DNC.yml which specifies which modules to use to create the configuration. Given that the only information i could get from infra-hosts was the host's information itself this only does host configs but can be easily extended to support services, contacts etc. The execution is simple, checkout the infra hosts repository in sample_configs/infra-hosts folder and execute: ./DNC.py You should get all hosts at standard output. Finally a simple YAML based file module allows overrides to be specified. I'm willing to move this forward so i'd definitely like to hear your comments Regards, Christos PS: This is my first python script from scratch so don't be very hard on me :)

4 6

Upcoming items and reminder that I will be out for a bit
by Kevin Fenzi 19 Jun '12

19 Jun '12

Greetings. First, I thought I would send a reminder that I am going to be out on vacation, starting this thursday (2012-06-21) and arriving back home (2012-07-04). If you need me to do something before I leave, please ask me soon. Because I am an information junkie, I will possibly check emails, but I don't intend to answer anything but super urgent things. Also, I intend to not look back at IRC, so if you say something on IRC while I am gone, don't expect me to know about it. :) Secondly, I thought I would list out pending things if folks want to schedule and get those done while I am gone. ;) Mediawiki 119 updgrade - needs plugins packaged/built. - needs testing in stg. - needs scheduled Migration of lists.fedorahosted.org to new machine - needs fixing up for what went wrong last time - needs scheduling. Migration to hosted01/02 - In my testing, the new gluster has been great. - needs more brutal testing. - needs scheduling. Switch to cgit from gitweb-caching - Setup testing in stg? - Decide about compat redirects or not - schedule Infrastructure Security FAD - I setup a base page at: https://fedoraproject.org/wiki/FAD_Infrastructure_Security_2012 - Needs more details and nailing down what we need for budget. - Perhaps a table of interested person, location, cost to location A, cost to location B, etc. Those are all the shorter term ones. kevin

1 0

Plan for tomorrow's Fedora infrastructure meeting (2012-06-14 at 18UTC)
by Kevin Fenzi 14 Jun '12

14 Jun '12

The infrastructure team will be having it's weekly meeting tomorrow, 2012-06-14 at 18:00 UTC in #fedora-meeting on the freenode network. Suggested topics: #topic New folks introductions and Apprentice tasks. If any new folks want to give a quick one line bio or any apprentices would like to ask general questions, they can do so here. #topic two factor auth status #topic Applications status / discussion Check in on status of our applications: pkgdb, fas, bodhi, koji, community, voting, tagger, packager, dpsearch, etc. If there's new releases, bugs we need to work around or things to note. #topic FAD ? #topic Upcoming Tasks/Items #topic 2012-06-14 smooge out. #topic 2012-06-18 remove people with pkgdb bugzilla issues. #topic 2012-06-21 to 2012-07-04 Kevin is off on trains and boats. #topic 2012-06-26 Fedora 15 end of life. #topic 2012-06-28 Seth at jury duty. #topic 2012-07-05 nag fi-apprentices #topic 2012-07-12 drop inactive apprentices. #topic 2012-08-07 to 2012-08-21 F18 Alpha Freeze #topic 2012-08-21 F18 Alpha release. #topic Open Floor Submit your agenda items, as tickets in the trac instance and send a note replying to this thread. More info here: https://fedoraproject.org/wiki/Infrastructure/Meetings#Meetings Thanks kevin

1 1

Meeting Agenda Item: Introduction Sumit Rai
by sumit 14 Jun '12

14 Jun '12

Hi Everyone, I am Sumit Rai, I live in India (Time Zone +5:30 GMT). I have been using Linux for past two years. I have basic knowledge of Linux. I would like to improve my skills and learn new things by Joining Fedora Infrastructure team. I am interested in system administration work, and writing scripts. I am looking forward to contributing 4 hours per week. I am looking forward to see you people on weekly meeting and learn how the team works. My irc name is sumitrai. Regards, Sumit Rai PS: I am also looking for a mentor/sponsor.

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure June 2012