infrastructure November 2006

infrastructure@lists.fedoraproject.org

29 participants
41 discussions

Meeting tomorow
by Mike McGrath 22 Nov '06

22 Nov '06

So tomorow we normally have a meeting in #fedora-admin. I have a hunch many people won't be there because of Thanksgiving but in the interest of being international I'd like to encourage those that can be there to come and meet and discuss. Take minutes and send them to the list and add them to the wiki. http://fedoraproject.org/wiki/Infrastructure/Meetings -Mike

1 0

Possibly Unneeded Database Attributes
by TomLy 21 Nov '06

21 Nov '06

While straightening out the fedoraPerson schema I'm running into attributes in the current accountsDB that aren't used or are all the same value for all users. My plan is to get rid of these if possible. We can add attributes later if needed. Here they are in no particular order: internal_comments -- this is always blank right now wiki_prefs -- also always blank orts_orgid -- always Fedora Comments, concerns, chairs?? ~tom

4 5

Indroducing Myself
by Jeffrey Ollie 21 Nov '06

21 Nov '06

Hello everyone, I thought that I'd take a moment to introduce myself. I'm a Linux and network administrator for a community college in central Iowa. I've been using various forms of Red Hat/Fedora Linux for 10+ years, and am proficient in C, Python, shell, some Perl. I currently maintain ~21 packages in Fedora Extras. My primary interest in joining the infrastructure list is helping with the development of the new version control system (yes, I'm aware that a dedicated list for that purpose is in the works). I mostly use Subversion for my own tasks but I'm interested in helping with Mercurial, Bazzaar-NG, and Git. I may volunteer for other tasks as time/interest permits. Jeff

1 0

Re: CVS is busted
by Mike McGrath 21 Nov '06

21 Nov '06

On 18 Nov 2006 09:31:26 -0600, Jason L Tibbitts III <tibbs(a)math.uh.edu> wrote: > >>>>> "AK" == Avi Kivity <avi(a)argo.co.il> writes: > > AK> Just curious: why the read-only mount? shouldn't the RAID have > AK> continued in degraded mode? > > Probably because something else bad happened that just completely > screwed up the SCSI bus and corrupted data on multiple disks. > > - J< > We're talking about multiple failures across multiple drives, possibly a backplane. Here's the current plan. 1) Move proxy 3-4 into the f.rh.c cluster so we can take our new dells back. 2) Grab one of the new Dells and build the new cvs box. This will allow us to A) trust the hardware (we're all a little wary about the current cvs box) and B) build a new box with atleast access to the old box if we're missing something. It will also allow us greater capacity with regards to future growth and the whole FC+FC=Fedora thing. 3) Restore backups to the new cvs box. 4) test test test 5) Release to the wild and fix bugs as needed. 6) Take the old cvs box and run full diagnostics before we rebuild it (it'll be come one of our db servers, either primary or backup) Right now mgalgoci is working working on steps 1 and 2. When they are done I'll be on step 3 and we'll need a few people for 4. We'll probably discuss in #fedora-extras when the time comes. Bottom line, this sucks but we're working on it. Should be up and better than ever by Monday. -Mike

3 6

DRAFT: CVS Restored, Help Needed in Verification
by Warren Togami 20 Nov '06

20 Nov '06

Hey folks, this is a draft that I will soon be sending to fedora-devel-list and fedora-extras-list. Anything should be corrected? ========= DRAFT BEGINS HERE ========== Sometime Early Friday, November 17th we suffered some kind of catastrophic multi-disk or SCSI backplane failure in cvs.fedora.redhat.com. Thanks to volunteer efforts of Fedora Infrastructure and tremendous help from Red Hat IS/IT, we are now nearing restoration of cvs.fedora.redhat.com from a backup on a powerful new server graciously donated by Dell. We are restoring from a backup that began November 16th, 2006 at 8:01PM MST (Friday, November 17, 2006 at 03:01) and finished 100 minutes later. Because this backup is not an exact snapshot, it is possible that checkins that happened after 8:01PM are already in the restored data. https://www.redhat.com/archives/fedora-extras-commits/2006-November/date.ht… We require your help to verify that all checkins are properly in /cvs/extras and everything generally is working as expected. Please pay careful attention to everything that changed in November 16th and 17th according to the mailing list archive. My personal records indicate that no CVS imports happened after this backup, so we are fine in that regard. PACKAGE OWNERS - especially kevin, corsepiu, remi and qspencer are to verify and fix their own CVS modules, or explicitly identify someone else to do it on their behalf. EVERYONE ELSE - please keep an eye on CVS. SCREAM LOUDLY on fedora-extras-list if you see anything that is wrong. Sorry about any inconvenience or disruption that this unexpected problem may have caused. Everyone owes a beer to the Fedora Infrastructure and Red Hat IS/IT teams. =) Thank you for supporting the Fedora Project. Warren Togami wtogami(a)redhat.com

1 0

Metalinker and Fedora
by Rahul Sundaram 20 Nov '06

20 Nov '06

Hi The infrastructure team can look at implementing this. Having a set of metalinks along linked from the download area is no big deal and generating them is easy. Not sure if anything else is required here. http://www.linux.com/article.pl?sid=06/11/01/1641247 Rahul

3 4

cvs update
by Matthew Galgoci 20 Nov '06

20 Nov '06

Some time on friday, cvs-int.fedora.phx.redhat.com sustained undetermined storage problems and resulting filesystem corruption. As best I can figure, we had a one drive in a raid6 array drop offline, and another disk in that array emit scsi errors. Now, you're probably thinking, this is raid6, it should have been able to sustain losing two disks and keep on going. Well, you're right and you're wrong. If two disks had simply dropped out of the array, we'd be fine. That wasn't the case however. Somewhere in the equation is data corruption. raid is great up until your hardware corrupts the data. To support this claim, all you need to do is realize that we sustained numerous ext3 errors and had the journal abort, and the root fs went read-only. I did my level best to revive the system on friday and saturday. I was able to get it pxe booted onto rescue media, which helped recovery immensely. I took numerous screen shots to chronical what I went through as I attempted to recover the raid6 arrays and the logical volumes. http://people.redhat.com/~mgalgoci/cvs-int.jpg http://people.redhat.com/~mgalgoci/cvs-int2.jpg http://people.redhat.com/~mgalgoci/cvs-int3.jpg http://people.redhat.com/~mgalgoci/cvs-int4.jpg http://people.redhat.com/~mgalgoci/fedora-cvs5.jpg http://people.redhat.com/~mgalgoci/fedora-cvs6.jpg http://people.redhat.com/~mgalgoci/fedora-cvs8.jpg http://people.redhat.com/~mgalgoci/fedora-cvs9.jpg http://people.redhat.com/~mgalgoci/fedora-cvs10.jpg http://people.redhat.com/~mgalgoci/fedora-cvs11.jpg http://people.redhat.com/~mgalgoci/fedora-cvs12.jpg http://people.redhat.com/~mgalgoci/fedora-cvs13.jpg http://people.redhat.com/~mgalgoci/fedora-cvs14.jpg http://people.redhat.com/~mgalgoci/fedora-cvs15.jpg http://people.redhat.com/~mgalgoci/fedora-cvs18.jpg http://people.redhat.com/~mgalgoci/fedora-cvs17.jpg http://people.redhat.com/~mgalgoci/fedora-cvs16.jpg http://people.redhat.com/~mgalgoci/fedora-cvs19.jpg http://people.redhat.com/~mgalgoci/fedora-cvs20.jpg After #20, I said the hell with it, time to move on. We've installed one of the new Dell 2950 machines that Dell was kind enough to donate to the Fedora Project. Mike McGrath is in the process of updatifying and restorifying the data from backups. I have a Dell tech coming on site again today to do some more work on the old new cvs-int server. I think we know what the issues are on it and we'll have it usable again in the next day or so. In the mean time, I think we need to take a look at all the Dell fedora boxes and check the scsi drives in them. There are known issues with certain drive firmware that cause drives to go offline and report spurrious errors. The relevant Dell update is here: http://support.us.dell.com/support/downloads/download.aspx?c=us&cs=555&l=en… We'll need downtime and hands on site to do this update. I'm sure Stacy will be able to assist. -- Matthew Galgoci GIS Production Operations Red Hat, Inc 919.754.3700 x44155

2 1

Fwd: Host DOWN alert for proxy2!
by Mike McGrath 20 Nov '06

20 Nov '06

Buhhh, anyone know what these are from? -Mike ---------- Forwarded message ---------- From: nagios(a)fedoraproject.org <nagios(a)fedoraproject.org> Date: Nov 19, 2006 8:53 PM Subject: Host DOWN alert for proxy2! To: mmcgrath(a)fedoraproject.org ***** Nagios ***** Notification Type: PROBLEM Host: proxy2 State: DOWN Address: proxy2.fedora.phx.redhat.com Info: CRITICAL - Plugin timed out after 10 seconds Date/Time: Sun Nov 19 19:53:46 MST 2006

4 4

OpenSUSE Build Service?
by Jeroen Janssen 19 Nov '06

19 Nov '06

Hi, I'm not sure if this is the correct Fedora mailinglist, but has anyone had a look at the OpenSuSe Build Service pages ( http://en.opensuse.org/Build_Service )? There are two presentations (from FOSDEM) about it and it seems they have also done some experimenting with building non SuSe distributions with this build system (like Fedora Core 4). Is this something that also could benefit the Fedora Project? Best regards, Jeroen Janssen

5 5

Project hosting thoughts
by Jesse Keating 19 Nov '06

19 Nov '06

Last night I decided I wanted more of a project presence for pungi than an hg web interface. So I started exploring options. 108 was a disaster, I spent 30 minutes trying to find where to start a project, only to find out that there is no way to find that out, you have to ask somebody and then get direct URLS that aren't linked anywhere. Plus 108 doesn't support hg. So I went looking to setup my own software, because damned if I'm going to use Sourceforge, or anything based on it. Trac software has been around for a while, and I've stumbled across a few projects using it. Reading up on their website http://trac.edgewall.org/ the trac feature list looked pretty neat. More interestingly each project within a trac environment can have its own source control mechanism, and the have support for cvs, svn, hg, bzr, darcs, and an experimental git. The git support isn't all that great, but really, what the SCM support does is just provide a web view of the files. The more important thing is that you could get one of these source control repos at project creation time to do with as you see fit. Other things I like about trac, the wiki component uses the same markup (or seems to thus far) as MoinMoin. This will make it very easy to take content from a project and move it into the Fedora wiki should the need/desire arise. Also users wouldn't have to learn yet another markup language. The ticketing system is pretty simple and easy to use, wiki markup language works here too should you want it. Tickets can easily be set to block Milestones, and a Milestone view shows you the list of milestones and an easy view to see what tickets are blocking a milestone, or how many blockers have been fixed (and thus how close a milestone is to completion). There is also a 'timeline' view that shows you a running history of project activity, such as wiki edits, ticket creation/resolution, scm checkins, etc... with rss capability. This would make it very easy to aggregate and track the progress of a project. Some things that I'd like to see investigated: How decent is the support for multiple projects using one trac install? Can we use the Fedora account system as a web authentication (trac just uses web auth as far as I can tell) How easy is it to create a web 'project creation' page that would handle the initial setup of a project? As a proof of concept, is there any way I can make use of a xen guest to setup a trac instance for pungi? Some of the problems with using Trac would involve that it has to be installed on the same system as the SCM (at least for hg and git) or else regular syncs would have to be made with the actual scm location. But honestly this might not be so bad to separate out our 'hosted' project SCMs from our Fedora SCMs. Trac would run just fine on CentOS/RHEL4 but would also work on FC6 for a xen guest (much like CentOS/RHEL5) If we do setup a xen guest with a public IP, can we also set it to something like 'hosted.fedoraproject.org' ? -- Jesse Keating Release Engineer: Fedora

4 5

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure November 2006