Mirrormanager thoughts

List overview All Threads
Download

newer

older

FI Future

New accounts LDAP server running

Matt Domsch

1 Jan 2007 1 Jan '07

10:20 p.m.

Just a quick status on where we're at with the mirror manager system. Farshad, Mike, Luke, and I have all spent some time thinking about this.

Luke started, and I've added to, a page of what we'd like to see in a mirror management system. See http://fedoraproject.org/wiki/Infrastructure/MirrorManagement. Please review this and update/change as you like.

Farshad has made a good start implementing parts of this in TurboGears. I've asked for a Trac project into which his source code can be put so we can collaborate on it. I want to make sure we get the schema right for what we need to accomplish before we spend too much time coding.

Before we unleash it to we'll need to tie this into the Fedora Account System for user authentication, but that can be tackled in parallel to the rest of the system development.

Thanks, Matt

-- Matt Domsch Software Architect Dell Linux Solutions linux.dell.com & www.dell.com/linux Linux on Dell mailing lists @ http://lists.us.dell.com

Show replies by date

Mike McGrath

2 Jan 2 Jan

8:34 a.m.

On 1/1/07, Matt Domsch Matt_Domsch@dell.com wrote:

...

Before we unleash it to we'll need to tie this into the Fedora Account System for user authentication, but that can be tackled in parallel to the rest of the system development.

There were some discussions that this should not tie into the Fedora Account System though I can't remember what arguments there were for or against, anyone remember?

-Mike

seth vidal

8:42 a.m.

On Tue, 2007-01-02 at 08:34 -0600, Mike McGrath wrote:

...

On 1/1/07, Matt Domsch Matt_Domsch@dell.com wrote:

...
Before we unleash it to we'll need to tie this into the Fedora Account System for user authentication, but that can be tackled in parallel to the rest of the system development.

There were some discussions that this should not tie into the Fedora Account System though I can't remember what arguments there were for or against, anyone remember?

mirror admins don't need to sign a CLA or anything else to be mirror admins. And a fair number of them may resent it.

-sv

Jeffrey Ollie

9 a.m.

On Tue, 2007-01-02 at 09:42 -0500, seth vidal wrote:

...

On Tue, 2007-01-02 at 08:34 -0600, Mike McGrath wrote:

...
On 1/1/07, Matt Domsch Matt_Domsch@dell.com wrote:

...
Before we unleash it to we'll need to tie this into the Fedora Account System for user authentication, but that can be tackled in parallel to the rest of the system development.

There were some discussions that this should not tie into the Fedora Account System though I can't remember what arguments there were for or against, anyone remember?

mirror admins don't need to sign a CLA or anything else to be mirror admins. And a fair number of them may resent it.

Does signing the CLA need to be a prerequisite for getting an account in the Fedora Account System? In addition to mirror admins, that would be useful for the hosted projects site.

Jeff

Jesse Keating

9:15 a.m.

On Tuesday 02 January 2007 10:00, Jeffrey C. Ollie wrote:

...

Does signing the CLA need to be a prerequisite for getting an account in the Fedora Account System? In addition to mirror admins, that would be useful for the hosted projects site.

CLA is not a required part of a Fedora Account. It's required for things like package CVS access, EditGroup, and various other things. It could be made not mandatory for mirror admins.

I'd like to keep it mandatory or something similar to the CLA for hosted projects, to CYA should the project admin do something stupid.

-- Jesse Keating Release Engineer: Fedora

Matt Domsch

1 p.m.

On Mon, Jan 01, 2007 at 10:20:13PM -0600, Matt Domsch wrote:

...

Just a quick status on where we're at with the mirror manager system. Farshad, Mike, Luke, and I have all spent some time thinking about this.

Farshad has made a good start implementing parts of this in TurboGears. I've asked for a Trac project into which his source code can be put so we can collaborate on it.

With thanks to Mike and Jesse, https://hosted.fedoraproject.org/projects/mirrormanager/wiki/WikiStart now has a git repository with Farshad's code in it.

Thanks, Matt

-- Matt Domsch Software Architect Dell Linux Solutions linux.dell.com & www.dell.com/linux Linux on Dell mailing lists @ http://lists.us.dell.com

Matt Domsch

7 Jan 7 Jan

11:38 p.m.

https://hosted.fedoraproject.org/projects/mirrormanager/wiki/WikiStart

I've redone the mirrormanager schema and initialization data. I haven't redone the controllers or templates at all yet, so this won't run except in the tg-admin shell and tg-admin toolbox (catwalk is nice). Please take a look. I append the schema file for review. The hierarchy is as follows:

Site is the unit of administrative control. Sites have SiteAdmins which will be account names in the Fedora Account System. Sites have one or more Hosts, which are machines under the control of the Site. Sites also provide SiteIPs which are used for the rsync ACLs. These may or may not match Host IPs.

Some Hosts offer a private rsync port for other mirrors, particularly useful for disseminating embargoed releases. Can also be used to set up mirror tiering. This is described in the HostPrivateRsync block.

Some Hosts are private (e.g. within a company), but could still benefit from the geoip-like redirection, only using CIDR blocks rather than geoip. That's described in the HostIPBlocks.

Arch is architectures, there are 4 at present: source, i386, x86_64, and ppc.

Content is the highest level unit of data that can be synced. These look like fedora-core-6-i386.

Mirror is the combination of a Host and a Content. A Mirror may have a subset of the data provided by the Content.

Each Mirror may expose its data to end users via one or more MirrorURLs. Notably, via http, ftp, and rsync simultaneously, though the URLs for each may differ based on the services setup (which is annoying but true). Checking one of these is sufficient to know that the Mirror content exists and is current.

Your comments would be appreciated.

Thanks, Matt

-- Matt Domsch Software Architect Dell Linux Solutions linux.dell.com & www.dell.com/linux Linux on Dell mailing lists @ http://lists.us.dell.com from sqlobject import * from turbogears.database import PackageHub hub = PackageHub("mirrors") __connection__ = hub from mirrors.identity_models import * class Site(SQLObject): name = StringCol(alternateID=True) robot_email = StringCol(default=None) admin_active = BoolCol(default=False) user_active = BoolCol(default=True) private = BoolCol(default=False) admins = MultipleJoin('SiteAdmin') hosts = MultipleJoin('Host') ips = MultipleJoin('SiteIP') class SiteAdmin(SQLObject): username = StringCol() site = ForeignKey('Site') # IP addresses to go in the rsync ACLs class SiteIP(SQLObject): site = ForeignKey('Site') address = StringCol() class Host(SQLObject): name = StringCol() site = ForeignKey('Site') country = StringCol(default=None) internet2 = BoolCol(default=False) pull_from = ForeignKey('Site') mirrors = MultipleJoin('Mirror') private_rsyncs = MultipleJoin('HostPrivateRsync') ip_blocks = MultipleJoin('HostIPBlocks') # for other mirrors to pull from in tiering class HostPrivateRsync(SQLObject): host = ForeignKey('Host') url = StringCol() user = StringCol(default=None) password = StringCol(default=None) # some hosts only serve some IP CIDR blocks # such as private mirrors behind # company firewalls. This lets them first try # the global yum mirrorlist redirector which will # return them their local mirror URL class HostIPBlocks(SQLObject): host = ForeignKey('Host') cidr = StringCol() class Arch(SQLObject): name = StringCol(alternateID=True) class Content(SQLObject): # e.g. fedora-core-6-i386 name = StringCol(alternateID=True) # e.g. core, extras, updates, updates-testing, test, ... category = StringCol() # 5, 6, development version = StringCol() # default subdir starting below / # e.g. 'pub/fedora/core/6/$ARCH/' path = StringCol() canonical = StringCol() arch = ForeignKey('Arch') # these are paths below $ARCH/ # e.g. iso/, os/, debuginfo/ # of course Extras doesn't have these ;-( isos = StringCol(default=None) binarypackages = StringCol(default=None) repodata = StringCol(default=None) debuginfo = StringCol(default='debug/') repoview = StringCol(default=None) mirrors = MultipleJoin('Mirror') # one per Host per Content # fortunately these will be created/modified # by a crawler program rather than manually class Mirror(SQLObject): host = ForeignKey('Host') content = ForeignKey('Content') urls = MultipleJoin('MirrorURL') # sync status dvd_isos_synced = BoolCol(default=False) cd_isos_synced = BoolCol(default=False) binarypackages_synced = BoolCol(default=False) debuginfo_synced = BoolCol(default=False) repoview_synced = BoolCol(default=False) # One per protocol/path one can get at this same data # checking one MirrorURL is the same as checking them all class MirrorURL(SQLObject): mirror = ForeignKey('Mirror') # The equivalent of the dir structure on the masters # e.g. http://mirrors.kernel.org/fedora/core/6/i386/os/ path = StringCol(default=None)

Matt Domsch

8 Jan 8 Jan

4:06 p.m.

On Sun, Jan 07, 2007 at 11:38:31PM -0600, Matt Domsch wrote:

...

https://hosted.fedoraproject.org/projects/mirrormanager/wiki/WikiStart

I've redone the mirrormanager schema and initialization data. I haven't redone the controllers or templates at all yet, so this won't run except in the tg-admin shell and tg-admin toolbox (catwalk is nice). Please take a look. I append the schema file for review. The hierarchy is as follows:

More schema updates. Three new objects.

Product is a top-level content name, e.g. 'rhel' or 'fedora'.

Products can have many ProductVersions. Some versions are 'test' versions (e.g. 6.90). This needed as the path to fedora-test-6.90 looks like fedora/linux/core/test/6.90/ while the path to fedora-core-6 looks like fedora/linux/core/6/ (the extra 'test' indirection there...)

Category is meant to be able to capture the standard directory structure for 'core', 'extras', 'updates', 'updates-testing', 'test', and 'epel' cleanly. It uses $VERSION and $ARCH in the path names to be replaced when creating a Content.

...

Content is the highest level unit of data that can be synced.

Content refers to the combination of a Category, a ProductVersion, and an Arch, and is still the highest level unit of data that can be synced. These have names like fedora-core-6-i386. (product, category, version, arch).

The other objects are unchanged fundamentally. Your comments would be appreciated.

Thanks, Matt

-- Matt Domsch Software Architect Dell Linux Solutions linux.dell.com & www.dell.com/linux Linux on Dell mailing lists @ http://lists.us.dell.com from sqlobject import * from turbogears.database import PackageHub hub = PackageHub("mirrors") __connection__ = hub from mirrors.identity_models import * class Site(SQLObject): name = StringCol(alternateID=True) robot_email = StringCol(default=None) admin_active = BoolCol(default=False) user_active = BoolCol(default=True) private = BoolCol(default=False) admins = MultipleJoin('SiteAdmin') hosts = MultipleJoin('Host') ips = MultipleJoin('SiteIP') class SiteAdmin(SQLObject): username = StringCol() site = ForeignKey('Site') # IP addresses to go in the rsync ACLs class SiteIP(SQLObject): site = ForeignKey('Site') address = StringCol() class Host(SQLObject): name = StringCol() site = ForeignKey('Site') country = StringCol(default=None) internet2 = BoolCol(default=False) pull_from = ForeignKey('Site') mirrors = MultipleJoin('Mirror') private_rsyncs = MultipleJoin('HostPrivateRsync') ip_blocks = MultipleJoin('HostIPBlocks') # for other mirrors to pull from in tiering class HostPrivateRsync(SQLObject): host = ForeignKey('Host') url = StringCol() user = StringCol(default=None) password = StringCol(default=None) # some hosts only serve some IP CIDR blocks # such as private mirrors behind # company firewalls. This lets them first try # the global yum mirrorlist redirector which will # return them their local mirror URL class HostIPBlocks(SQLObject): host = ForeignKey('Host') cidr = StringCol() class Arch(SQLObject): name = StringCol(alternateID=True) class Category(SQLObject): # e.g. core, extras, updates, updates-testing, test, ... name = StringCol(alternateID=True) # with $VERSION and $ARCH for later substitution path = StringCol() canonicalhost = StringCol() sourcepkgpath = StringCol() sourceisopath = StringCol(default=None) contents = MultipleJoin('Content') class Product(SQLObject): name = StringCol(alternateID=True) versions = MultipleJoin('ProductVersion') class ProductVersion(SQLObject): name = StringCol() product = ForeignKey('Product') isTest = BoolCol(default=False) class Content(SQLObject): # e.g. fedora-core-6-i386 name = StringCol(alternateID=True) category = ForeignKey('Category') # 5, 6, development version = ForeignKey('ProductVersion') # default subdir starting below / # e.g. 'pub/fedora/core/6/$ARCH/' path = StringCol() arch = ForeignKey('Arch') # these are paths below $ARCH/ # e.g. iso/, os/, debuginfo/ # of course Extras doesn't have these ;-( isos = StringCol(default=None) binarypackages = StringCol(default=None) repodata = StringCol(default=None) debuginfo = StringCol(default='debug/') repoview = StringCol(default=None) mirrors = MultipleJoin('Mirror') # one per Host per Content # fortunately these will be created/modified # by a crawler program rather than manually class Mirror(SQLObject): host = ForeignKey('Host') content = ForeignKey('Content') urls = MultipleJoin('MirrorURL') # sync status dvd_isos_synced = BoolCol(default=False) cd_isos_synced = BoolCol(default=False) binarypackages_synced = BoolCol(default=False) debuginfo_synced = BoolCol(default=False) repoview_synced = BoolCol(default=False) # One per protocol/path one can get at this same data # checking one MirrorURL is the same as checking them all class MirrorURL(SQLObject): mirror = ForeignKey('Mirror') # The equivalent of the dir structure on the masters # e.g. http://mirrors.kernel.org/fedora/core/6/i386/os/ path = StringCol(default=None)

6323

Age (days ago)

6329

Last active (days ago)

infrastructure@lists.fedoraproject.org

7 comments

5 participants

tags (0)

participants (5)

Jeffrey Ollie
Jesse Keating
Matt Domsch
Mike McGrath
seth vidal