infrastructure July 2007

infrastructure@lists.fedoraproject.org

53 participants
51 discussions

fedorapeople.org is now available
by Seth Vidal 18 Jul '07

18 Jul '07

Hi Everyone, fedorapeople.org is now available for general use. What is fedorapeople.org?: It is a site where fedora contributors can upload files for sharing out with the world. It is perfect for uploading specfiles, srpms, patches, etc, etc. Each fedora contributor has 150M of quota-controlled space. Users can upload using scp, sftp or rsync. Once uploaded into the users public_html directory the files are available via http at: http://your_username.fedorapeople.org/. To connect to fedorapeople.org just use the ssh key you uploaded to your fedora account and then you can login via ssh to: fedorapeople.org What fedorapeople.org is NOT: - it is not a place for you to upload confidential or copyright-violating files. - it is not a shell for you to login and stay logged into - it is not a place for you to run your favorite proxy of whatever kind - it is not a database server - it is not a mail server - it is not a run-my-favorite-cgi-server - it is not a blog server We've tried our best to minimize and secure the services. Don't make a lot of unreasonable requests asking for us to undo that. :) For reasonable requests please file put them in the fedora infrastructure ticketing system: http://fedoraproject.org/wiki/Infrastructure/Tickets Let us know what breaks, -sv -- fedora-announce-list mailing list fedora-announce-list(a)redhat.com https://www.redhat.com/mailman/listinfo/fedora-announce-list

1 0

Fedora 7 update infofeed (RSS) unavailable
by Imtiaz Rahi 18 Jul '07

18 Jul '07

Upto Fedora Core 6 I used Fedora update and Extras infofeed. Which is not available for Fedora 7 updates. It is so useful to have these new release info in news reader and look at which update I should take. Will this be discontinued for F7? Cheers, Imtiaz Email: imtiaz.rahi(a)fedoraproject.org http://fedoraproject.org/wiki/ImtiazRahi

2 1

Moin update.
by Ray Van Dolson 18 Jul '07

18 Jul '07

Have been doing some Moin testing on my own (was able to get compareable delays after generating ~3500 Moin users on my own wiki). Turns out the #moin-dev guys don't like hotshots (sounds like it is pretty unreliable) and prefer cProfile (which is in Python 2.5). I cooked up this little patch to add cProfile support for the standalone moin server -- maybe useful to someone interested in doing profiling of the wiki later: http://fedorapeople.org/~rayvd/moin/moin-cProfile/ After looking at the profile information, it seems that creating an index/cache of page subscription regexp's pointing to the subscribed users would speed things up a lot. I don't think this would be too bad to write. The #moin-dev guys agree, and would be willing to up-port any patch I make for 1.5.8 into their 1.6 code. They are currently writing a whole new storage subsystem, but obviously it will be some time before it's stable and even longer before Fedora makes use of it would be my guess. See http://fedoraproject.org/wiki/RayVanDolson#Moin for my project notes on this. My own personal wiki contains the most information about what I've tried so far. Ray

2 2

Common problems
by Mike McGrath 16 Jul '07

16 Jul '07

I've been meaning to do this for a long time: http://fedoraproject.org/wiki/Infrastructure/CommonProblems Please add to it and make it not stink :) -Mike

3 12

Python, VCSs, ssh keys and Transifex
by Dimitris Glezos 15 Jul '07

15 Jul '07

Hi all. There is a darkish cloud of security uncertainty in my sky (and Fedora's!), so it's better to discuss this as early as possible to avoid any late notices. I'm working on a web app that will help translation submission by allowing fedora translators (members of cvsl10n group) to commit translations to systems they don't have direct access to. Think: hosted.fpo (svn/hg/git). https://hosted.fedoraproject.org/projects/transifex The idea is that transifex will act as a proxy/mediator for translation commits. A translator will login to the transifex instance running on a host like `translate.fpo`, choose a module, a PO file to upload, and a destination file and click "submit". The system will commit the file for him. Underneath this is achieved by having the VCS admin create a user (eg. fedora-transifex) with a dedicated SSH key, and give it write access to the specific modules accepting translations. The transifex admin will then hook the repo and module up with the system. Each commit will be done by the "fedora-transifex" user, and the actual user's details (name, surname, email, fedora username) will be written in the commit message and Changelog file. Transifex supports filaname filters, so even if a module maintainer can't add ACLs to the repo, he can define them on the transifex side; for example, .*/po/(LINGUAS|Changelog|.*po$). To put things in perspective, if we do this *right*, then *any* remote VCS could be hooked up. In the future, we could add a layer of "approval" before commits, so that the language maintainer (which we probably trust more than a john doe user with cvsl10n access) approves queued messages to be pushed. Or, we could give the option to a dev (for DVCS) to pull instead of the webapp to push. To the implementation details now, transifex will become the client to the remote VCSs. Once the user clicks "submit PO", the webapp should commit (and push). The security question is how do we handle SSH keys? - Where do we store them? Best place would be ~/.ssh/, because not all VCS commands support SSH options to point to a different config file. - Right before running the checkout/pull and checkin/push commands, the environment should be right so that the commands run by the webapp will succeed over SSH. So an option the webapp (just like anyone) will have to "type" the passphrase to unlock the key. Or, use ssh-agent. And probably SELinux. What's the best approach with the minimum compromise risk? - Where are we going to actually store the keys and passphrase? On-disk or in the DB? If we store them encrypted, where do we store the salt? - Do we need a different process running that handles these security requests? Seeking the knowledge and experience of the wise, the humble developer comes to you with seriously cold feet. -d -- Dimitris Glezos Jabber ID: glezos(a)jabber.org, GPG: 0xA5A04C3B http://dimitris.glezos.com/ "He who gives up functionality for ease of use loses both and deserves neither." (Anonymous) --

4 8

mirror out of date
by David Juran 13 Jul '07

13 Jul '07

Hello. I've noticed that the Fedora mirror ftp://falkor.skane.se/pub/mirrors/fedora/linux/updates/7/x86_64/ (listed on http://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f7&arch=x…) is lacking several packages (e.g. vte-0.16.6-1) that where pushed quite some time ago. On a related topic, do we have some kind of script that crawls the mirrors on a regular basis and makes sure they are updated? -- David Juran Sr. Consultant Red Hat +358-504-146348

3 2

Bugzilla links update; page update and redirects required
by Rahul Sundaram 12 Jul '07

12 Jul '07

Hi https://bugzilla.redhat.com/bugzilla/index.cgi has a couple of old link to: http://fedora.redhat.com/download/mirrors.html instead of http://fedoraproject.org/wiki/Distribution and http://fedora.redhat.com/participate/communicate/ instead of http://fedoraproject.org/wiki/Communicate Redirects should probably be setup and the links updated in the bugzilla page. Rahul

2 2

DB Upgrade complete
by Mike McGrath 11 Jul '07

11 Jul '07

With the exception cacti (I'll grab it tomorrow), all databases are now running off of db2. The migration went off very well (with one exception below). The total outage lasted about 30 minutes longer than expected though all services (except for koji) were only down for a few minutes. For those that aren't familiar, this upgrade was for both mysql and postgresql: mysql-server-3.23.58-1 -> mysql-server-5.0.22-2.1 postgresql-server-7.4.6-2.EL3 -> postgresql-server-8.1.9-1.el5 Considering all of the stuff that could have gone wrong, I'm pretty happy. As always with these things keep your eyes open for any strange issues that may pop up. Right now though all sites seem 'up' and I pushed a koji build through. The one bump we had was an ignorant mistake on my part. While doing a final trial run on db2 before the upgrade (yesterday afternoon) I mistook a terminal. Long story short a table drop and a rm lead to us losing all accounts changes between 12:00 and 4:00 p.m. central time. (about 4 hours or so). /me smacks forehead. If you guys hear anything regarding that let me know and we'll work something out (so far all I know of is someone had to re-apply for extras-cvs) So whats left from here? I've updated the ticket: https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/25 Here's a summary: 1) Wait a week for db2 so we know its fine 2) I have a few databases left on db1 that I don't *think* are in use. I'll be sending those to the list soon for verification. If we're not I'll archive and zip them up. They were not migrated to the new box. 3) upgrade db1 4) move mysql services back to db1 (it has less ram then db2. postgresql is hit much heavier so we'll leave it on the box with more processors and ram) 5) Setup a dump from each db to the other. I'd think 4 times / day would be enough, maybe 2 or 3. We'll have to see how much resources it actually takes. We'll have to do some tweaking. Toshio pointed out that our postgres config is pretty vanilla. There are probably many tweaks we can do to it so I've included it in this email. I'm a MySQL guy so any assistance on the postgres side of things would be appreciated. If you have any questions about our database, our real concern is koji, its the most used database in terms of traffic and it is, by far, our largest. Here's the cacti graphs db1: https://admin.fedoraproject.org/cacti/graph_view.php?action=preview&host_id… db2: https://admin.fedoraproject.org/cacti/graph_view.php?action=preview&host_id… -Mike # ----------------------------- # PostgreSQL configuration file # ----------------------------- # # This file consists of lines of the form: # # name = value # # (The '=' is optional.) White space may be used. Comments are introduced # with '#' anywhere on a line. The complete list of option names and # allowed values can be found in the PostgreSQL documentation. The # commented-out settings shown in this file represent the default values. # # Please note that re-commenting a setting is NOT sufficient to revert it # to the default value, unless you restart the postmaster. # # Any option can also be given as a command line switch to the # postmaster, e.g. 'postmaster -c log_connections=on'. Some options # can be changed at run-time with the 'SET' SQL command. # # This file is read on postmaster startup and when the postmaster # receives a SIGHUP. If you edit the file on a running system, you have # to SIGHUP the postmaster for the changes to take effect, or use # "pg_ctl reload". Some settings, such as listen_addresses, require # a postmaster shutdown and restart to take effect. #--------------------------------------------------------------------------- # FILE LOCATIONS #--------------------------------------------------------------------------- # The default values of these variables are driven from the -D command line # switch or PGDATA environment variable, represented here as ConfigDir. #data_directory = 'ConfigDir' # use data in another directory #hba_file = 'ConfigDir/pg_hba.conf' # host-based authentication file #ident_file = 'ConfigDir/pg_ident.conf' # IDENT configuration file # If external_pid_file is not explicitly set, no extra pid file is written. #external_pid_file = '(none)' # write an extra pid file #--------------------------------------------------------------------------- # CONNECTIONS AND AUTHENTICATION #--------------------------------------------------------------------------- # - Connection Settings - listen_addresses = '*' # what IP address(es) to listen on; # comma-separated list of addresses; # defaults to 'localhost', '*' = all port = 5432 max_connections = 200 # note: increasing max_connections costs ~400 bytes of shared memory per # connection slot, plus lock space (see max_locks_per_transaction). You # might also need to raise shared_buffers to support more connections. #superuser_reserved_connections = 2 #unix_socket_directory = '' #unix_socket_group = '' #unix_socket_permissions = 0777 # octal #bonjour_name = '' # defaults to the computer name # - Security & Authentication - #authentication_timeout = 60 # 1-600, in seconds #ssl = off #password_encryption = on #db_user_namespace = off # Kerberos #krb_server_keyfile = '' #krb_srvname = 'postgres' #krb_server_hostname = '' # empty string matches any keytab entry #krb_caseins_users = off # - TCP Keepalives - # see 'man 7 tcp' for details #tcp_keepalives_idle = 0 # TCP_KEEPIDLE, in seconds; # 0 selects the system default #tcp_keepalives_interval = 0 # TCP_KEEPINTVL, in seconds; # 0 selects the system default #tcp_keepalives_count = 0 # TCP_KEEPCNT; # 0 selects the system default #--------------------------------------------------------------------------- # RESOURCE USAGE (except WAL) #--------------------------------------------------------------------------- # - Memory - #shared_buffers = 65536 # min 16 or max_connections*2, 8KB each #temp_buffers = 1000 # min 100, 8KB each #max_prepared_transactions = 5 # can be 0 or more # note: increasing max_prepared_transactions costs ~600 bytes of shared memory # per transaction slot, plus lock space (see max_locks_per_transaction). #work_mem = 1024 # min 64, size in KB #maintenance_work_mem = 16384 # min 1024, size in KB #max_stack_depth = 2048 # min 100, size in KB # - Free Space Map - #max_fsm_pages = 20000 # min max_fsm_relations*16, 6 bytes each #max_fsm_relations = 1000 # min 100, ~70 bytes each # - Kernel Resource Usage - #max_files_per_process = 1000 # min 25 #preload_libraries = '' # - Cost-Based Vacuum Delay - #vacuum_cost_delay = 0 # 0-1000 milliseconds #vacuum_cost_page_hit = 1 # 0-10000 credits #vacuum_cost_page_miss = 10 # 0-10000 credits #vacuum_cost_page_dirty = 20 # 0-10000 credits #vacuum_cost_limit = 200 # 0-10000 credits # - Background writer - #bgwriter_delay = 200 # 10-10000 milliseconds between rounds #bgwriter_lru_percent = 1.0 # 0-100% of LRU buffers scanned/round #bgwriter_lru_maxpages = 5 # 0-1000 buffers max written/round #bgwriter_all_percent = 0.333 # 0-100% of all buffers scanned/round #bgwriter_all_maxpages = 5 # 0-1000 buffers max written/round #--------------------------------------------------------------------------- # WRITE AHEAD LOG #--------------------------------------------------------------------------- # - Settings - #fsync = on # turns forced synchronization on or off #wal_sync_method = fsync # the default is the first option # supported by the operating system: # open_datasync # fdatasync # fsync # fsync_writethrough # open_sync #full_page_writes = on # recover from partial page writes #wal_buffers = 8 # min 4, 8KB each #commit_delay = 0 # range 0-100000, in microseconds #commit_siblings = 5 # range 1-1000 # - Checkpoints - #checkpoint_segments = 3 # in logfile segments, min 1, 16MB each #checkpoint_timeout = 300 # range 30-3600, in seconds #checkpoint_warning = 30 # in seconds, 0 is off # - Archiving - #archive_command = '' # command to use to archive a logfile # segment #--------------------------------------------------------------------------- # QUERY TUNING #--------------------------------------------------------------------------- # - Planner Method Configuration - #enable_bitmapscan = on #enable_hashagg = on #enable_hashjoin = on #enable_indexscan = on #enable_mergejoin = on #enable_nestloop = on #enable_seqscan = on #enable_sort = on #enable_tidscan = on # - Planner Cost Constants - #effective_cache_size = 1000 # typically 8KB each #random_page_cost = 4 # units are one sequential page fetch # cost #cpu_tuple_cost = 0.01 # (same) #cpu_index_tuple_cost = 0.001 # (same) #cpu_operator_cost = 0.0025 # (same) # - Genetic Query Optimizer - #geqo = on #geqo_threshold = 12 #geqo_effort = 5 # range 1-10 #geqo_pool_size = 0 # selects default based on effort #geqo_generations = 0 # selects default based on effort #geqo_selection_bias = 2.0 # range 1.5-2.0 # - Other Planner Options - #default_statistics_target = 10 # range 1-1000 #constraint_exclusion = off #from_collapse_limit = 8 #join_collapse_limit = 8 # 1 disables collapsing of explicit # JOINs #--------------------------------------------------------------------------- # ERROR REPORTING AND LOGGING #--------------------------------------------------------------------------- # - Where to Log - #log_destination = 'stderr' # Valid values are combinations of # stderr, syslog and eventlog, # depending on platform. # This is used when logging to stderr: redirect_stderr = on # Enable capturing of stderr into log # files # These are only used if redirect_stderr is on: log_directory = 'pg_log' # Directory where log files are written # Can be absolute or relative to PGDATA log_filename = 'postgresql-%a.log' # Log file name pattern. # Can include strftime() escapes log_truncate_on_rotation = on # If on, any existing log file of the same # name as the new log file will be # truncated rather than appended to. But # such truncation only occurs on # time-driven rotation, not on restarts # or size-driven rotation. Default is # off, meaning append to existing files # in all cases. log_rotation_age = 1440 # Automatic rotation of logfiles will # happen after so many minutes. 0 to # disable. log_rotation_size = 0 # Automatic rotation of logfiles will # happen after so many kilobytes of log # output. 0 to disable. # These are relevant when logging to syslog: #syslog_facility = 'LOCAL0' #syslog_ident = 'postgres' # - When to Log - #client_min_messages = notice # Values, in order of decreasing detail: # debug5 # debug4 # debug3 # debug2 # debug1 # log # notice # warning # error #log_min_messages = notice # Values, in order of decreasing detail: # debug5 # debug4 # debug3 # debug2 # debug1 # info # notice # warning # error # log # fatal # panic #log_error_verbosity = default # terse, default, or verbose messages #log_min_error_statement = panic # Values in order of increasing severity: # debug5 # debug4 # debug3 # debug2 # debug1 # info # notice # warning # error # panic(off) #log_min_duration_statement = -1 # -1 is disabled, 0 logs all statements # and their durations, in milliseconds. #silent_mode = off # DO NOT USE without syslog or # redirect_stderr # - What to Log - #debug_print_parse = off #debug_print_rewritten = off #debug_print_plan = off #debug_pretty_print = off #log_connections = off #log_disconnections = off #log_duration = off #log_line_prefix = '' # Special values: # %u = user name # %d = database name # %r = remote host and port # %h = remote host # %p = PID # %t = timestamp (no milliseconds) # %m = timestamp with milliseconds # %i = command tag # %c = session id # %l = session line number # %s = session start timestamp # %x = transaction id # %q = stop here in non-session # processes # %% = '%' # e.g. '<%u%%%d> ' #log_statement = 'none' # none, mod, ddl, all #log_hostname = off #--------------------------------------------------------------------------- # RUNTIME STATISTICS #--------------------------------------------------------------------------- # - Statistics Monitoring - #log_parser_stats = off #log_planner_stats = off #log_executor_stats = off #log_statement_stats = off # - Query/Index Statistics Collector - #stats_start_collector = on #stats_command_string = off #stats_block_level = off #stats_row_level = off #stats_reset_on_server_start = off #--------------------------------------------------------------------------- # AUTOVACUUM PARAMETERS #--------------------------------------------------------------------------- #autovacuum = off # enable autovacuum subprocess? #autovacuum_naptime = 60 # time between autovacuum runs, in secs #autovacuum_vacuum_threshold = 1000 # min # of tuple updates before # vacuum #autovacuum_analyze_threshold = 500 # min # of tuple updates before # analyze #autovacuum_vacuum_scale_factor = 0.4 # fraction of rel size before # vacuum #autovacuum_analyze_scale_factor = 0.2 # fraction of rel size before # analyze #autovacuum_vacuum_cost_delay = -1 # default vacuum cost delay for # autovac, -1 means use # vacuum_cost_delay #autovacuum_vacuum_cost_limit = -1 # default vacuum cost limit for # autovac, -1 means use # vacuum_cost_limit #--------------------------------------------------------------------------- # CLIENT CONNECTION DEFAULTS #--------------------------------------------------------------------------- # - Statement Behavior - #search_path = '$user,public' # schema names #default_tablespace = '' # a tablespace name, '' uses # the default #check_function_bodies = on #default_transaction_isolation = 'read committed' #default_transaction_read_only = off #statement_timeout = 0 # 0 is disabled, in milliseconds # - Locale and Formatting - #datestyle = 'iso, mdy' #timezone = unknown # actually, defaults to TZ # environment setting #australian_timezones = off #extra_float_digits = 0 # min -15, max 2 #client_encoding = sql_ascii # actually, defaults to database # encoding # These settings are initialized by initdb -- they might be changed #lc_messages = 'C' # locale for system error message # strings #lc_monetary = 'C' # locale for monetary formatting #lc_numeric = 'C' # locale for number formatting #lc_time = 'C' # locale for time formatting # - Other Defaults - #explain_pretty_print = on #dynamic_library_path = '$libdir' #--------------------------------------------------------------------------- # LOCK MANAGEMENT #--------------------------------------------------------------------------- #deadlock_timeout = 1000 # in milliseconds #max_locks_per_transaction = 64 # min 10 # note: each lock table slot uses ~220 bytes of shared memory, and there are # max_locks_per_transaction * (max_connections + max_prepared_transactions) # lock table slots. #--------------------------------------------------------------------------- # VERSION/PLATFORM COMPATIBILITY #--------------------------------------------------------------------------- # - Previous Postgres Versions - #add_missing_from = off #backslash_quote = safe_encoding # on, off, or safe_encoding #default_with_oids = off #escape_string_warning = off #regex_flavor = advanced # advanced, extended, or basic #sql_inheritance = on # - Other Platforms & Clients - #transform_null_equals = off #--------------------------------------------------------------------------- # CUSTOMIZED OPTIONS #--------------------------------------------------------------------------- #custom_variable_classes = '' # list of custom variable class names

3 8

git.fedoraproject.org
by Roland McGrath 11 Jul '07

11 Jul '07

Just noticed the git server is all kinds of broken for the original uses that worked before the setup got as organized as it is now. git-ls-remote http://git.fedoraproject.org/kernel/roland.git/ git-ls-remote git://git.fedoraproject.org/kernel/roland.git/ If you would like to rename my directories, that is fine. But they need to work by some URL that people can use with git. Thanks, Roland

1 0

ticket problem, and bugzilla problem
by Christian Iseli 11 Jul '07

11 Jul '07

Dear all, I'm kinda stuck here, and would need some help :-) The first part of the problem is that I am trying to grab all the CLOSED Fedora Package Review tickets in bugzilla and: - the XMLRPC interface keeps breaking with "500 Internal Server Error" - the web query page (simple) just sits there, pondering my query... I need to grab those tickets to be able to update my PackageStatus page. So I tried to report the problem through a ticket at: https://admin.fedoraproject.org/tickets/index.pl and I get: Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, webmaster(a)fedora.redhat.com and inform them of the time the error occurred, and anything you might have done that may have caused the error. More information about this error may be available in the server error log. Apache/2.0.52 (Red Hat) Server at 10.8.32.123 Port 8080 ---- Any help appreciated. Regards, Christian

3 4

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure July 2007