infrastructure February 2018

infrastructure@lists.fedoraproject.org

36 participants
38 discussions

Deploying Hubs to our Openshift instance
by Aurelien Bompard 18 Feb '18

18 Feb '18

Hey folks! I would like to try and do the Fedora Hubs deployment in our Openshift instance. The thing is, I have never deployed anything in Openshift, much less using our ansible playbooks. Do you know of a documentation I could read up on to understand what our `openshift/project`, `openshift/secret-file` and `openshift/*` roles work, and how I could use them? I'm trying to read the playbooks for the apps we already have in openshift, but there's too much magic going on for me. Any introduction document would help. I did try to read the F docs, but did not find anything in the SOPs or on the wiki. Thanks! Aurélien

5 8

Unable to push changes (access denied)
by Antonio Trande 16 Feb '18

16 Feb '18

Hello. I can't push changes via GIT for 'coin-or-CoinUtils' package: $ git push Counting objects: 3, done. Delta compression using up to 2 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 418 bytes | 418.00 KiB/s, done. Total 3 (delta 2), reused 0 (delta 0) remote: FATAL: W refs/heads/master rpms/coin-or-CoinUtils sagitter DENIED by fallthru remote: error: hook declined to update refs/heads/master To ssh://pkgs.fedoraproject.org/rpms/coin-or-CoinUtils ! [remote rejected] master -> master (hook declined) error: failed to push some refs to 'ssh://sagitter@pkgs.fedoraproject.org/rpms/coin-or-CoinUtils' -- --- Antonio Trande Fedora Project mailto 'sagitter at fedoraproject dot org' GPG key: 0x5E212EE1D35568BE GPG key server: https://keys.fedoraproject.org/

2 1

A proof-of-concept for delta'ing repodata
by Jonathan Dieter 16 Feb '18

16 Feb '18

<tl;dr> I've come up with a method of splitting repodata into chunks that can be downloaded and combined with chunks that are already on the local system to create a byte-for-byte copy of the compressed repodata. Tools and scripts are at: https://www.jdieter.net/downloads/ </tl;dr> Background: With DNF, we're currently downloading ~20MB of repository data every time the updates repository changes. When casync was released, I wondered if we could use it to only download the deltas for the repodata. At Flock last summer, I ran some tests against the uncompressed repodata and saw a reduction of 30-40% from one day to the next, which seemed low, but was a good starting point. Unfortunately, due to the way casync separates each file into thousands of compressed chunks, building each file required thousands of (serial) downloads which, even on a decent internet connection, took *forever*. When I talked through the idea with Kevin and Patrick, they also pointed out that our mirrors might not be too keen on the idea of adding thousands of tiny files that change every day. The Solution(?): One potential solution to the "multitude of files" problem is to merge the chunks back into a single file, and use HTTP ranges to only download the parts of the file we want. An added bonus is that most web servers are configured to support hundreds of ranges in one request, which greatly reduces the number of requests we have to make. The other problem with casync is that it's chunk separation is naïve, which is why we were only achieving 30-40% savings. But we know what the XML file is supposed to look like, so we can separate the chunks on the tag boundaries in the XML. So I've ditched casync altogether and put together a proof-of-concept (tentatively named zchunk) that takes an XML file, compresses each tag separately, and then concatenates all of them into one file. The tool also creates an index file that tells you the sha256sum for each compressed chunk and the location of the chunk in the file. I've also written a small script that will download a zchunk off the internet. If you don't specify an old file, it will just download everything, but if you specify an old file, it will download the index of the new file and compare the sha256sums of each chunk. Any checksums that match will be taken from the old file, and the rest will be downloaded. In testing, I've seen savings ranging from 10% (December 17 to today) to 95% (yesterday to today). Remaining problems: * Zchunk files are bigger than their gzip equivalents. This ranges from 5% larger for filelists.xml to 300% larger for primary.xml. This can be greatly reduced by chunking primary.xml based on srpm rather than rpm, which brings the size increase for primary.xml down to roughly 30%. * Many changes to the metadata can mean a large number of ranges requested. I ran a check on our mirrors, and three (out of around 150 that had the file I was testing) don't honor range requests at all, and three others only honor a small number in a single request. A further seven didn't respond at all (not sure if that had anything to do with the range requests), and the rest supported between 256 and 512 ranges in a single request. We can reduce the number of ranges requested by always ordering our packages by date. This would ensure that new packages are grouped at the end of the xml where they will be grabbed in one contiguous range. * Zchunk files use zlib (it gives better compression than xz with such small chunks), but, because they use a custom zdict, they are not gz files. This means that we'll need new tools to read and write them. (And I am volunteering to do the work here) The tools: The proof-of-concept tools are all sitting in https://www.jdieter.net/downloads/zchunk-scripts/ They are full of ugly hacks, especially when it comes to parsing the XML, there's little to no error reporting, and I didn't comment them well at all, but they should work. If all you want to do is download zchunks, you need to run dl_zchunk.py with the url you want to download (ending in .zck) as the first parameter. Repodata for various days over the last few weeks is at: https://www.jdieter.net/downloads/zchunk-test/ You may need to hover over the links to see which is which. The downloads directory is also available over rsync at rsync://jdieter.net/downloads/zchunk-test. dl_zchunk.py doesn't show anything if you download the full file, but if you run the command with an old file as the second parameter, it will show four numbers: bytes taken from the old file, bytes downloaded from the new, total downloaded bytes and total uploaded bytes. zchunk.py creates a .zck file. To group chunks by source rpm in primary.xml, run ./zchunk.py <file> rpm:sourcerpm unzchunk.py decompresses a .zck file to stdout I realize that there's a lot to digest here, and it's late, so I know I missed something. Please let me know if you have any suggestions, criticisms or flames, though it might be a few hours before I respond. Jonathan

4 8

Meeting Agenda Item: Introduction Travis Newton
by Travis Newton 15 Feb '18

15 Feb '18

Hello everyone! I've been working as a network admin in the financial industry. I've been in the Red Hat family of distros ever since I discovered Linux (introduced to Fedora Core 2 back in 2004). I've always been comfortable with Fedora and wanted to volunteer and give back to the distro that taught me so much. I was a DevOps Engineer in a full CentOS environment which helped me learn a little bit of Bash, HAProxy, Varnish, RabbitMQ, and a few other tools. I also have a couple of dedicated servers that I manage and learn off of to better myself as a Linux admin. All of my professional background has been with Microsoft products, but I've been bit by the open source bug. I'm hoping to learn as much as I can about better ways to do things and larger infrastructures like Fedora such as HA, build systems, and more. I'm just an information sponge. I have a home lab so I'm really looking towards sharpening my skills. On my todo list are a few items like learning Python and C++ and expanding my networking skills. My IRC handle is greyaxe90 and look forward with chatting and learning with everyone! Travis

1 0

Re: [Rpm-ecosystem] A proof-of-concept for delta'ing repodata
by Igor Gnatenko 14 Feb '18

14 Feb '18

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On Tue, 2018-02-13 at 12:09 +0000, Michael Schroeder wrote: > On Tue, Feb 13, 2018 at 10:52:14AM +0100, Igor Gnatenko wrote: > > This would "break" DNF, because libsolv is assigning Id's by the order of > > packages in metadata. So if something requires "webserver" and there is > > "nginx" > > and "httpd" providing it (without versions), then lowest Id is picked up > > (not > > going into details of this). > > No, this is not correct. Libsolv doesn't use the Id to pick a package, > exactly to be independent on the package order of the repository. Oh, could you link me source code for this part? I tried to look it up, but haven't found. Thanks! - -- - -Igor Gnatenko -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEhLFO09aHZVqO+CM6aVcUvRu8X0wFAlqEgP8ACgkQaVcUvRu8 X0z1zw//Vpu6NgZEHZUyes7Bjb5ABmIlP5b9XCo5d0GHAWAfq761eFV4qot/y8H2 ErLDA4+OttOgag+PWNLo5Jpx2pN39r0ccqTWpISfb7lMpumKhlqBAvojSu6tM1nr 9Z3FtBm4EG7pSyjzEQW6Z2BmLovLQOUUncoCR0hbntOB5XK3X/96lkfxmW/8UqRE 62g2SCJHNjz4w74Xc23/ZAkCZtAby6Dw7KJ0DwyPEkl4bRM2GAJhZsnTw8EVHgnH q4v1aSmYezG4GvKNAF/mepMrKJVxMD1JdkaOsFpzaKqFU/QjNpfwDlEGPzMvnCgA ZmVQ7SYMFPGg8KiTUfsIWExm5qTEfmrN0e7RmZgHtFmS/H+h1H40ZUGFfd60k+Q/ pr5z1phc2bZ8iew2SscORj8IYe4xT3oOWbcabLGL1w1aM6QobFckavSL39K1Rpfz YyKGqYOKhENlduhghYvB8+aBM357MVZTrJM5+x9u9NmMiuvH7e/bXQPxhd7LzTCS ESsO7ryWh5kypccHgizrVLKliz+S3Nl+kf9oQLY98gVyxbMNkhfYDkp0gUY46Zvo 0LoTZ8QcajYiryJzOfi4C+7sJpozowxmasHYb15miCspnA1O3PIa82MgrlS77E8B 8gkbCckeGCxaAa3dXa2JgpENBznQ7DgAZaraQT+ZSxb1jF7QPW8= =O1D0 -----END PGP SIGNATURE-----

1 0

[release] - fedora-packages 4.1.0
by Clement Verna 14 Feb '18

14 Feb '18

Greetings, A new release of fedora-packages is available. This release enable the packages to be searched by the package point of contact. The full changelog is available in here[0]. It is currently deployed in stg[1] and it will be deployed in prod in next coming days. [0] - https://github.com/fedora-infra/fedora-packages/blob/develop/CHANGELOG.rst#… [1] - https://apps.stg.fedoraproject.org/packages Clément

2 1

Meeting Agenda Item: Introduction Dennis Otugo
by Wesley Otugo 13 Feb '18

13 Feb '18

Hello World! My name is Dennis W. Otugo. I am from Lagos, Nigeria. My IRC Handle is wesleyotugo. I am familiar with Python & Javascript. I do not have any certification yet, I am currently training for one. I am physically involved in various tech communities but not nothing open source as there is no available community here. But, I am involved with CommOps. I want to learn as much as I possibly can. I would like to work on any as to gain experience as much as I possibly can. Thanks.

1 0

[release] mdapi: 2.10.4 and 2.10.5
by Pierre-Yves Chibon 12 Feb '18

12 Feb '18

Good Morning Everyone, I cut two new releases of mdapi this morning: 2.10.4 and 2.10.5 the last one fixing an issue for fedora-packages that Clément ran into. Here are the corresponding changelogs: * Fri Feb 09 2018 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 2.10.5-1 - Update to 2.10.5 - Fix the srcpkg endpoint by making it search for a direct match before trying the sub-packages * Fri Feb 09 2018 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 2.10.4-1 - Update to 2.10.4 - Fix the content_type header on JSONP output This is currently happily running in stg. Happy querying! Pierre

1 1

Outage: Switch Outage - 2018-02-11 15:00 UTC
by Stephen John Smoogen 11 Feb '18

11 Feb '18

There will be an outage starting at 2018-02-11 15:00 UTC, which will last approximately 2 hours. To convert UTC to your local time, take a look at https://fedoraproject.org/wiki/UTCHowto or run: date -d '2018-02-11 15:00 UTC' Reason for outage: The network switches in our PHX2 colocation require a firmware upgrade to deal with false overheating problems. The updates should be short 5 to 10 minute disruptions per switch. Affected Services: Bodhi - https://admin.fedoraproject.org/updates/ Buildsystem - http://koji.fedoraproject.org/ GIT / Source Control Docs - https://docs.fedoraproject.org/ Email system Fedora Account System - https://admin.fedoraproject.org/accounts/ Fedora Community - https://admin.fedoraproject.org/community/ Fedora People - https://fedorapeople.org/ Main Website - https://fedoraproject.org/ Mirror Manager - https://admin.fedoraproject.org/mirrormanager/ Package Database - https://admin.fedoraproject.org/pkgdb/ QA Services Secondary Architectures Spins - https://spins.fedoraproject.org/ Translation Services - https://translate.fedoraproject.org/ Wiki - https://fedoraproject.org/wiki/ Unaffected Services: Torrent - https://torrent.fedoraproject.org/ Mirror List - https://mirrors.fedoraproject.org/ Start - https://start.fedoraproject.org/ Ticket Link: Contact Information: Please join #fedora-admin in irc.freenode.net or add comments to the ticket for this outage above. -- Stephen J Smoogen.

2 1

[PATCH] - Fix fedora-packages Sources tab
by Clement Verna 11 Feb '18

11 Feb '18

Hi all, I have attached a patch to allow httpd to clone the git repo of package so that fedora-packages can display the spec file and patches. Thanks Clément --- PATCH ----- From 5b818a229d3dfdcfc0d880ce45b82ed093eba73d Mon Sep 17 00:00:00 2001 From: Clement Verna <cverna(a)tutanota.com> Date: Fri, 9 Feb 2018 17:16:08 +0100 Subject: [PATCH 1/1] Allow httpd to write the git repo used by the source tab The Source tab is using git repository that are cloned under /var/cache/fedoracommunity/git.fp.o. This commit allow httpd to access this folder. Signed-off-by: Clement Verna <cverna(a)tutanota.com> --- roles/packages3/web/files/packages-httpd.conf | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/roles/packages3/web/files/packages-httpd.conf b/roles/packages3/web/files/packages-httpd.conf index 4d255a9bc..ae873c3ac 100644 --- a/roles/packages3/web/files/packages-httpd.conf +++ b/roles/packages3/web/files/packages-httpd.conf @@ -61,6 +61,10 @@ WSGIScriptAlias /packages /usr/share/fedoracommunity/production/apache/fedoracom WSGIProcessGroup fedoracommunity </Directory> +<Directory /var/cache/fedoracommunity/git.fedoraproject.org> + Require all granted +</Directory> + <Directory /var/cache/fedoracommunity/packages/icons/> # If someone tries to access an icon that doesn't exist, # then send them to the default icon. This is used by -- 2.14.3

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure February 2018