infrastructure March 2018

infrastructure@lists.fedoraproject.org

38 participants
74 discussions

Goodbye nvr.rsplit('-', 2), hello modularity
by Patrick Uiterwijk 19 Mar '18

19 Mar '18

Hi all, If you maintain any application in Fedora Infra (or outside) that tries to parse things out of RPM names, you might be interested in this. For those wondering where I've been spending most of my time the last few weeks: I've been deep in the internals of Bodhi fixing all kinds of issues I've found between it and Modularity (turns out enabling composing was just the beginning). At some point, I was informed that for modules, they are throwing all kinds of old assumptions out of the door. One of those is the good old Name-Version-Release in RPM land. Modules do not have an NVR, they have an NSVC: Name, Stream, Version, Context. Name is the name of the module ("nodejs"). Stream is a stream, like "6", "7" or "8", these are crucial for the parallel availability that modules are trying to accomplish. Version is a date/time stamp, used to indicate new versions in the same name-stream. Context is a hash of some information in the built module that makes it so that the nodejs-6-20170314 built for Fedora 28 has a different identifier than the one built for Fedora 29. There are some additional fields for installed things (arch and profile), but those aren't really important for things trying to parse/show module names I think. One of the interesting things they wanted to also allow: dashes in streams. As a consequence, when you get an N-S-V.C as modules are represented in Koji builds, doing a .rsplit('-', 2) will not give you Name, Stream, Version.Context per se. You could totally have a module called nodejs-my-stream-5-20170314.abcd, with name=nodejs, stream=my-stream-5, version=20170314, context=abcd. There is no way for you to independently figure out what the NSVC components are, you will need to ask Koji, and use its name, version and release fields (with name=name, version=stream, release=version(.context)). Also as a consequence, modules should not be dash-separated in anything the users see. For user consumption, they are Name:Stream:Version:Context, so you may need to manually convert between one representation and the other if you need to look up in koji or other systems versus display to users. Also, note that if you are touching older modules in Koji (e.g. because you go through all koji builds), modules before February 19th will be lacking the Context field. However, I've been told that those modules will not be used, so unless you or your app goes spelunking in koji (for fun and profit), you should never see those. I hope that this is useful information for anyone else finding themselves having to parse NVRs/NSVs. Regards, Patrick

4 3

Freeze break request: refresh fedmsg crl.pem
by Kevin Fenzi 18 Mar '18

18 Mar '18

Greetings. Our current crl.pem for fedmsg (cert revocation list) expires on monday. I'd like to regenerate it with a expire time out 6 months and push it out to all hosts that use fedmsg, hopefully asap. So, this is: * Generate new crl.pem for fedmsg in ansible-private. * Run proxy playbooks with -t fedmsg/proxy to copy the new crl.pem to https://fedoraproject.org/fedmsg/crl.pem * fedmsg using services should download it and use it. +1s? kevin

3 2

Freeze Break Request: fix git hooks on new packages
by Kevin Fenzi 16 Mar '18

16 Mar '18

Greetings. It seems the git-check-perms cron job is still not 100% right. ;) Currently it's running as 'nobody' and thus doesn't have perms to setup right right hooks. ;( I think the following should fix it (also removes a MAILTO that doesn't matter anymore). diff --git a/roles/gitolite/check_fedmsg_hooks/tasks/main.yml b/roles/gitolite/check_fedmsg_hooks/tasks/main.yml index a22018e..51bff93 100644 --- a/roles/gitolite/check_fedmsg_hooks/tasks/main.yml +++ b/roles/gitolite/check_fedmsg_hooks/tasks/main.yml @@ -7,8 +7,8 @@ cron_file=ansible-git-check-perms minute=10 hour="0, 12" - user=nobody - job="MAILTO=root /usr/local/bin/git-check-perms /srv/git/repositories --check=fedmsg-hook -f" + user=root + job="/usr/local/bin/git-check-perms /srv/git/repositories --check=fedmsg-hook -f" tags: - git - gitolite +1s? kevin

3 3

FBR: Update download servers to new kernel.org mirrors
by Stephen John Smoogen 16 Mar '18

16 Mar '18

Add the following to the tier1 mirrors [smooge@batcave01 ansible (master)]$ git diff origin/master diff --git a/inventory/group_vars/download b/inventory/group_vars/download index 36b605e..e754529 100644 --- a/inventory/group_vars/download +++ b/inventory/group_vars/download @@ -66,3 +66,6 @@ dl_tier1: - ultra.linux.cz # 195.113.15.27 - wpi.edu # 130.215.36.26 - zaphod.gtlib.gatech.edu # 128.61.111.12 + - sjc.edge.kernel.org # 147.75.69.165 + - ewr.edge.kernel.org # 147.75.197.195 + - ams.edge.kernel.org # 147.75.101.1 -- Stephen J Smoogen.

3 2

FBR: To update pungi on branched-composer.phx2.fedoraproject.org and rawhide-composer.phx2.fedoraproject.org to pungi-4.1.22-10.fc27
by Mohan Boddu 16 Mar '18

16 Mar '18

pungi-4.1.22-10 should finally fix the issue with modular rpms getting into Everything repo Pungi Bug: https://pagure.io/pungi/issue/862 Fix: https://pagure.io/pungi/issue/862#comment-499692 Koji Build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1059055

3 2

VDO
by Miroslav Suchý 15 Mar '18

15 Mar '18

After I seen the talk about VDO: https://www.youtube.com/watch?v=7CGr5LEAfRY I went ahead and tried it. I tried it on small (12GB) sample of Copr data and I saved 20-30% data. I then deployed it on production server retrace.fedoraproject.org and saved there 15% out of 2TB. Several notes: The RPM packages are only available for RHEL. Recently the public github has been updated with version of VDO, which should work on Fedora. No RPM available for Fedora yet though. Here are my notes: # vdo create --name=vdo1 --device=/dev/vdb --vdoLogicalSize=1T # mkfs.ext4 /dev/mapper/vdo1 Some slides suggest to use -E discard. This is not needed for new volumes. See vdo-devel mailing list for reasoning. # mount -o discard /dev/mapper/vdo1 /mnt/a Probably best scenario is to create vdo on whole disk and then make PV from whole VDO disk and put LVM on top of that. When you already have data, you cannot convert your disk to VDO. You need create new VDO disk, format it, mount it and copy the data. In my case, due limited space for migration I created a small LVM volume "vdosrv": # lvcreate -L 80G -n vdosrv vol0 I converted it to VDO which claims to have 2TB of vitual total size # vdo create --name=srv18 --device=/dev/vol0/vdosrv --vdoLogicalSize=2T # mkfs.ext4 /dev/mapper/srv18 See how many space is *actually* free. # vdostats # mkdir /mnt/srv18 # mount -o discard /dev/mapper/srv18 /mnt/srv18 Now I moved some data from old volume to /mnt/srv18. I shrank the FS of old volume, I shrank the old LV. And then I enlarged the new LV. # lvresize -L +400G /dev/vol0/vdosrv and let know vdo that it can use the new space: # vdo growPhysical -n srv18 I repeated this several times. I have to say that my experience is so far good. I hope that my notes help someone. Miroslav

8 12

Weekly Infrastructure Meeting 2018-03-15
by Stephen John Smoogen 14 Mar '18

14 Mar '18

Join us during the Ides of March as we greet Juliu.. oh sorry different meeting This shared document is for the next fedora infrastructure meeting. = Preamble = The infrastructure team will be having its weekly meeting tomorrow, 2018-03-14 at 18:00 UTC in #fedora-meeting on the freenode network. We have a gobby document (see: https://fedoraproject.org/wiki/Gobby ) fedora-infrastructure-meeting-next is the document. Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of today is included in this email. If you have something to discuss, add the topic to the discussion area with your name. If you would like to teach other folks about some application or setup in our infrastructure, please add that topic and your name to the learn about section. = Introduction = We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2018-03-14) #meetingname infrastructure #topic aloha #chair smooge relrod nirik pingou puiterwijk tflink = Let new people say hello = #topic New folks introductions #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves = Status / Information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #topic announcements and information #info Infrastructure hackfest in april: https://fedoraproject.org/wiki/Infrastructure_Hackathon_2018 #info We are in F28 Beta Freeze! All changes to frozen hosts must get +2 on list before being applied - kevin #info = Things we should discuss = We use this section to bring up discussion topics. Things we want to talk about as a group and come up with some consensus /suor decision or just brainstorm a problem or issue. If there are none of these we skip this section. (Use #topic your discussion topic - your username) #topic Ticket cleanup #info none this week. #topic Turning off pkgdb #info It has been a depracated service but some things still refer to it #info what needs to be done to finish it off? #topic Darkserver and tagger retirements #info next steps/action items? = Apprentice office hours = #topic Apprentice Open office minutes #info A time where apprentices may ask for help or look at problems. Here we will discuss any apprentice questions, try and match up people looking for things to do with things to do, progress, testing anything like that. = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the i/nfo in this section. In the event we don't find someone to teach about something, we skip this section and just move on to open floor.) #topic Learn about: #info none this week = Meeting end stuff = #topic Open Floor #endmeeting -- Stephen J Smoogen.

1 0

FBR - OSBS authoritative registry
by Clement Verna 14 Mar '18

14 Mar '18

Hi, We currently set the authoritative registry value to registry.example.com this patch fixes that to set it to registry.fp.o. This is closing the following tickets [0][1] +1 to apply ? [0] - https://pagure.io/atomic-wg/issue/272 [1] - https://pagure.io/releng/issue/6780 --- playbooks/groups/buildvm.yml | 4 ++-- playbooks/groups/osbs-cluster.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/playbooks/groups/buildvm.yml b/playbooks/groups/buildvm.yml index e6d80eb0f..7343a2f9b 100644 --- a/playbooks/groups/buildvm.yml +++ b/playbooks/groups/buildvm.yml @@ -99,7 +99,7 @@ koji_hub: 'https://koji.stg.fedoraproject.org/kojihub', sources_command: 'fedpkg sources', build_type: 'prod', - authoritative_registry: 'registry.example.com', + authoritative_registry: 'registry.fedoraproject.org', vendor: 'Fedora Project', verify_ssl: true, use_auth: true, @@ -131,7 +131,7 @@ koji_hub: 'https://koji.fedoraproject.org/kojihub', sources_command: 'fedpkg sources', build_type: 'prod', - authoritative_registry: 'registry.example.com', + authoritative_registry: 'registry.fedoraproject.org', vendor: 'Fedora Project', verify_ssl: true, use_auth: true, diff --git a/playbooks/groups/osbs-cluster.yml b/playbooks/groups/osbs-cluster.yml index 39105b03b..476934d2a 100644 --- a/playbooks/groups/osbs-cluster.yml +++ b/playbooks/groups/osbs-cluster.yml @@ -731,7 +731,7 @@ koji_hub: 'https://{{koji_url}}/kojihub', sources_command: 'fedpkg sources', build_type: 'prod', - authoritative_registry: 'registry.example.com', + authoritative_registry: 'registry.fedoraproject.org', vendor: 'Fedora Project', verify_ssl: true, use_auth: true, @@ -763,7 +763,7 @@ koji_hub: 'https://{{koji_url}}/kojihub', sources_command: 'fedpkg sources', build_type: 'prod', - authoritative_registry: 'registry.example.com', + authoritative_registry: 'registry.fedoraproject.org', vendor: 'Fedora Project', verify_ssl: true, use_auth: true, --

3 4

FBR: To update pungi on branched-composer.phx2.fedoraproject.org and rawhide-composer.phx2.fedoraproject.org to pungi-4.1.22-9.fc27
by Mohan Boddu 14 Mar '18

14 Mar '18

This will fix the issue with empty module variant and also skipping buildinstall phase on variants. Koji build - https://koji.fedoraproject.org/koji/buildinfo?buildID=1058071

3 2

FBR: Fix the trigger upload script in to create AMIs manually
by Sayan Chowdhury 13 Mar '18

13 Mar '18

Hi, fedimg contains a trigger upload script that helps in uploading the raw images in case the upload fails. This patch fixes the script to use the third parameter in the function that would push the compose metadata. +1s? From: Sayan Chowdhury <sayan.chowdhury2012(a)gmail.com> Date: Tue, 13 Mar 2018 17:07:53 +0530 Subject: [PATCH 1/1] fedimg: Fix the trigger_upload script to create AMIs manually --- roles/fedimg/files/trigger_upload.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/roles/fedimg/files/trigger_upload.py b/roles/fedimg/files/trigger_upload.py index 5a5208d..29907f1 100644 --- a/roles/fedimg/files/trigger_upload.py +++ b/roles/fedimg/files/trigger_upload.py @@ -16,7 +16,7 @@ from fedimg.services.ec2 import EC2Service, EC2ServiceException import fedimg.uploader from fedimg.util import virt_types_from_url -if len(sys.argv) != 2: +if len(sys.argv) != 3: print 'Usage: trigger_upload.py <rawxz_image_url>' sys.exit(1) @@ -26,5 +26,10 @@ log = logging.getLogger('fedmsg') upload_pool = multiprocessing.pool.ThreadPool(processes=4) url = sys.argv[1] +compose_id = sys.argv[2] -fedimg.uploader.upload(upload_pool, [url]) +compose_meta = { + 'compose_id': compose_id +} + +fedimg.uploader.upload(upload_pool, [url], compose_meta=compose_meta) -- 2.9.4 -- Sayan Chowdhury <https://sayanchowdhury.dgplug.org/> Senior Software Engineer, Fedora Engineering - Emerging Platform GPG Fingerprint : 0F16 E841 E517 225C 7D13 AB3C B023 9931 9CD0 5C8B

3 4

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure March 2018