infrastructure February 2018

infrastructure@lists.fedoraproject.org

36 participants
38 discussions

Flagstate: an index server for containers
by Owen Taylor 26 Feb '18

26 Feb '18

The Docker registry protocol has very limited search and index functionality - you can (if enabled on the server) list all repositories, list tags for each repository, and then download the manifest for each tag. This obviously is very inefficient if you want to ask questions like: * "What containers are available with the org.flatpak.metadata" and what are their human readable names? * "Is there a newer version available of any of these 50 installed applications" (especially without fingerprinting your machine by asking about each application) We're planning to distribute desktop applications as Flatpaks on registry.fedoraproject.org, and to be able make GNOME Software and updates work properly we need a way to efficiently query the metadata of the registry. To handle this, I wrote a new server "Flagstate" - https://github.com/owtaylor/flagstate - It sits alongside a docker/distribution registry instance. At initialization, it retrieves all the metadata from the registry server and stores it in a database. It then incrementally updates the database in response to webhook notifications from the registry. The index server supports queries. A hypothetical URL would be: https://registry.fedoraproject/index/static?architecture=amd64&annotation:o… Which returns a JSON dump of the metadata for matching containers. The response is designed to be cacheable with consistent ordering and Etag support. See https://github.com/owtaylor/flagstate/blob/master/docs/protocol.md for details. In addition to the Flatpak use case, this also would be useful for the Cockpit project that wants to be able to provide a nice interface for browsing Cockpit plugins that are packaged as container images. Additional future use cases that could be accomplished with pretty simple extension of the Flagstate code include providing a backend for CLI searches - docker search, podman search, ..., and providing a more comprehensive web frontend than what is currently on registry.fedoraproject.org - allow seeing names, descriptions, and so forth extracted from container labels. I'd like to propose that we work toward a deployment of this on Fedora infrastructure. Mini-FAQ ======= Can't this be done by just traversing the registry in a cron job and writing a static index? Without the incremental update process, it's going to be impossible to provide a reasonable update frequency for a medium-large index. It would be possible to static-generate a fixed set of queries instead of having dynamic responses, but I think the current approach is more flexible - it avoids having to hard-code, say, Flatpak specifics in the server configuration. Are there existing projects that could be used instead of writing our own? Not that I'm aware of - 'reg', which is used for the current HTML frontend for registry.fedoraproject.org, takes the above approach of traversing the entire registry and writing a static index. How is the data stored? Data is stored in a a PostgreSQL database - this is purely a shadow of the data as canonically stored in the registry, so has minimal backup needs. PostgreSQL 9.4 or newer is needed for the jsonb functionality, though this requirement could be removed if necessary. Why the language/license choice? Flagstate is writing in Go and license under the Apache License 2.0 to match the docker/distribution codebase and maximize the chance of creating a community. Is it secure? It's hard to say without some careful checking. The index only indexes public information and care is taken when building SQL queries. DOS is always a tricky thing to handle on any unauthenticated API - setting the statement_timeout postgresql parameter might would make accidental DOS harder.

1 0

Reinstall of staging openshift
by Kevin Fenzi 25 Feb '18

25 Feb '18

Greetings. Our staging openshift is still version 3.5, and still warns/yells about nodes not having enough disk space (with 3.4 we allocated 20GB to them, but in 3.5 and later they want at least 40G). So, I'd like to take our staging openshift down, reprovision it with more space for the nodes and also get it up to 3.7 Hopefully this won't take too long, and since we have no persistent data everything should just reprovision via ansible. I'd like to look at doing this tomorrow. If you are using staging openshift and would prefer some other time/date, please let me know. Thanks, kevin

2 2

What causes newly registered account on fas to be spamcheck_manual?
by Zamir SUN 23 Feb '18

23 Feb '18

Hi Infra team, This week two people reached me asking about logging into Pagure. Both of them tried to register on FAS, but they are automatically set as spamcheck_manual immediately after they registered and even cannot login to sign CLA/FPCA. As far as I know, one of them is using gmail and another is using a emain from their own domain - I think it's not caused by main domains. So I want to know if there are certain rules that we don't know, or spamcheck rules which users from China might easily get into? Thanks! -- Ziqian SUN (Zamir) GPG : 1D86 6D4A 49CE 4BBD 72CF FCF5 D856 6E11 F2A0 525E Want to know more about Fedora? Visit https://fedoraproject.org/wiki/ Ready to contribute? See https://whatcanidoforfedora.org/

3 2

Active Fedora releases and the EOL
by Chenxiong Qi 23 Feb '18

23 Feb '18

Hi, When I query active Fedora releases from PDC, I get Fedora 24 and 25, both of them were already EOL. I'm confused why 24 and 25 are still active releases? What is the definition of "active release", and how to understand the relationship between active release and EOL? Thanks! -- Regards, Chenxiong Qi

1 0

[PATCH] git/hooks: only use first line as project description
by Todd Zullinger 22 Feb '18

22 Feb '18

If a description file contains multiple lines the script reads the whole file and uses them in the X-Project: header. This leaves subsequent headers in the message body. Here's an example from this repository: Subject: [ansible] Redefine master_repomd settings that don't have defaults. Keywords: ansible X-Project: ansible playbook/files/etc repository for fedora infrastructure. Message-Id: <20180209174002.696C8FB72586(a)batcave01.phx2.fedoraproject.org> Date: Fri, 9 Feb 2018 17:40:02 +0000 (UTC) This is the public repository, do not commit sensitive or confidential information here. X-Git-Refname: refs/heads/master X-Git-Oldrev: db817ee0f3bbd99b762a2d6b78d84212a0770935 X-Git-Newrev: d65b46a20984939465a6209205662f0030d80f54 commit d65b46a20984939465a6209205662f0030d80f54 A further potential feature could be added to include the additional lines from the description file in the beginning of the message body, if they are desired. --- Hi all, This is something I've seen for ages and thought should be fixed (either by this patch and/or another which supports multi-line description files properly). I have an alternate version of this patch which side-steps the issue by adding the X-Project header last, so if it's multiple lines it won't screw up other headers. I think this is the proper fix though. If multi-line description files are really something which are desired, then a commit on top of this change could add support for reading the remaining lines from the description file and add them to the message body. roles/git/hooks/files/git.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/roles/git/hooks/files/git.py b/roles/git/hooks/files/git.py index 72adff1f7..586fbffd2 100644 --- a/roles/git/hooks/files/git.py +++ b/roles/git/hooks/files/git.py @@ -202,7 +202,8 @@ def get_project_description(): description = os.path.join(git_dir, 'description') if os.path.exists(description): try: - projectdesc = open(description).read().strip() + with open(description, 'r') as f: + projectdesc = f.readline().strip() except: pass if projectdesc.startswith('Unnamed repository;'): -- 2.16.2

2 1

[PATCH] remove cron job for atomic-cd repo
by Dusty Mabe 22 Feb '18

22 Feb '18

This is not used and we agreed to remove it. See [1] [2]. [1] https://pagure.io/releng/issue/7269 [2] https://pagure.io/atomic-wg/issue/349 Signed-off-by: Dusty Mabe <dusty(a)dustymabe.com> --- roles/releng/files/atomic-cd | 4 ---- roles/releng/tasks/main.yml | 5 ----- 2 files changed, 9 deletions(-) delete mode 100644 roles/releng/files/atomic-cd diff --git a/roles/releng/files/atomic-cd b/roles/releng/files/atomic-cd deleted file mode 100644 index da90f7153..000000000 --- a/roles/releng/files/atomic-cd +++ /dev/null @@ -1,4 +0,0 @@ -# continous delivery of atomic host -MAILTO=releng-cron(a)lists.fedoraproject.org -*/15 * * * * root TMPDIR=`mktemp -d /tmp/atomic-cd.XXXXXX` && pushd $TMPDIR >& /dev/null && git clone -q https://pagure.io/releng.git && cd releng && LANG=en_US.UTF-8 /usr/local/bin/lock-wrapper atomic-cd ./scripts/build-test-ostree >& /dev/null && popd >& /dev/null && rm -rf $TMPDIR - diff --git a/roles/releng/tasks/main.yml b/roles/releng/tasks/main.yml index 94ba9d814..6a8d81e79 100644 --- a/roles/releng/tasks/main.yml +++ b/roles/releng/tasks/main.yml @@ -167,11 +167,6 @@ copy: src="docker-updates" dest=/etc/cron.d/docker-updates when: inventory_hostname.startswith('compose-x86-01') -# put cron job in for continuously deliverying atomic host -- name: atomic-cd compose cron - copy: src="atomic-cd" dest=/etc/cron.d/atomic-cd - when: inventory_hostname.startswith('compose-x86-01') - # put cron job in for purging nightly amis - name: purge nightly amis cron copy: src="purge-amis" dest=/etc/cron.d/purge-amis -- 2.14.3

3 3

[release] simple-koji-ci: 0.3.0
by Pierre-Yves Chibon 22 Feb '18

22 Feb '18

Good Morning Everyone, I just cut a new release of simple-koji-ci: 0.3.0. Here is its changelog: * Thu Feb 22 2018 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.3.0-1 - Ensure the license info are consistently GPLv2+ - Flag the PR once the build has been submitted - Support building in the target branch if known, otherwise default to rawhide - Mention the commit tested in the flag added - Trigger a new build on PR updates It's running in prod already, I'm monitoring it to see if it behaves as expected :) Happy pull-requesting! Pierre

1 0

[release] dist-git 1.6
by Michal Novotny 21 Feb '18

21 Feb '18

Hello, I released a new version of DistGit, now waiting in testing: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-50a505e816. Change summary: - add 'fedmsgs', 'old_paths', and 'default_namespace' config options - remove domain_read_all_domains_state SELinux rule - require dist-git-selinux - give optional map permission to git_system_t on git_user_content_t - update requires to work for all environments - make the package completely distribution-agnostic You can read more about the release here: https://github.com/release-engineering/dist-git/wiki The package does the same thing as before but it is now more configurable. Best regards! clime

1 0

bodhi-3.4.0b0 deployed to stg
by Randy Barlow 19 Feb '18

19 Feb '18

I've deployed bodhi-3.4.0b0[0] to staging[1] and done my own testing on it. I'll let it sit for a week or so and if no regressions are found I plan to deploy in the next available window. [0] https://bodhi.stg.fedoraproject.org/docs/user/release_notes.html [1] https://bodhi.stg.fedoraproject.org

1 0

bodhi-3.3.0b0 deployed to staging
by Randy Barlow 19 Feb '18

19 Feb '18

Greetings! I've deployed bodhi-3.3.0b0[0] to staging[1] and done my own testing on it. I'll let it sit for a week or so and if no regressions are found I plan to deploy in the next available window. P.S. This one does have one really cool feature - the web UI now allows you to see running composes[2]. [0] https://bodhi.stg.fedoraproject.org/docs/user/release_notes.html [1] https://bodhi.stg.fedoraproject.org [2] https://bodhi.stg.fedoraproject.org/composes/

1 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure February 2018