copr-commits August 2022

copr-commits@lists.fedorahosted.org

1 participants
31 discussions

[copr/copr] branch main updated (8b2c8fe9 -> 496949a9)
by pagure＠pagure.io 29 Aug '22

29 Aug '22

This is an automated email from the git hooks/post-receive script. frostyx pushed a change to branch main in repository copr/copr. from 8b2c8fe9 frontend: add link to /user/repositories/ on the user detail page add 496949a9 frontend: show the batch ID (if assigned) in the build detail No new revisions were added by this update. Summary of changes: frontend/coprs_frontend/coprs/templates/coprs/detail/build.html | 8 ++++++++ 1 file changed, 8 insertions(+) -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

[copr/copr] branch main updated (a426bc8e -> 8b2c8fe9)
by pagure＠pagure.io 29 Aug '22

29 Aug '22

This is an automated email from the git hooks/post-receive script. frostyx pushed a change to branch main in repository copr/copr. from a426bc8e frontend: stg frontend instance should link to stg FAS instance add 8b2c8fe9 frontend: add link to /user/repositories/ on the user detail page No new revisions were added by this update. Summary of changes: frontend/coprs_frontend/coprs/templates/coprs/show/user.html | 3 +++ 1 file changed, 3 insertions(+) -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

[copr/copr] branch main updated (8244a626 -> a426bc8e)
by pagure＠pagure.io 25 Aug '22

25 Aug '22

This is an automated email from the git hooks/post-receive script. frostyx pushed a change to branch main in repository copr/copr. from 8244a626 frontend: test-cover rebuild permission denied add a426bc8e frontend: stg frontend instance should link to stg FAS instance No new revisions were added by this update. Summary of changes: frontend/coprs_frontend/coprs/config.py | 2 ++ frontend/coprs_frontend/coprs/context_processors.py | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

[copr/copr] branch main updated (827c257a -> 8244a626)
by pagure＠pagure.io 22 Aug '22

22 Aug '22

This is an automated email from the git hooks/post-receive script. frostyx pushed a change to branch main in repository copr/copr. from 827c257a docs: how to release copr fixes add 806b117b frontend: fix two new pylint issues from #2266 add e99fcc88 frontend: add unit tests for comps add af6eb23a frontend: we only upload comps as bytes add 4cdfe1e4 frontend: cover web-UI build cancel action add 69ef9737 frontend: cover /api_v3/build-chroots/list add 4e941772 frontend: drop unused method add 8244a626 frontend: test-cover rebuild permission denied No new revisions were added by this update. Summary of changes: .../coprs_frontend/coprs/logic/builds_logic.py | 12 ----- frontend/coprs_frontend/coprs/models.py | 9 ++-- .../coprs/views/apiv3_ns/__init__.py | 2 +- frontend/coprs_frontend/tests/coprs_test_case.py | 9 +++- frontend/coprs_frontend/tests/data/comps.xml | 1 + frontend/coprs_frontend/tests/request_test_api.py | 53 +++++++++++++++++++--- .../tests/test_apiv3/test_build_chroots.py | 41 +++++++++++++++++ frontend/coprs_frontend/tests/test_comps.py | 29 ++++++++++++ .../tests/test_logic/test_builds_logic.py | 13 +++++- .../test_views/test_coprs_ns/test_coprs_general.py | 11 ++++- 10 files changed, 151 insertions(+), 29 deletions(-) create mode 100644 frontend/coprs_frontend/tests/data/comps.xml create mode 100644 frontend/coprs_frontend/tests/test_comps.py -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

[copr/copr] branch main updated (a32b52b4 -> 827c257a)
by pagure＠pagure.io 22 Aug '22

22 Aug '22

This is an automated email from the git hooks/post-receive script. frostyx pushed a change to branch main in repository copr/copr. from a32b52b4 frontend: add page for exploring projects add 827c257a docs: how to release copr fixes No new revisions were added by this update. Summary of changes: doc/how_to_release_copr.rst | 90 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 70 insertions(+), 20 deletions(-) -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

[copr/copr] branch main updated (36349b2e -> a32b52b4)
by pagure＠pagure.io 22 Aug '22

22 Aug '22

This is an automated email from the git hooks/post-receive script. msuchy pushed a change to branch main in repository copr/copr. from 36349b2e frontend: add test for updating chroot via APIv3 add a32b52b4 frontend: add page for exploring projects No new revisions were added by this update. Summary of changes: frontend/coprs_frontend/coprs/__init__.py | 2 + frontend/coprs_frontend/coprs/logic/coprs_logic.py | 16 +++++++ frontend/coprs_frontend/coprs/logic/stat_logic.py | 27 +++++++++++- frontend/coprs_frontend/coprs/models.py | 16 +++++++ .../coprs_frontend/coprs/templates/explore.html | 50 ++++++++++++++++++++++ frontend/coprs_frontend/coprs/views/explore_ns.py | 31 ++++++++++++++ 6 files changed, 141 insertions(+), 1 deletion(-) create mode 100644 frontend/coprs_frontend/coprs/templates/explore.html create mode 100644 frontend/coprs_frontend/coprs/views/explore_ns.py -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

[copr/copr] annotated tag copr-backend-1.159-1 created (now 50374164)
by pagure＠pagure.io 16 Aug '22

16 Aug '22

This is an automated email from the git hooks/post-receive script. praiskup pushed a change to annotated tag copr-backend-1.159-1 in repository copr/copr. at 50374164 (tag) tagging 51a618ac85ca4046fbc5daf074e647d47d271c49 (commit) replaces copr-frontend-1.190-1 by Pavel Raiskup on Tue Aug 16 14:35:14 2022 +0200 - Log ----------------------------------------------------------------- Tagging package [copr-backend] version [copr-backend-1.159-1] in directory [backend/]. ----------------------------------------------------------------------- This annotated tag includes the following new commits: new 51a618ac Automatic commit of package [copr-backend] release [1.159-1]. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

[copr/copr] branch main updated: Automatic commit of package [copr-backend] release [1.159-1].
by pagure＠pagure.io 16 Aug '22

16 Aug '22

This is an automated email from the git hooks/post-receive script. praiskup pushed a commit to branch main in repository copr/copr. The following commit(s) were added to refs/heads/main by this push: new 51a618ac Automatic commit of package [copr-backend] release [1.159-1]. 51a618ac is described below commit 51a618ac85ca4046fbc5daf074e647d47d271c49 Author: Pavel Raiskup <praiskup(a)redhat.com> AuthorDate: Tue Aug 16 14:35:14 2022 +0200 Automatic commit of package [copr-backend] release [1.159-1]. Created by command: /usr/bin/tito tag --- .tito/packages/copr-backend | 2 +- backend/copr-backend.spec | 7 ++++++- backend/setup.py | 2 +- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/.tito/packages/copr-backend b/.tito/packages/copr-backend index 955d1ba7..12e557b4 100644 --- a/.tito/packages/copr-backend +++ b/.tito/packages/copr-backend @@ -1 +1 @@ -1.158-1 backend/ +1.159-1 backend/ diff --git a/backend/copr-backend.spec b/backend/copr-backend.spec index 34b71d2b..edd8a33e 100644 --- a/backend/copr-backend.spec +++ b/backend/copr-backend.spec @@ -9,7 +9,7 @@ %global copr_common_version 0.13.1.dev Name: copr-backend -Version: 1.158 +Version: 1.159 Release: 1%{?dist} Summary: Backend for Copr @@ -244,6 +244,11 @@ useradd -r -g copr -G lighttpd -s /bin/bash -c "COPR user" copr %exclude %{_pkgdocdir}/playbooks %changelog +* Tue Aug 16 2022 Pavel Raiskup <praiskup(a)redhat.com> 1.159-1 +- count only hits from an appropriate CDN hostname +- add option for infinite number of attempts to the hitcounter script +- print more reasonable output from AWS hitcounter script + * Tue Aug 16 2022 Jiri Kyjovsky <j1.kyjovsky(a)gmail.com> 1.158-1 - log every request that is sent to frontend diff --git a/backend/setup.py b/backend/setup.py index 2078a699..9d7dd49c 100644 --- a/backend/setup.py +++ b/backend/setup.py @@ -16,7 +16,7 @@ __url__ = "https://pagure.io/copr/copr" setup( name='copr-backend', - version="1.158", + version="1.159", description=__description__, long_description=long_description, author=__author__, -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

[copr/copr] 03/03: backend: count only hits from an appropriate CDN hostname
by pagure＠pagure.io 16 Aug '22

16 Aug '22

This is an automated email from the git hooks/post-receive script. praiskup pushed a commit to branch main in repository copr/copr. commit d29241459b520ee0016ab5dbe1b193be101b1b44 Author: Jakub Kadlcik <frostyx(a)email.cz> AuthorDate: Wed Aug 10 00:34:39 2022 +0200 backend: count only hits from an appropriate CDN hostname --- backend/conf/crontab/daily | 3 +++ backend/run/copr-aws-s3-hitcounter | 48 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/backend/conf/crontab/daily b/backend/conf/crontab/daily index 70f467f1..e6403a11 100644 --- a/backend/conf/crontab/daily +++ b/backend/conf/crontab/daily @@ -2,3 +2,6 @@ runuser -c "PYTHONUNBUFFERED=1 python3 /usr/bin/copr_prune_results.py &> /dev/null" - copr runuser -c "PYTHONUNBUFFERED=1 python3 /usr/bin/copr_prune_srpms.py &> /dev/null" - copr + +# Optional - Counting downloaded RPMs through Amazon CDN +# runuser -c "PYTHONUNBUFFERED=1 python3 /usr/bin/copr-aws-s3-hitcounter &> /dev/null" - copr diff --git a/backend/run/copr-aws-s3-hitcounter b/backend/run/copr-aws-s3-hitcounter index f7031f4b..5b177b77 100755 --- a/backend/run/copr-aws-s3-hitcounter +++ b/backend/run/copr-aws-s3-hitcounter @@ -29,6 +29,10 @@ from copr_backend.helpers import setup_script_logger # We will allow only this hostname to delete files from the S3 storage PRODUCTION_HOSTNAME = "copr-be.aws.fedoraproject.org" +DEVEL_HOSTNAME = "copr-be-dev.aws.fedoraproject.org" + +PRODUCTION_CDN_HOSTNAME = "download.copr.fedorainfracloud.org" +DEVEL_CDN_HOSTNAME = "download.copr-dev.fedorainfracloud.org" log = logging.getLogger(__name__) @@ -122,6 +126,37 @@ def parse_access_file(path): return accesses +def get_cdn_hostname(args): + """ + The devel and production accesses are mixed together. Which ones do we want + to count? + """ + # If a CDN hostname was explicitly specified when calling the script + if args.cdn_hostname: + return args.cdn_hostname + + # Count hits from devel CDN hostname on devel instance + hostname = gethostname() + if hostname == DEVEL_HOSTNAME: + return DEVEL_CDN_HOSTNAME + + # Default to production hits. Don't worry, we don't accidentally + # remove them from any other instance + return PRODUCTION_CDN_HOSTNAME + + +def check_different_cdn_hostname(accesses, cdn_hostname): + """ + If a list of HTTP accesses contain any access for a different CDN hostname + (e.g. for devel instance when the script is running on production), return + its value. Otherwise `None`. + """ + for access in accesses: + if access["x-host-header"] != cdn_hostname: + return access["x-host-header"] + return None + + def get_arg_parser(): """ Generate argument parser for this script @@ -147,6 +182,12 @@ def get_arg_parser(): help=("If true, try infinite number of attempts when contacting the " "frontend. Do not use this option for cron tasks because the " "number of simultaneously running instances might go up")) + parser.add_argument( + "--cdn-hostname", + help=("By default the devel instance counts only hits from devel, and " + "the production instance from production. You can override this " + "by explicitly specifying the CDN hostname of interest, e.g. {0}" + .format(PRODUCTION_CDN_HOSTNAME))) return parser @@ -157,6 +198,7 @@ def main(): parser = get_arg_parser() args = parser.parse_args() tmp = tempfile.mkdtemp(prefix="copr-aws-s3-hitcounter-") + cdn_hostname = get_cdn_hostname(args) if args.verbose: log.setLevel(logging.DEBUG) @@ -169,6 +211,12 @@ def main(): raw = gunzip(gz) accesses = parse_access_file(raw) + different_cdn = check_different_cdn_hostname(accesses, cdn_hostname) + if different_cdn: + log.debug("Skipping: %s (different hostname: %s)", + s3file, different_cdn) + continue + log.info("[%s/%s] %s (%s accesses)", i, len(files), s3file, len(accesses)) -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

[copr/copr] 02/03: backend: add option for infinite number of attempts to the hitcounter script
by pagure＠pagure.io 16 Aug '22

16 Aug '22

This is an automated email from the git hooks/post-receive script. praiskup pushed a commit to branch main in repository copr/copr. commit 21701e00f86bbef421ddd6a1b75b91d6e1d4cd5a Author: Jakub Kadlcik <frostyx(a)email.cz> AuthorDate: Tue Aug 9 23:40:30 2022 +0200 backend: add option for infinite number of attempts to the hitcounter script I don't think this will be useful in production for two reasons. We don't want to use this option for cron tasks. Also, in production, once we process an access file, we delete it from S3. Therefore re-running the script won't count anything twice nor waste any time with already counted files. But this option is useful when debugging. I don't want the script to die after hours of counting accesses that would be counted again when re-running the script. --- backend/copr_backend/hitcounter.py | 9 +++++++-- backend/run/copr-aws-s3-hitcounter | 9 ++++++++- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/backend/copr_backend/hitcounter.py b/backend/copr_backend/hitcounter.py index 1c53ebd2..2a2e0327 100644 --- a/backend/copr_backend/hitcounter.py +++ b/backend/copr_backend/hitcounter.py @@ -59,7 +59,7 @@ def url_to_key_strings(url): return [] -def update_frontend(accesses, log, dry_run=False): +def update_frontend(accesses, log, dry_run=False, try_indefinitely=False): """ Increment frontend statistics based on these `accesses` """ @@ -86,7 +86,12 @@ def update_frontend(accesses, log, dry_run=False): "from_backend", ) if not dry_run: - SafeRequest(auth=opts.frontend_auth, log=log).post(url, result) + request = SafeRequest( + auth=opts.frontend_auth, + log=log, + try_indefinitely=try_indefinitely, + ) + request.post(url, result) def get_hit_data(accesses, log): diff --git a/backend/run/copr-aws-s3-hitcounter b/backend/run/copr-aws-s3-hitcounter index 8b795ae2..f7031f4b 100755 --- a/backend/run/copr-aws-s3-hitcounter +++ b/backend/run/copr-aws-s3-hitcounter @@ -141,6 +141,12 @@ def get_arg_parser(): "--verbose", action="store_true", help=("Print verbose information about what is going on")) + parser.add_argument( + "--try-indefinitely", + action="store_true", + help=("If true, try infinite number of attempts when contacting the " + "frontend. Do not use this option for cron tasks because the " + "number of simultaneously running instances might go up")) return parser @@ -170,7 +176,8 @@ def main(): # a scenario when we increment the accesses on the frontend but then # leave the s3 file untouched, which would result in parsing and # incrementing from the same file again in the next run - update_frontend(accesses, log=log, dry_run=args.dry_run) + update_frontend(accesses, log=log, dry_run=args.dry_run, + try_indefinitely=args.try_indefinitely) s3.delete_file(s3file) # Clean all temporary files -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

copr-commits August 2022