copr-commits May 2015

copr-commits@lists.fedorahosted.org

3 participants
54 discussions

[copr] tag 'copr-frontend-1.59-1' created
by vgologuz＠fedoraproject.org 29 May '15

29 May '15

Repository : http://git.fedorahosted.org/cgit/copr.git New tag : copr-frontend-1.59-1 Referencing: 43fbf6dcc762cdfee38f7b5ab788be4704ea269b

1 0

[copr] master: Automatic commit of package [copr-frontend] release [1.59-1]. (503ef44)
by vgologuz＠fedoraproject.org 29 May '15

29 May '15

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : master >--------------------------------------------------------------- commit 503ef44f55ede9d9679596b6d5076d4da03dfc58 Author: Valentin Gologuzov <vgologuz(a)redhat.com> Date: Sat May 30 00:18:38 2015 +0200 Automatic commit of package [copr-frontend] release [1.59-1]. >--------------------------------------------------------------- frontend/copr-frontend.spec | 11 ++++++++++- rel-eng/packages/copr-frontend | 2 +- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/frontend/copr-frontend.spec b/frontend/copr-frontend.spec index 1b9f227..22e1e45 100644 --- a/frontend/copr-frontend.spec +++ b/frontend/copr-frontend.spec @@ -5,7 +5,7 @@ %endif Name: copr-frontend -Version: 1.58 +Version: 1.59 Release: 1%{?dist} Summary: Frontend for Copr @@ -192,6 +192,15 @@ service logstash condrestart %doc documentation/python-doc %changelog +* Sat May 30 2015 Valentin Gologuzov <vgologuz(a)redhat.com> 1.59-1 +- Front page rendering takes too much time due to a long sql query. + Simplified until issue is resolved. +- fix regression: show again additional buildroot packages for + modified chroots at overview page +- reject build_chroot status update for `failed`, `cancelled` and + `succeded` states; added some logs +- new logo + * Wed May 20 2015 Valentin Gologuzov <vgologuz(a)redhat.com> 1.58-1 - backend api: handle to mark all running/starting builds as pending - add to example url to Fedora instance of FedMenu diff --git a/rel-eng/packages/copr-frontend b/rel-eng/packages/copr-frontend index 77d7fb3..3285ed0 100644 --- a/rel-eng/packages/copr-frontend +++ b/rel-eng/packages/copr-frontend @@ -1 +1 @@ -1.58-1 frontend/ +1.59-1 frontend/

1 0

[copr] master: [frontend] front page rendering takes too much time due to a long sql query. Simplified until issue is resolved. (f9ee878)
by vgologuz＠fedoraproject.org 29 May '15

29 May '15

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : master >--------------------------------------------------------------- commit f9ee87862ab6f03ca6de8f6b057592455fd658d2 Author: Valentin Gologuzov <vgologuz(a)redhat.com> Date: Fri May 29 23:22:05 2015 +0200 [frontend] front page rendering takes too much time due to a long sql query. Simplified until issue is resolved. >--------------------------------------------------------------- .../coprs_frontend/coprs/templates/coprs/show.html | 3 ++- .../coprs/views/coprs_ns/coprs_general.py | 4 +++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/frontend/coprs_frontend/coprs/templates/coprs/show.html b/frontend/coprs_frontend/coprs/templates/coprs/show.html index d8ae35d..7e76f07 100644 --- a/frontend/coprs_frontend/coprs/templates/coprs/show.html +++ b/frontend/coprs_frontend/coprs/templates/coprs/show.html @@ -44,7 +44,8 @@ </div> <div> <a class="menu" href="{{url_for('recent_ns.recent')}}"> - {% if g.user: %} My recent builds {% else %} Recent builds {% endif %} + Recent builds +  </a> {% if users_builds: %} {% for build in users_builds: %} diff --git a/frontend/coprs_frontend/coprs/views/coprs_ns/coprs_general.py b/frontend/coprs_frontend/coprs/views/coprs_ns/coprs_general.py index 0101f20..09a415e 100644 --- a/frontend/coprs_frontend/coprs/views/coprs_ns/coprs_general.py +++ b/frontend/coprs_frontend/coprs/views/coprs_ns/coprs_general.py @@ -38,7 +38,9 @@ def coprs_show(page=1): coprs = paginator.sliced_query # flask.g.user is none when no user is logged - showing builds from everyone - users_builds = builds_logic.BuildsLogic.get_recent_tasks(flask.g.user, 5) + # TODO: builds_logic.BuildsLogic.get_recent_tasks(flask.g.user, 5) takes too much time, optimize sql + # users_builds = builds_logic.BuildsLogic.get_recent_tasks(flask.g.user, 5) + users_builds = builds_logic.BuildsLogic.get_recent_tasks(None, 5) waiting_tasks = len(list(builds_logic.BuildsLogic.get_build_task_queue())) running_tasks = len(list(builds_logic.BuildsLogic

1 0

[copr] master: [frontend] fix regression: show again additional buildroot packages for modified chroots at overview page (137591d)
by vgologuz＠fedoraproject.org 29 May '15

29 May '15

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : master >--------------------------------------------------------------- commit 137591dff3d5d36f393a3aeb1a8ebd9486c7244b Author: Valentin Gologuzov <vgologuz(a)redhat.com> Date: Fri May 29 22:31:13 2015 +0200 [frontend] fix regression: show again additional buildroot packages for modified chroots at overview page >--------------------------------------------------------------- frontend/coprs_frontend/coprs/models.py | 4 ++++ .../coprs/templates/coprs/detail/overview.html | 9 ++++++++- 2 files changed, 12 insertions(+), 1 deletions(-) diff --git a/frontend/coprs_frontend/coprs/models.py b/frontend/coprs_frontend/coprs/models.py index a4317a8..2b77b8e 100644 --- a/frontend/coprs_frontend/coprs/models.py +++ b/frontend/coprs_frontend/coprs/models.py @@ -231,6 +231,10 @@ class Copr(db.Model, helpers.Serializer): modified_chroots.append(chroot) return modified_chroots + def is_release_arch_modified(self, name_release, arch): + if "{}-{}".format(name_release, arch) in [chroot.name for chroot in self.modified_chroots]: + return True + return False class CoprPermission(db.Model, helpers.Serializer): diff --git a/frontend/coprs_frontend/coprs/templates/coprs/detail/overview.html b/frontend/coprs_frontend/coprs/templates/coprs/detail/overview.html index e108870..7c22359 100644 --- a/frontend/coprs_frontend/coprs/templates/coprs/detail/overview.html +++ b/frontend/coprs_frontend/coprs/templates/coprs/detail/overview.html @@ -52,7 +52,14 @@ {{ repo.name_release_human|capitalize }} </td> <td> - {{ repo.arch_list|join(", ") }} + {% for arch in repo.arch_list %} + {{ arch }} + {% if copr.is_release_arch_modified(repo.name_release, arch) %} + <a id="modified-chroot-{{repo.name_release}}-{{arch}}">[modified]</a> + {% endif %} + <br /> + {% endfor %} + </td> <!-- <td> <center> {{ repo.dl_stat }} </center> <br />

1 0

[copr] master: [backend] catch exception during Worker.can_start_job (13fbd14)
by vgologuz＠fedoraproject.org 29 May '15

29 May '15

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : master >--------------------------------------------------------------- commit 13fbd1480ee1ca5b893646d6f8f0cd14f0404ba1 Author: Valentin Gologuzov <vgologuz(a)redhat.com> Date: Fri May 29 17:40:18 2015 +0200 [backend] catch exception during Worker.can_start_job >--------------------------------------------------------------- backend/backend/daemons/dispatcher.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/backend/backend/daemons/dispatcher.py b/backend/backend/daemons/dispatcher.py index 7cb5c65..44d25d5 100644 --- a/backend/backend/daemons/dispatcher.py +++ b/backend/backend/daemons/dispatcher.py @@ -413,7 +413,12 @@ class Worker(multiprocessing.Process): if not job: return - if not self.can_start_job(job): + try: + if not self.can_start_job(job): + self.notify_job_grab_about_task_end(job) + return + except Exception: + self.log.exception("Failed to check if job can be started") self.notify_job_grab_about_task_end(job) return

1 0

[copr] master: note about BuildsLogic.get_build_task_queue (5999b29)
by vgologuz＠fedoraproject.org 29 May '15

29 May '15

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : master >--------------------------------------------------------------- commit 5999b294b3c3c9bb4e1e4418a4f149d446352586 Author: Valentin Gologuzov <vgologuz(a)redhat.com> Date: Fri May 29 17:39:38 2015 +0200 note about BuildsLogic.get_build_task_queue >--------------------------------------------------------------- .../coprs_frontend/coprs/logic/builds_logic.py | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/frontend/coprs_frontend/coprs/logic/builds_logic.py b/frontend/coprs_frontend/coprs/logic/builds_logic.py index df5f73c..3bd32a7 100644 --- a/frontend/coprs_frontend/coprs/logic/builds_logic.py +++ b/frontend/coprs_frontend/coprs/logic/builds_logic.py @@ -63,6 +63,10 @@ class BuildsLogic(object): models.BuildChroot.status == helpers.StatusEnum("starting"), and_( models.BuildChroot.status.in_([ + # Bug 1206562 - Cannot delete Copr because it incorrectly thinks + # there are unfinished builds. Solution: `failed` but unfinished + # (ended_on is null) builds should be rescheduled. + # todo: we need to be sure that correct `failed` set is set together wtih `ended_on` helpers.StatusEnum("running"), helpers.StatusEnum("failed")]), models.Build.started_on < int(time.time() - 1.1 * MAX_BUILD_TIMEOUT), models.Build.ended_on.is_(None)

1 0

[copr] master: [frontend] typo (245d57d)
by vgologuz＠fedoraproject.org 28 May '15

28 May '15

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : master >--------------------------------------------------------------- commit 245d57d7559324798b24f506d80a2844e3954b10 Author: Valentin Gologuzov <vgologuz(a)redhat.com> Date: Thu May 28 17:55:58 2015 +0200 [frontend] typo >--------------------------------------------------------------- .../coprs/views/backend_ns/backend_general.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/frontend/coprs_frontend/coprs/views/backend_ns/backend_general.py b/frontend/coprs_frontend/coprs/views/backend_ns/backend_general.py index f84bfce..32cc77f 100644 --- a/frontend/coprs_frontend/coprs/views/backend_ns/backend_general.py +++ b/frontend/coprs_frontend/coprs/views/backend_ns/backend_general.py @@ -108,7 +108,7 @@ def starting_build(): if "build_id" in flask.request.json and "chroot" in flask.request.json: build = BuildsLogic.get_by_id(flask.request.json["build_id"]) - chroot = flask.request.get("chroot") + chroot = flask.request.json.get("chroot") if build and chroot and not build.canceled: log.info("mark build {} chroot {} as starting".format(build.id, chroot)) BuildsLogic.update_state_from_dict(build, {

1 0

[copr] master: [frontend] reject build_chroot status update for `failed`, `cancelled` and `succeded` states; added some logs (05cb4a4)
by vgologuz＠fedoraproject.org 28 May '15

28 May '15

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : master >--------------------------------------------------------------- commit 05cb4a4588753d1c3dc59a31d6c29b5daf3fa2a5 Author: Valentin Gologuzov <vgologuz(a)redhat.com> Date: Thu May 28 17:39:43 2015 +0200 [frontend] reject build_chroot status update for `failed`, `cancelled` and `succeded` states; added some logs >--------------------------------------------------------------- .../coprs_frontend/coprs/logic/builds_logic.py | 8 +++++++- .../coprs/views/backend_ns/backend_general.py | 18 ++++++++++-------- 2 files changed, 17 insertions(+), 9 deletions(-) diff --git a/frontend/coprs_frontend/coprs/logic/builds_logic.py b/frontend/coprs_frontend/coprs/logic/builds_logic.py index 9d3dc3e..df5f73c 100644 --- a/frontend/coprs_frontend/coprs/logic/builds_logic.py +++ b/frontend/coprs_frontend/coprs/logic/builds_logic.py @@ -14,6 +14,7 @@ from coprs import models from coprs import helpers from coprs import signals from coprs.constants import DEFAULT_BUILD_TIMEOUT, MAX_BUILD_TIMEOUT +from coprs.helpers import StatusEnum from coprs.logic import coprs_logic from coprs.logic import users_logic @@ -175,11 +176,16 @@ class BuildsLogic(object): @classmethod def update_state_from_dict(cls, build, upd_dict): + log.info("Updating build: {} by: {}".format(build.id, upd_dict)) if "chroot" in upd_dict: # update respective chroot status for build_chroot in build.build_chroots: if build_chroot.name == upd_dict["chroot"]: - if "status" in upd_dict: + + if "status" in upd_dict and build_chroot.status not in [ + # this states is terminal + StatusEnum("failed"), StatusEnum("succeeded"), StatusEnum("canceled") + ]: build_chroot.status = upd_dict["status"] db.session.add(build_chroot) diff --git a/frontend/coprs_frontend/coprs/views/backend_ns/backend_general.py b/frontend/coprs_frontend/coprs/views/backend_ns/backend_general.py index 207c8f9..f84bfce 100644 --- a/frontend/coprs_frontend/coprs/views/backend_ns/backend_general.py +++ b/frontend/coprs_frontend/coprs/views/backend_ns/backend_general.py @@ -108,14 +108,15 @@ def starting_build(): if "build_id" in flask.request.json and "chroot" in flask.request.json: build = BuildsLogic.get_by_id(flask.request.json["build_id"]) - - if build and not build.canceled: - BuildsLogic.update_state_from_dict(build, { - "chroot": flask.request.json["chroot"], - "status": StatusEnum("starting") - }) - db.session.commit() - result["can_start"] = True + chroot = flask.request.get("chroot") + if build and chroot and not build.canceled: + log.info("mark build {} chroot {} as starting".format(build.id, chroot)) + BuildsLogic.update_state_from_dict(build, { + "chroot": chroot, + "status": StatusEnum("starting") + }) + db.session.commit() + result["can_start"] = True return flask.jsonify(result) @@ -160,6 +161,7 @@ def reschedule_build_chroot(): build_chroot = build.chroots_dict_by_name.get(chroot) run_statuses = set([StatusEnum("starting"), StatusEnum("running")]) if build_chroot and build_chroot.status in run_statuses: + log.info("rescheduling build {} chroot: {}".format(build.id, build_chroot.name)) BuildsLogic.update_state_from_dict(build, { "chroot": chroot, "status": StatusEnum("pending")

1 0

[copr] master: [backend] config cleanup (92131b6)
by vgologuz＠fedoraproject.org 28 May '15

28 May '15

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : master >--------------------------------------------------------------- commit 92131b63745ce85c491691a2963f71317931d60b Author: Valentin Gologuzov <vgologuz(a)redhat.com> Date: Thu May 28 15:00:32 2015 +0200 [backend] config cleanup >--------------------------------------------------------------- backend/backend/daemons/backend.py | 2 - backend/backend/daemons/dispatcher.py | 11 +-- backend/backend/daemons/job_grab.py | 5 +- backend/backend/daemons/log.py | 77 ------------------------- backend/backend/frontend.py | 4 +- backend/backend/helpers.py | 20 +------ backend/conf/copr-be.conf.example | 23 ------- backend/conf/copr-be.local.conf | 19 ------ backend/tests/deamons/test_backend.py | 9 +--- backend/tests/deamons/test_dispatcher.py | 10 +-- backend/tests/deamons/test_job_grab.py | 4 +- backend/tests/deamons/test_log.py | 6 +- backend/tests/test_frontend.py | 2 +- backend/tests/vm_manager/test_check.py | 1 - backend/tests/vm_manager/test_event_handle.py | 2 - backend/tests/vm_manager/test_manager.py | 1 - backend/tests/vm_manager/test_spawn.py | 1 - backend/tests/vm_manager/test_terminate.py | 2 - 18 files changed, 19 insertions(+), 180 deletions(-) diff --git a/backend/backend/daemons/backend.py b/backend/backend/daemons/backend.py index 21f9958..bcf4d44 100644 --- a/backend/backend/daemons/backend.py +++ b/backend/backend/daemons/backend.py @@ -69,8 +69,6 @@ class CoprBackend(object): # what:str} self.frontend_client = FrontendClient(self.opts) self.is_running = False - if not os.path.exists(self.opts.worker_logdir): - os.makedirs(self.opts.worker_logdir, mode=0o750) self.log = get_redis_logger(self.opts, "backend.main", "backend") diff --git a/backend/backend/daemons/dispatcher.py b/backend/backend/daemons/dispatcher.py index 1b43ede..7cb5c65 100644 --- a/backend/backend/daemons/dispatcher.py +++ b/backend/backend/daemons/dispatcher.py @@ -47,29 +47,26 @@ class Worker(multiprocessing.Process): multiprocessing.Process.__init__(self, name="worker-builder") self.opts = opts + self.worker_num = worker_num + self.group_id = group_id + + self.log = get_redis_logger(self.opts, self.logger_name, "worker") # job management stuff self.task_queue = Queue("copr-be-{0}".format(str(group_id))) self.task_queue.connect() # event queue for communicating back to dispatcher - self.worker_num = worker_num - self.group_id = group_id - self.kill_received = False self.lock = lock self.frontend_client = frontend_client - - self.log = get_redis_logger(self.opts, self.logger_name, "worker") - self.vm_name = None self.vm_ip = None self.rc = None self.vmm = VmManager(self.opts) - @property def logger_name(self): return "backend.worker-{}-{}".format(self.group_name, self.worker_num) diff --git a/backend/backend/daemons/job_grab.py b/backend/backend/daemons/job_grab.py index 246e3e0..36aea77 100644 --- a/backend/backend/daemons/job_grab.py +++ b/backend/backend/daemons/job_grab.py @@ -147,10 +147,11 @@ class CoprJobGrab(Process): Retrieve tasks from frontend and runs appropriate handlers """ try: - r = get("{0}/waiting/".format(self.opts.frontend_url), + r = get("{0}/backend/waiting/".format(self.opts.frontend_base_url), auth=("user", self.opts.frontend_auth)) except RequestException as e: - self.log.exception("Error retrieving jobs from {}: {}".format(self.opts.frontend_url, e)) + self.log.exception("Error retrieving jobs from {}: {}" + .format(self.opts.frontend_base_url, e)) return try: diff --git a/backend/backend/daemons/log.py b/backend/backend/daemons/log.py index 8a5dedb..f74962e 100644 --- a/backend/backend/daemons/log.py +++ b/backend/backend/daemons/log.py @@ -21,83 +21,6 @@ from .. import helpers from ..constants import default_log_format -class CoprBackendLog(Process): - - """Log mechanism where items from the events queue get recorded - - :param Bunch opts: backend config - :param events: multiprocessing.Queue to listen - for events from other backend components - - """ - - def __init__(self, opts, events): - - # base class initialization - Process.__init__(self, name="logger") - - self.opts = opts - self.events = events - - logdir = os.path.dirname(self.opts.logfile) - if not os.path.exists(logdir): - os.makedirs(logdir, mode=0o750) - - def setup_log_handler(self): - """ - Configures standard python logger - """ - sys.stderr.write("Running setup handler {} \n".format(self.opts)) - # setup a log file to write to - logging.basicConfig(filename=self.opts.logfile + "_old.log", level=logging.DEBUG) - - self.log({"when": time.time(), "who": self.__class__.__name__, - "what": "Logger initiated"}) - - def log(self, event): - """ - Format event into the log message - - :param event: dict-like object - - Expected **event** keys: - - `when`: unixtime - - `who`: event producer [worker|logger|job|main] - - `what`: content - """ - when = time.strftime("%F %T", time.gmtime(event["when"])) - msg = "{0} : {1}: {2}".format(when, - event["who"], - event["what"].strip()) - try: - if self.opts.verbose: - sys.stderr.write("{0}\n".format(msg)) - sys.stderr.flush() - # logging.debug(msg) - - except (IOError, OSError) as e: - - sys.stderr.write("Could not write to logfile {0} - {1}\n".format( - self.opts.logfile, e)) - - # event format is a dict {when:time, who:[worker|logger|job|main], - # what:str} - def run(self): - """ - Starts logger process - """ - setproctitle("CoprLog") - self.setup_log_handler() - - try: - while True: - event = self.events.get() - if "when" in event and "who" in event and "what" in event: - self.log(event) - except KeyboardInterrupt: - return - - level_map = { "info": logging.INFO, "debug": logging.DEBUG, diff --git a/backend/backend/frontend.py b/backend/backend/frontend.py index 86d5e22..52c91b6 100644 --- a/backend/backend/frontend.py +++ b/backend/backend/frontend.py @@ -10,7 +10,7 @@ class FrontendClient(object): def __init__(self, opts): super(FrontendClient, self).__init__() - self.frontend_url = opts.frontend_url + self.frontend_url = "{}/backend".format(opts.frontend_base_url) self.frontend_auth = opts.frontend_auth self.msg = None @@ -21,7 +21,7 @@ class FrontendClient(object): """ headers = {"content-type": "application/json"} - url = "{0}/{1}/".format(self.frontend_url, url_path) + url = "{}/{}/".format(self.frontend_url, url_path) auth = ("user", self.frontend_auth) self.msg = None diff --git a/backend/backend/helpers.py b/backend/backend/helpers.py index 08b7794..3755113 100644 --- a/backend/backend/helpers.py +++ b/backend/backend/helpers.py @@ -104,15 +104,10 @@ class BackendConfigReader(object): opts = Bunch() opts.results_baseurl = _get_conf( - cp, "backend", "results_baseurl", "http://copr") + cp, "backend", "results_baseurl", "http://copr-be") - # TODO: this should be built from frontend_base_url + '/backend' - opts.frontend_url = _get_conf( - cp, "backend", "frontend_url", "http://coprs/rest/api") - - # We need this to access public api opts.frontend_base_url = _get_conf( - cp, "backend", "frontend_base_url", "http://coprs/") + cp, "backend", "frontend_base_url", "http://copr-fe") opts.frontend_auth = _get_conf( cp, "backend", "frontend_auth", "PASSWORDHERE") @@ -205,21 +200,10 @@ class BackendConfigReader(object): cp, "backend", "log_dir", "/var/log/copr/") opts.log_level = _get_conf( cp, "backend", "log_level", "info") - opts.logfile = _get_conf( - cp, "backend", "logfile", "/var/log/copr/backend.log") - opts.error_logfile = _get_conf( - cp, "backend", "error_logfile", "/var/log/copr/backend_error.log") opts.verbose = _get_conf( cp, "backend", "verbose", False, mode="bool") - opts.worker_logdir = _get_conf( - cp, "backend", "worker_logdir", "/var/log/copr/workers/") - opts.terminate_vars = _get_conf(cp, "backend", "terminate_vars", "").split(",") opts.prune_days = _get_conf(cp, "backend", "prune_days", None, mode="int") - opts.prune_script = _get_conf(cp, "backend", "prune_script", None, mode="path") - - opts.spawn_in_advance = _get_conf( - cp, "backend", "spawn_in_advance", False, mode="bool") # ssh options opts.ssh = Bunch() diff --git a/backend/conf/copr-be.conf.example b/backend/conf/copr-be.conf.example index 4a34206..1f9f768 100644 --- a/backend/conf/copr-be.conf.example +++ b/backend/conf/copr-be.conf.example @@ -5,7 +5,6 @@ results_baseurl=https://copr-be.cloud.fedoraproject.org/results # default is http://coprs/rest/api -frontend_url=https://copr-fe.cloud.fedoraproject.org/backend frontend_base_url=http://copr-fe-dev.cloud.fedoraproject.org # must have same value as BACKEND_PASSWORD from have frontend in /etc/copr/copr.conf @@ -43,16 +42,6 @@ group0_terminate_playbook=/srv/copr-work/provision/terminatepb-PC.yml group0_max_workers=8 group0_max_vm_total=2 -# That variables will be send to -# terminate_playbook ansible script. -# Currently supported parameters: -# - ip -# IP address of builder which should be terminated -# - vm_name -# name of VM (if provided by spawn script) -# default is None (empty) -#terminate_vars=ip - # directory where results are stored # should be accessible from web using 'results_baseurl' URL # no default @@ -63,13 +52,6 @@ destdir=/var/lib/copr/public_html/results # default is 10 sleeptime=30 -# path to log file -# default is /var/log/copr/backend.log -logfile=/var/log/copr/backend.log - -# default is /var/log/copr/workers/ -worker_logdir=/var/log/copr/workers/ - # exit on worker failure # default is false #exit_on_worker=false @@ -90,11 +72,6 @@ worker_logdir=/var/log/copr/workers/ # minimum age for builds to be pruned prune_days=14 -# Spawn builder in advance, before we get task? -# It save time, but consume resources even when -# nothing is in queue -#spawn_in_advance=false - [builder] # default is 1800 timeout=3600 diff --git a/backend/conf/copr-be.local.conf b/backend/conf/copr-be.local.conf index 0eeb2bf..7897e1c 100644 --- a/backend/conf/copr-be.local.conf +++ b/backend/conf/copr-be.local.conf @@ -5,7 +5,6 @@ results_baseurl=https://copr-be.cloud.fedoraproject.org/results # default is http://coprs/rest/api -frontend_url=https://copr-fe.cloud.fedoraproject.org/backend frontend_base_url=http://copr-fe-dev.cloud.fedoraproject.org # must have same value as BACKEND_PASSWORD from have frontend in /etc/copr/copr.conf @@ -41,17 +40,6 @@ group0_spawn_playbook=/srv/copr-work/provision/builderpb-PC.yml group0_terminate_playbook=/srv/copr-work/provision/terminatepb-PC.yml group0_max_workers=8 - -# That variables will be send to -# terminate_playbook ansible script. -# Currently supported parameters: -# - ip -# IP address of builder which should be terminated -# - vm_name -# name of VM (if provided by spawn script) -# default is None (empty) -#terminate_vars=ip - # directory where results are stored # should be accessible from web using 'results_baseurl' URL # no default @@ -65,13 +53,6 @@ sleeptime=30 # default is 8 num_workers=5 -# path to log file -# default is /var/log/copr/backend.log -logfile=/tmp/copr/backend.log - -# default is /var/log/copr/workers/ -worker_logdir=/tmp/log/copr/workers/ - # exit on worker failure # default is false #exit_on_worker=false diff --git a/backend/tests/deamons/test_backend.py b/backend/tests/deamons/test_backend.py index e723efe..d36797a 100644 --- a/backend/tests/deamons/test_backend.py +++ b/backend/tests/deamons/test_backend.py @@ -72,7 +72,6 @@ class TestBackend(object): self.mp_patcher = mock.patch("backend.daemons.backend.multiprocessing") self.mc_mp = self.mp_patcher.start() - self.worker_logdir = os.path.join(self.tmp_dir_path, "workers") self.config_file = "/dev/null/copr.conf" self.ext_opts = {} @@ -80,7 +79,6 @@ class TestBackend(object): self.bc_obj = MagicMock() self.opts = Bunch( - worker_logdir=self.worker_logdir, build_groups=[ { "id": 0, @@ -97,7 +95,6 @@ class TestBackend(object): ], exit_on_worker=False, sleeptime=1, - frontend_url="http://example.com/backend", frontend_base_url="http://example.com", frontend_auth="foobar", @@ -154,12 +151,7 @@ class TestBackend(object): self.be = CoprBackend(None, self.ext_opts) def test_constructor(self): - - assert not os.path.exists(self.worker_logdir) self.init_be() - assert os.path.exists(self.worker_logdir) - # import ipdb; ipdb.set_trace() - assert self.be.config_reader == self.bc_obj assert self.bc_obj.read.called @@ -173,6 +165,7 @@ class TestBackend(object): def test_clean_task_queue_ok(self, init_be): mc_queue = MagicMock(length=5) + def decr(): mc_queue.length -= 1 diff --git a/backend/tests/deamons/test_dispatcher.py b/backend/tests/deamons/test_dispatcher.py index 143043e..2f5221e 100644 --- a/backend/tests/deamons/test_dispatcher.py +++ b/backend/tests/deamons/test_dispatcher.py @@ -115,22 +115,19 @@ class TestDispatcher(object): self.terminate_pb = "/terminate.yml" self.opts = Bunch( ssh=Bunch(transport="paramiko"), - spawn_in_advance=False, - frontend_url="http://example.com/", + frontend_base_url="http://example.com", frontend_auth="12345678", build_groups={ - "3": { + 3: { "spawn_playbook": self.spawn_pb, "terminate_playbook": self.terminate_pb, "name": "3" } }, - terminate_vars=[], fedmsg_enabled=False, sleeptime=0.1, do_sign=True, - worker_logdir=self.tmp_dir_path, timeout=1800, destdir=self.tmp_dir_path, results_baseurl="/tmp", @@ -142,7 +139,7 @@ class TestDispatcher(object): self.try_spawn_args = '-c ssh {}'.format(self.spawn_pb) self.worker_num = 2 - self.group_id = "3" + self.group_id = 3 self.ip = "192.168.1.1" self.worker_callback = MagicMock() @@ -169,7 +166,6 @@ class TestDispatcher(object): self.worker.vmm = MagicMock() - def set_ip(*args, **kwargs): self.worker.vm_ip = self.vm_ip diff --git a/backend/tests/deamons/test_job_grab.py b/backend/tests/deamons/test_job_grab.py index a5736fe..fc06315 100644 --- a/backend/tests/deamons/test_job_grab.py +++ b/backend/tests/deamons/test_job_grab.py @@ -85,7 +85,6 @@ class TestJobGrab(object): self.log_file = os.path.join(self.log_dir, "copr.log") self.opts = Bunch( - logfile=self.log_file, verbose=False, build_groups=[ {"id": 0, "name": "x86", @@ -95,8 +94,7 @@ class TestJobGrab(object): "max_vm_per_user": 5}, ], destdir="/dev/null", - frontend_base_url="http://example.com/", - frontend_url="http://example.com/backend", + frontend_base_url="http://example.com", frontend_auth="foobar", results_baseurl="http://example.com/results/", sleeptime=1, diff --git a/backend/tests/deamons/test_log.py b/backend/tests/deamons/test_log.py index 30002c2..8b4ed52 100644 --- a/backend/tests/deamons/test_log.py +++ b/backend/tests/deamons/test_log.py @@ -20,7 +20,7 @@ else: import pytest import backend.daemons.log as log_module -from backend.daemons.log import CoprBackendLog, RedisLogHandler +from backend.daemons.log import RedisLogHandler @pytest.yield_fixture @@ -52,9 +52,7 @@ class TestLog(object): self.log_dir = os.path.join(self.tmp_dir_path, "copr") self.log_file = os.path.join(self.log_dir, "copr.log") self.opts = Bunch( - logfile=self.log_file, verbose=False, - log_dir=self.log_dir ) print("\n log dir: {}".format(self.log_dir)) @@ -72,7 +70,7 @@ class TestLog(object): @pytest.fixture def init_log(self): - self.cbl = CoprBackendLog(self.opts, self.queue) + pass # todo: replace with RedisLogHandler + helpers.get_redis_logger # def test_constructor(self): diff --git a/backend/tests/test_frontend.py b/backend/tests/test_frontend.py index ad85e4b..b2ee80f 100644 --- a/backend/tests/test_frontend.py +++ b/backend/tests/test_frontend.py @@ -35,7 +35,7 @@ class TestFrontendClient(object): def setup_method(self, method): self.opts = Bunch( - frontend_url="http://example.com/", + frontend_base_url="http://example.com/", frontend_auth="12345678", ) self.fc = FrontendClient(self.opts) diff --git a/backend/tests/vm_manager/test_check.py b/backend/tests/vm_manager/test_check.py index e7d6172..2f20080 100644 --- a/backend/tests/vm_manager/test_check.py +++ b/backend/tests/vm_manager/test_check.py @@ -84,7 +84,6 @@ class TestChecker(object): "terminate_playbook": self.terminate_pb_path, "name": "base", "archs": ["i386", "x86_64"], - # "terminate_vars": ["vm_name", "ip"], } }, diff --git a/backend/tests/vm_manager/test_event_handle.py b/backend/tests/vm_manager/test_event_handle.py index 9db7f12..f1e7192 100644 --- a/backend/tests/vm_manager/test_event_handle.py +++ b/backend/tests/vm_manager/test_event_handle.py @@ -89,14 +89,12 @@ class TestEventHandle(object): "name": "base", "archs": ["i386", "x86_64"], "vm_max_check_fails": 2, - # "terminate_vars": ["vm_name", "ip"], } }, fedmsg_enabled=False, sleeptime=0.1, do_sign=True, - # worker_logdir=self., timeout=1800, # destdir=self.tmp_dir_path, results_baseurl="/tmp", diff --git a/backend/tests/vm_manager/test_manager.py b/backend/tests/vm_manager/test_manager.py index 270d087..787cb6a 100644 --- a/backend/tests/vm_manager/test_manager.py +++ b/backend/tests/vm_manager/test_manager.py @@ -65,7 +65,6 @@ class TestManager(object): fedmsg_enabled=False, sleeptime=0.1, do_sign=True, - # worker_logdir=self., timeout=1800, # destdir=self.tmp_dir_path, results_baseurl="/tmp", diff --git a/backend/tests/vm_manager/test_spawn.py b/backend/tests/vm_manager/test_spawn.py index 729315b..3b9e849 100644 --- a/backend/tests/vm_manager/test_spawn.py +++ b/backend/tests/vm_manager/test_spawn.py @@ -87,7 +87,6 @@ class TestSpawner(object): fedmsg_enabled=False, sleeptime=0.1, do_sign=True, - # worker_logdir=self., timeout=1800, # destdir=self.tmp_dir_path, results_baseurl="/tmp", diff --git a/backend/tests/vm_manager/test_terminate.py b/backend/tests/vm_manager/test_terminate.py index 6ef716c..ef79ad4 100644 --- a/backend/tests/vm_manager/test_terminate.py +++ b/backend/tests/vm_manager/test_terminate.py @@ -82,14 +82,12 @@ class TestTerminate(object): "terminate_playbook": self.terminate_pb_path, "name": "base", "archs": ["i386", "x86_64"], - # "terminate_vars": ["vm_name", "ip"], } }, fedmsg_enabled=False, sleeptime=0.1, do_sign=True, - # worker_logdir=self., timeout=1800, # destdir=self.tmp_dir_path, results_baseurl="/tmp",

1 0

[copr] tag 'copr-backend-1.68-1' created
by vgologuz＠fedoraproject.org 28 May '15

28 May '15

Repository : http://git.fedorahosted.org/cgit/copr.git New tag : copr-backend-1.68-1 Referencing: fd4499a1f4c07efdbd1532d5b4163f75cca0870e

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

copr-commits May 2015