copr-devel December 2012

copr-devel@lists.fedorahosted.org

4 participants
47 discussions

[copr] skvidal-backend: more todo's highlighting a bug (b336c6d)
by Seth Vidal 08 Dec '12

08 Dec '12

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : skvidal-backend >--------------------------------------------------------------- commit b336c6d22bc863b44410089d936504d2b2065c2b Author: Seth Vidal <skvidal(a)fedoraproject.org> Date: Sat Dec 8 13:04:44 2012 -0500 more todo's highlighting a bug >--------------------------------------------------------------- TODO-backend | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/TODO-backend b/TODO-backend index 058330e..4cc5ca1 100644 --- a/TODO-backend +++ b/TODO-backend @@ -15,3 +15,5 @@ - work on a way to find and cancel a specific build that's happening other than just killing the instance - determine if it is properly checking the timeout from a dead instance - maybe dump out the PID of the worker that is running so we know which one to kill? +- failure/success not being returned correctly. Should check for 'fail' in the directories and return based on + that. also anything lacking success is a failure.

1 0

[copr] skvidal-backend: add another TODO note (04b2f89)
by skvidal＠fedorahosted.org 07 Dec '12

07 Dec '12

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : skvidal-backend >--------------------------------------------------------------- commit 04b2f891e06aaa4ea79188d641594af34d24fba7 Author: Seth Vidal <skvidal(a)fedoraproject.org> Date: Fri Dec 7 17:05:54 2012 -0500 add another TODO note >--------------------------------------------------------------- TODO-backend | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/TODO-backend b/TODO-backend index a978e13..058330e 100644 --- a/TODO-backend +++ b/TODO-backend @@ -14,3 +14,4 @@ - refactor mockremote/dispatcher.worker together? - work on a way to find and cancel a specific build that's happening other than just killing the instance - determine if it is properly checking the timeout from a dead instance +- maybe dump out the PID of the worker that is running so we know which one to kill?

1 0

[copr] skvidal-backend: remove/add todo's (25b9a4f)
by Seth Vidal 07 Dec '12

07 Dec '12

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : skvidal-backend >--------------------------------------------------------------- commit 25b9a4f18ee342a0bbc797c796f27ec8756ee705 Author: Seth Vidal <skvidal(a)fedoraproject.org> Date: Fri Dec 7 16:47:52 2012 -0500 remove/add todo's >--------------------------------------------------------------- TODO-backend | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/TODO-backend b/TODO-backend index e532fc9..a978e13 100644 --- a/TODO-backend +++ b/TODO-backend @@ -7,9 +7,10 @@ - LOADS of fixme and catching weird conditions - make logging from mockremote more sane and coinsistent - mock configs should be pushed to instances at creation time - - should include enabled package_state pluging - single url to repos, not mirrorlists - consider making each worker return job to a completed queue so the primary process can do other kinds of notification - email notifications from backend? - refactor mockremote/dispatcher.worker together? +- work on a way to find and cancel a specific build that's happening other than just killing the instance +- determine if it is properly checking the timeout from a dead instance

1 0

[copr] skvidal-backend: log how long the instance spawn takes (24fbf8d)
by Seth Vidal 07 Dec '12

07 Dec '12

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : skvidal-backend >--------------------------------------------------------------- commit 24fbf8d00b341e931ab436d21836e025f91343f0 Author: Seth Vidal <skvidal(a)fedoraproject.org> Date: Fri Dec 7 10:48:52 2012 -0500 log how long the instance spawn takes >--------------------------------------------------------------- backend/dispatcher.py | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/backend/dispatcher.py b/backend/dispatcher.py index a62f5f1..9b0399d 100644 --- a/backend/dispatcher.py +++ b/backend/dispatcher.py @@ -105,6 +105,7 @@ class Worker(multiprocessing.Process): def spawn_instance(self): """call the spawn playbook to startup/provision a building instance""" self.callback.log('spawning instance begin') + start = time.time() stats = callbacks.AggregateStats() playbook_cb = SilentPlaybookCallbacks(verbose=False) @@ -117,6 +118,8 @@ class Worker(multiprocessing.Process): play.run() self.callback.log('spawning instance end') + self.callback.log('Instance spawn/provision took %s sec' % (time.time() - start)) + if self.ip: return self.ip

1 0

[copr] skvidal-backend: add TODO list from my paper lists (cc1f0ba)
by skvidal＠fedorahosted.org 06 Dec '12

06 Dec '12

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : skvidal-backend >--------------------------------------------------------------- commit cc1f0ba6e55734f7f453a47321cbd778b6453de3 Author: Seth Vidal <skvidal(a)fedoraproject.org> Date: Thu Dec 6 17:25:35 2012 -0500 add TODO list from my paper lists >--------------------------------------------------------------- TODO-backend | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-) diff --git a/TODO-backend b/TODO-backend new file mode 100644 index 0000000..e532fc9 --- /dev/null +++ b/TODO-backend @@ -0,0 +1,15 @@ + +- change instance type by build request for more mem/procs/extend timeouts + - use extra-vars? + - need ansible 0.9? +- auto-timer/cleanup script for old instances that may have been orphaned +- prune out builders when we drop the number of them active +- LOADS of fixme and catching weird conditions +- make logging from mockremote more sane and coinsistent +- mock configs should be pushed to instances at creation time + - should include enabled package_state pluging + - single url to repos, not mirrorlists +- consider making each worker return job to a completed queue so the primary + process can do other kinds of notification +- email notifications from backend? +- refactor mockremote/dispatcher.worker together?

1 0

[copr] skvidal-backend: example config file (85be22f)
by skvidal＠fedorahosted.org 06 Dec '12

06 Dec '12

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : skvidal-backend >--------------------------------------------------------------- commit 85be22f2287f53a38538194fb7bdc6000c09707d Author: Seth Vidal <skvidal(a)fedoraproject.org> Date: Thu Dec 6 16:49:48 2012 -0500 example config file >--------------------------------------------------------------- copr-be.conf.example | 16 ++++++++++++++++ 1 files changed, 16 insertions(+), 0 deletions(-) diff --git a/copr-be.conf.example b/copr-be.conf.example new file mode 100644 index 0000000..ffeb235 --- /dev/null +++ b/copr-be.conf.example @@ -0,0 +1,16 @@ +[backend] +results_baseurl=http://copr-be.cloud.fedoraproject.org/results +frontend_url=http://copr-fe.cloud.fedoraproject.org/backend +frontend_auth=backend_password_from_fe_config +spawn_playbook=/srv/copr-work/provision/builderpb.yml +terminate_playbook=/srv/copr-work/provision/terminatepb.yml +jobsdir=/srv/copr-work/jobs +destdir=/srv/copr-repo/results +sleeptime=30 +num_workers=5 +logfile=/srv/copr-work/logs/copr.log +worker_logdir=/srv/copr-work/logs/workers/ + + +[builder] +timeout=3600

1 0

[copr] skvidal-backend: output what pkgs we're building to the logs (3094750)
by Seth Vidal 06 Dec '12

06 Dec '12

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : skvidal-backend >--------------------------------------------------------------- commit 3094750f307303f0f1058240f4787a4e7405505a Author: Seth Vidal <skvidal(a)fedoraproject.org> Date: Thu Dec 6 16:41:10 2012 -0500 output what pkgs we're building to the logs >--------------------------------------------------------------- backend/dispatcher.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/backend/dispatcher.py b/backend/dispatcher.py index c113e8c..a62f5f1 100644 --- a/backend/dispatcher.py +++ b/backend/dispatcher.py @@ -272,7 +272,8 @@ class Worker(multiprocessing.Process): # start the build - most importantly license checks. - self.callback.log('Starting build: builder=%r timeout=%r destdir=%r chroot=%r repos=%r' % (ip, job.timeout, job.destdir, chroot, str(job.repos))) + self.callback.log('Starting build: id=%r builder=%r timeout=%r destdir=%r chroot=%r repos=%r' % (job.build_id,ip, job.timeout, job.destdir, chroot, str(job.repos))) + self.callback.log('building pkgs: %s' % ' '.join(job.pkgs)) try: chrootlogfile = chroot_destdir + '/mockremote.log' mr = mockremote.MockRemote(builder=ip, timeout=job.timeout,

1 0

[copr] skvidal-backend: post status to frontend using python-requests (707d001)
by Seth Vidal 06 Dec '12

06 Dec '12

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : skvidal-backend >--------------------------------------------------------------- commit 707d00179d06651528caa41cae30ca573afcd83d Author: Seth Vidal <skvidal(a)fedoraproject.org> Date: Thu Dec 6 15:16:16 2012 -0500 post status to frontend using python-requests >--------------------------------------------------------------- backend/dispatcher.py | 61 +++++++++++++++++++++++++++++++++++++++--------- 1 files changed, 49 insertions(+), 12 deletions(-) diff --git a/backend/dispatcher.py b/backend/dispatcher.py index db800cc..c113e8c 100644 --- a/backend/dispatcher.py +++ b/backend/dispatcher.py @@ -3,7 +3,6 @@ import os import sys -import shutil import multiprocessing import time import Queue @@ -15,13 +14,11 @@ import ansible import ansible.playbook import ansible.errors from ansible import callbacks +import requests - -#FIXME - this should be emitting to the per-worker log file -# so we can know what is going on and where class SilentPlaybookCallbacks(callbacks.PlaybookCallbacks): ''' playbook callbacks - quietly! ''' @@ -162,22 +159,60 @@ class Worker(multiprocessing.Process): jobdata.timeout = build['timeout'] jobdata.destdir = self.opts.destdir + '/' + build['copr']['owner']['name'] + '/' + build['copr']['name'] + '/' jobdata.build_id = build['id'] + jobdata.results = self.opts.results_baseurl + '/' + build['copr']['owner']['name'] + '/' + build['copr']['name'] + '/' jobdata.copr_id = build['copr']['id'] jobdata.user_id = build['user_id'] return jobdata + # maybe we move this to the callback? + def post_to_frontend(self, data): + """send data to frontend""" + + headers = {'content-type': 'application/json'} + url='%s/update_builds/' % self.opts.frontend_url + auth=('user', self.opts.frontend_auth) + + msg = None + try: + r = requests.post(url, data=json.dumps(data), auth=auth, + headers=headers) + if r.status_code != 200: + msg = 'Failed to submit to frontend: %s: %s' % (r.status_code, r.text) + except requests.RequestException, e: + msg = 'Post request failed: %s' % e + + if msg: + self.callback.log(msg) + return False + + return True + + # maybe we move this to the callback? + def mark_started(self, job): + + build = {'id':job.build_id, + 'started_on': job.started_on, + 'results': job.results, + } + data = {'builds':[build]} + + if not self.post_to_frontend(data): + raise errors.CoprWorkerError, "Could not communicate to front end to submit status info" + + # maybe we move this to the callback? def return_results(self, job): - """write out a completed json file to the results dir and submit the results to the frontend""" self.callback.log('%s status %s. Took %s seconds' % (job.build_id, job.status, job.ended_on - job.started_on)) - jobfilename = os.path.basename(job.jobfile) - shutil.move(job.jobfile, job.destdir + '/' + jobfilename) - # reform json file with completed info (completed time, url, status, etc) - # post it via requests to url with passcode - # profit + build = {'id':job.build_id, + 'ended_on': job.ended_on, + 'status': job.status, + } + data = {'builds':[build]} + + if not self.post_to_frontend(data): + raise errors.CoprWorkerError, "Could not communicate to front end to submit results" - #FIXME - this should either return job status/results - # into a queue or it should submit results directly to the frontend + os.unlink(job.jobfile) def run(self): # worker should startup and check if it can function @@ -214,6 +249,8 @@ class Worker(multiprocessing.Process): status = 1 job.started_on = time.time() + self.mark_started(job) + for chroot in job.chroots: chroot_destdir = job.destdir + '/' + chroot

1 0

[copr] skvidal-backend: - touch up how mockremote logs downloads to make it less confusing - note for tomorrow - (9515168)
by Seth Vidal 06 Dec '12

06 Dec '12

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : skvidal-backend >--------------------------------------------------------------- commit 9515168b490f853813aaed3b0605160e2245a3d3 Author: Seth Vidal <skvidal(a)fedoraproject.org> Date: Thu Dec 6 02:51:28 2012 -0500 - touch up how mockremote logs downloads to make it less confusing - note for tomorrow - >--------------------------------------------------------------- backend/dispatcher.py | 5 +++++ backend/mockremote.py | 4 ++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/backend/dispatcher.py b/backend/dispatcher.py index 69aafd6..db800cc 100644 --- a/backend/dispatcher.py +++ b/backend/dispatcher.py @@ -171,6 +171,11 @@ class Worker(multiprocessing.Process): self.callback.log('%s status %s. Took %s seconds' % (job.build_id, job.status, job.ended_on - job.started_on)) jobfilename = os.path.basename(job.jobfile) shutil.move(job.jobfile, job.destdir + '/' + jobfilename) + + # reform json file with completed info (completed time, url, status, etc) + # post it via requests to url with passcode + # profit + #FIXME - this should either return job status/results # into a queue or it should submit results directly to the frontend diff --git a/backend/mockremote.py b/backend/mockremote.py index 3d39f3d..2c641c8 100755 --- a/backend/mockremote.py +++ b/backend/mockremote.py @@ -187,11 +187,11 @@ class CliLogCallBack(DefaultCallBack): self.log(msg) def start_download(self, pkg): - msg = "Start Download: %s" % pkg + msg = "Start retrieve results for: %s" % pkg self.log(msg) def end_download(self, pkg): - msg = "End Download: %s" % pkg + msg = "End retrieve results for: %s" % pkg self.log(msg) def error(self, msg):

1 0

[copr] skvidal-backend: log output a little simpler to read (c4f21e6)
by skvidal＠fedorahosted.org 06 Dec '12

06 Dec '12

Repository : http://git.fedorahosted.org/cgit/copr.git On branch : skvidal-backend >--------------------------------------------------------------- commit c4f21e6c2b94af0a400d70730d21f0b6f2b437fc Author: Seth Vidal <skvidal(a)fedoraproject.org> Date: Thu Dec 6 02:34:48 2012 -0500 log output a little simpler to read >--------------------------------------------------------------- backend/dispatcher.py | 2 +- copr-be.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/backend/dispatcher.py b/backend/dispatcher.py index fc9a958..69aafd6 100644 --- a/backend/dispatcher.py +++ b/backend/dispatcher.py @@ -76,7 +76,7 @@ class WorkerCallback(object): now = time.strftime('%F %T') try: - open(self.logfile, 'a').write(str(now) + ':' + msg + '\n') + open(self.logfile, 'a').write(str(now) + ': ' + msg + '\n') except (IOError, OSError), e: print >>sys.stderr, 'Could not write to logfile %s - %s' % (self.logfile, str(e)) diff --git a/copr-be.py b/copr-be.py index 6564134..6631ac4 100644 --- a/copr-be.py +++ b/copr-be.py @@ -92,7 +92,7 @@ class CoprBackend(object): def log(self, msg): now = time.strftime('%F %T') - output = str(now) + ':' + msg + output = str(now) + ': ' + msg if not self.opts.daemonize: print output

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

copr-devel December 2012