Here is my plan for upgrading fedmsg/datanommer/datagrepper. I'm not sure it'll be able to get done before freeze on Tuesday at this point... but I'd like to try to. I need somebody else to tell me if this is crazy.
Everything is working as expected in staging. Production has two servers, which adds a little bit more difficulty.
stage 0: preliminary things - need a sysadmin-db to create a datagrepper database - need to know domain/port for a memcached server - need shared storage on datagrepper{01,02} in /var/www/datagrepper_output (only datagrepper01 will be writing to it, but both will need to serve it via httpd)
stage 1: upgrade fedmsg and datanommer on busgateway01 - yum update fedmsg python-datanommer-{models,consumer} - run the two alembic upgrades (which need to be run in reverse order) - restart all the fedmsg services - verify everything is working: - fedmsg irc is spitting out messages - datanommer database is receiving new messages (verify with datanommer-latest)
stage 2: upgrade datagrepper on datagrepper{01,02} - add datagrepper::runner to datagrepper01 manifest - merge datagrepper from modules-staging into modules - yum update fedmsg python-datanommer-models datagrepper - create tables in datagrepper database - run-puppet nowait
stage 3: upgrade fedmsg on the rest of production - a sysadmin-main (presumably) will need to do this
On Sat, 12 Oct 2013 16:52:40 -0500 Ian Weller ian@ianweller.org wrote:
Here is my plan for upgrading fedmsg/datanommer/datagrepper. I'm not sure it'll be able to get done before freeze on Tuesday at this point... but I'd like to try to. I need somebody else to tell me if this is crazy.
You're crazy! :)
just kidding.
Everything is working as expected in staging. Production has two servers, which adds a little bit more difficulty.
stage 0: preliminary things
- need a sysadmin-db to create a datagrepper database
I think toshio has this set for you now.
- need to know domain/port for a memcached server
Either memcached03 or memcached04.
- need shared storage on datagrepper{01,02} in /var/www/datagrepper_output (only datagrepper01 will be writing to
it, but both will need to serve it via httpd)
I've setup a shared gluster volume for this.
stage 1: upgrade fedmsg and datanommer on busgateway01
- yum update fedmsg python-datanommer-{models,consumer}
- run the two alembic upgrades (which need to be run in reverse order)
- restart all the fedmsg services
- verify everything is working:
- fedmsg irc is spitting out messages
- datanommer database is receiving new messages (verify with datanommer-latest)
Sounds good.
stage 2: upgrade datagrepper on datagrepper{01,02}
- add datagrepper::runner to datagrepper01 manifest
- merge datagrepper from modules-staging into modules
- yum update fedmsg python-datanommer-models datagrepper
- create tables in datagrepper database
- run-puppet nowait
ok.
stage 3: upgrade fedmsg on the rest of production
- a sysadmin-main (presumably) will need to do this
I think we are already updated from the mass updates last week, but if not we can do this.
kevin
On Mon, Oct 14, 2013 at 03:12:03PM -0600, Kevin Fenzi wrote:
On Sat, 12 Oct 2013 16:52:40 -0500 Ian Weller ian@ianweller.org wrote:
Here is my plan for upgrading fedmsg/datanommer/datagrepper. I'm not sure it'll be able to get done before freeze on Tuesday at this point... but I'd like to try to. I need somebody else to tell me if this is crazy.
You're crazy! :)
just kidding.
Everything is working as expected in staging. Production has two servers, which adds a little bit more difficulty.
stage 0: preliminary things
- need a sysadmin-db to create a datagrepper database
I think toshio has this set for you now.
- need to know domain/port for a memcached server
Either memcached03 or memcached04.
- need shared storage on datagrepper{01,02} in /var/www/datagrepper_output (only datagrepper01 will be writing to
it, but both will need to serve it via httpd)
I've setup a shared gluster volume for this.
stage 1: upgrade fedmsg and datanommer on busgateway01
- yum update fedmsg python-datanommer-{models,consumer}
- run the two alembic upgrades (which need to be run in reverse order)
- restart all the fedmsg services
- verify everything is working:
- fedmsg irc is spitting out messages
- datanommer database is receiving new messages (verify with datanommer-latest)
Sounds good.
stage 2: upgrade datagrepper on datagrepper{01,02}
- add datagrepper::runner to datagrepper01 manifest
- merge datagrepper from modules-staging into modules
- yum update fedmsg python-datanommer-models datagrepper
- create tables in datagrepper database
- run-puppet nowait
ok.
stage 3: upgrade fedmsg on the rest of production
- a sysadmin-main (presumably) will need to do this
I think we are already updated from the mass updates last week, but if not we can do this.
Yes, fedmsg is 0.7.0 across the board.
Status at end of day today: - datanommer 0.6.0 is in production - datagrepper 0.2.1 is in production
Various bugfixes and enhancements happened. My favorite: each message now has a permalink: https://apps.fedoraproject.org/datagrepper/id?id=2013-a441f82b-9a42-41c7-a05...
Logging in doesn't work since Flask-OpenID's way of handling stateless auth stores data in /tmp/flask-openid on the system which doesn't work when you have multiple servers. Planning to move to the FAS-OpenID plugin for Flask. So the job queue system can't even be used yet -- but when I get that fixed, it's ready :)
Additionally there are some bugs with overreaching exception handlers on long-running jobs, so that'll be taken care of as well.
Fortunately all the things that have bugs are new features, and everything that used to work still does. :)
datagrepper 0.2.2 should be out the door this week, and not long after we should see a change request on this list (since freeze starts tomorrow).
infrastructure@lists.fedoraproject.org