Justin,
I don't know if this is the cause or not, but I will tell you what I've
done in case it helps someone. My first commit to badges was figuring
out that the script that ran daily to check for the X Year Badger badges
wasnt running. The length of the list was so long that the connection
timed out just trying to pull down the list of contributors to check.
My fix was to just loop over the alphabet and pull in all of the "A"
names, all the way to the "Z" names. I imagine that if the script fails
to run now, for whatever reason, it does it 26 times. :)
A better solution could involve ( and Im doing this from memory, so
apologies if this is a wild goose chase )
- having puiterwijk or some other person smart about DBs write a query
to pull out all of the badge anniversary dates and put a text file of
results somewhere this script could pull in
- having the loop check each time through to see if the database
responds, waiting in between responses, before giving up
- rewriting all of fedmsg in python3 ( maybe dont do this - I think it
would be painful )
I haven't been around much lately due to work/life, but I just updated
my ssh key. If I haven't been removed from the badges-admin group for
being largely absent for several months. (which I believe is the primary
reason people are removed) I will try to look at it this week.
This would be a good task for a new contributor to look into as well. I
don't remember if all the scripts for badges are in ansible, but I
imagine they are.
Regards,
Zach
--
Zach Villers
zach@znix.site
On Mon, Apr 10, 2017, at 07:44 PM, Justin W. Flory wrote:
> Hi all,
>
> I had about 150-200 of these emails come in over the last few minutes.
> Is there any kind of action we can take to quiet these down or get to
> the bottom of what's causing them? This also happens intermittently
> outside of the past 30 minutes too.
>
> Thanks!
>
>
> -------- Forwarded Message --------
> Subject: fedmsg error log badges-backend01.phx2.fedoraproject.org
> Date: Mon, 10 Apr 2017 23:41:09 -0000
> From: fedmsg@fedoraproject.org
> To: sysadmin-badges-members@fedoraproject.org
>
> Message
> -------
> [2017-04-10 23:41:09][moksha.hub ERROR]
> ```python
> Couldn't talk to pkgdb2 for u'mbs/mbs.fedoraproject.org', 404,
> u'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>404
> Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not
> found on the server. If you entered the URL manually please check your
> spelling and try again.</p>\n'
> ```
>
>
> Process Details
> ---------------
> - host: badges-backend01.phx2.fedoraproject.org
> - PID: 31829
> - name: fedmsg-hub
> - command: /usr/bin/python /usr/bin/fedmsg-hub
> - msg_id: 2017-18d2a3e1-2cb0-45a2-97d8-aba549a4e8fb
>
> Callstack that lead to the logging statement
> --------------------------------------------
> ```python
> File "/usr/lib64/python2.7/threading.py", line 784 in __bootstrap
> self.__bootstrap_inner()
> File "/usr/lib64/python2.7/threading.py", line 811 in __bootstrap_inner
> self.run()
> File "/usr/lib64/python2.7/threading.py", line 764 in run
> self.__target(*self.__args, **self.__kwargs)
> File
> "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line
> 167 in _worker
> result = context.call(ctx, function, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/twisted/python/context.py",
> line 118 in callWithContext
> return self.currentContext().callWithContext(ctx, func, *args, **kw)
> File "/usr/lib64/python2.7/site-packages/twisted/python/context.py",
> line 81 in callWithContext
> return func(*args,**kw)
> File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py",
> line 191 in _work
> self.consume(message)
> File "/usr/lib/python2.7/site-packages/fedbadges/consumers.py", line
> 237 in consume
> for recipient in badge_rule.matches(msg):
> File "/usr/lib/python2.7/site-packages/fedbadges/rules.py", line 236
> in matches
> if not self.criteria.matches(msg):
> File "/usr/lib/python2.7/site-packages/fedbadges/utils.py", line 94 in
> inner
> return method(self, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/fedbadges/rules.py", line 368
> in matches
> child.matches(msg) for child in self.children
> File "/usr/lib/python2.7/site-packages/fedbadges/rules.py", line 66 in
> <lambda>
> "not": lambda x: all([not item for item in x])
> File "/usr/lib/python2.7/site-packages/fedbadges/rules.py", line 368
> in <genexpr>
> child.matches(msg) for child in self.children
> File "/usr/lib/python2.7/site-packages/fedbadges/utils.py", line 96 in
> inner
> log.exception(e)
> ```
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/fedbadges/utils.py", line 94,
> in inner
> return method(self, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/fedbadges/rules.py", line 371,
> in matches
> return self.specialization.matches(msg)
> File "/usr/lib/python2.7/site-packages/fedbadges/rules.py", line 418,
> in matches
> user=expectation['user'],
> File "/usr/lib/python2.7/site-packages/fedbadges/utils.py", line 145,
> in get_pkgdb_packages_for
> return _getter(user)
> File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line
> 1053, in decorate
> should_cache_fn)
> File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line
> 657, in get_or_create
> async_creator) as value:
> File "/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py", line
> 158, in __enter__
> return self._enter()
> File "/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py", line
> 98, in _enter
> generated = self._enter_create(createdtime)
> File "/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py", line
> 149, in _enter_create
> created = self.creator()
> File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line
> 625, in gen_value
> created_value = creator()
> File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line
> 1049, in creator
> return fn(*arg, **kw)
> File "/usr/lib/python2.7/site-packages/fedbadges/utils.py", line 143,
> in _getter
> return _get_pkgdb2_packages_for(config, user)
> File "/usr/lib/python2.7/site-packages/fedbadges/utils.py", line 166,
> in _get_pkgdb2_packages_for
> data = _get_page(1)
> File "/usr/lib/python2.7/site-packages/fedbadges/utils.py", line 159,
> in _get_page
> username, req.status_code, req.text))
> IOError: Couldn't talk to pkgdb2 for u'mbs/mbs.fedoraproject.org', 404,
> u'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>404
> Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not
> found on the server. If you entered the URL manually please check your
> spelling and try again.</p>\n'
>
> _______________________________________________
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to
> infrastructure-leave@lists.fedoraproject.org
> Email had 1 attachment:
> + signature.asc
> 1k (application/pgp-signature)
Zach Villers