Hey team,
Next week I'll be going offline for 3 weeks (two of them being really off the grid, I'm going to the European equivalent of the Burning Man festival...), so I'd like to check with you what errors I've seen our production Mailman3 instance do these last weeks.
I think that the main issue I haven't fixed yet is the fact that sometimes, for an unknown reason (yet), the archiving process dies, and email isn't archived anymore. The posts are still relayed to the subscribers, but the email are not archived. This can be detected by two ways:
* there is no running process called "/usr/bin/python3.4 /usr/libexec/mailman3/runner --runner=archive:0:1 -C /etc/mailman.cfg" (see the "runner" option on the command line)
* the "/var/spool/mailman3/archive/" directory starts filling up with files. It's usually empty.
The best way to fix this is currently to restart the mailman process with "systemctl restart mailman3". Then the unarchived emails in the backlog will be processed, the speed is variable but it's around 1 or 2 per second.
The other problems that I've been pinged about recently are really specific, it's either configuration problems or bugs that I've fixed now. When it's about emails being rejected, it's usually a good idea to start looking at the list's header filters (in Postorius).
If you have any questions, feel free to ask me. I'm leaving for the desert on June 30th, I'll start being mildly reacheable on July 13th, and will be back on Monday 18th.
Thanks!
Aurélien
On Thu, 23 Jun 2016 16:08:17 +0200 Aurelien Bompard aurelien@bompard.org wrote:
Hey team,
Next week I'll be going offline for 3 weeks (two of them being really off the grid, I'm going to the European equivalent of the Burning Man festival...), so I'd like to check with you what errors I've seen our production Mailman3 instance do these last weeks.
Sounds fun.
I think that the main issue I haven't fixed yet is the fact that sometimes, for an unknown reason (yet), the archiving process dies, and email isn't archived anymore.
...snip...
Yeah, we can watch for this.
The other problems that I've been pinged about recently are really specific, it's either configuration problems or bugs that I've fixed now. When it's about emails being rejected, it's usually a good idea to start looking at the list's header filters (in Postorius).
If you have any questions, feel free to ask me. I'm leaving for the desert on June 30th, I'll start being mildly reacheable on July 13th, and will be back on Monday 18th.
ok. Have safe travels. ;)
Oh, while I am looking at them... how about I go over the outstanding mailman3 bugs in trac we have?
https://fedorahosted.org/fedora-infrastructure/ticket/1002 "Enable syndication of fedora mailing lists"
* This is an old ticket and we had some hacky perl script to try and do rss feeds with lists. Does HK support rss? Is this a feature we can ask for upstream?
https://fedorahosted.org/fedora-infrastructure/ticket/4970 "where is list administration interface"
* perl-devel is not notifying on held messages. I am also seeing this with the 'websites' list. Should we move this to an upstream bug?
https://fedorahosted.org/fedora-infrastructure/ticket/4971 "Mailman 3 mbox archives have invalid Date: value"
* This says a fix was commited. Is it live now? If so, we can close?
https://fedorahosted.org/fedora-infrastructure/ticket/4975 "New subscription notifications from fedora-join mailing list come from root@localhost.fedoraproject.org"
* admin emails from root@localhost. ;( There's a dupe of this I just noticed below. ;) Closing the dupe, but should we file an upstream bug? or is this a config problem locally?
https://fedorahosted.org/fedora-infrastructure/ticket/4976 "Mailman 3 does accept confirmation message for discarding a moderated post"
* This has a upstream bug: https://gitlab.com/mailman/mailman/issues/169 So I closed it. ;)
https://fedorahosted.org/fedora-infrastructure/ticket/5341 "main translation page isntt available when searching on https://lists.fedoraproject.org/archives/ "
* Not sure what to do here. Perhaps we could ask Mo or Ryan to weigh in and see if there's some better way to do the UI for it? Or just close it with a cannot fix for now?
https://fedorahosted.org/fedora-infrastructure/ticket/5345 "firewalld-devel list archives not working properly"
* Not sure whats going on here. Any news on your investigations?
https://fedorahosted.org/fedora-infrastructure/ticket/5070 "Request: run MetricsGrimoire mlstats against all mailing lists monthly; provide access to database"
* Not sure where we are with this one. Does Mattdm have access to do the stats? Should we set them up to run automagically?
https://fedorahosted.org/fedora-infrastructure/ticket/2079 "querying the list of all the aliases that a given FAS id is on"
* Not sure how to handle this one either. It's an old one and would require querying lots of places. It's not really mailing list related fully either.
https://fedorahosted.org/fedora-infrastructure/ticket/5338 "mailman moderation mails From:"
* Closed duplicate.
https://fedorahosted.org/fedora-infrastructure/ticket/5304 "Subscribe a mailing list to an RSS feed"
* This is more a wordpress thing than mailing list.
https://fedorahosted.org/fedora-infrastructure/ticket/5169 "Fedora-centric favicon for hyperkitty"
* I think this is waiting for Mattdm to chime in.
Those are all the mailing list ones. ;)
kevin
ok. Have safe travels. ;)
Thanks!
Oh, while I am looking at them... how about I go over the outstanding mailman3 bugs in trac we have?
https://fedorahosted.org/fedora-infrastructure/ticket/1002 "Enable syndication of fedora mailing lists"
- This is an old ticket and we had some hacky perl script to try and do rss feeds with lists. Does HK support rss? Is this a feature we can ask for upstream?
It does not support it yet, but that's something that can be done upstream indeed.
https://fedorahosted.org/fedora-infrastructure/ticket/4970
"where is list administration interface"
- perl-devel is not notifying on held messages. I am also seeing this with the 'websites' list. Should we move this to an upstream bug?
Yeah, it's apparently an upstream bug, I just reported it.
https://fedorahosted.org/fedora-infrastructure/ticket/4971
"Mailman 3 mbox archives have invalid Date: value"
- This says a fix was commited. Is it live now? If so, we can close?
Yeah it's live, I'll close it.
https://fedorahosted.org/fedora-infrastructure/ticket/4975 "New subscription notifications from fedora-join mailing list come from root@localhost.fedoraproject.org"
- admin emails from root@localhost. ;( There's a dupe of this I just noticed below. ;) Closing the dupe, but should we file an upstream bug? or is this a config problem locally?
This is very strange, it does not look like a software bug, but the configuration seems correct too. Those emails should be sent with the *-owner@list.fp.o address. I'll investigate.
https://fedorahosted.org/fedora-infrastructure/ticket/5341
"main translation page isntt available when searching on https://lists.fedoraproject.org/archives/ "
- Not sure what to do here. Perhaps we could ask Mo or Ryan to weigh in and see if there's some better way to do the UI for it? Or just close it with a cannot fix for now?
Yeah, I don't know either. If Mo or Ryan have a quick idea, I'll implement it.
https://fedorahosted.org/fedora-infrastructure/ticket/5345
"firewalld-devel list archives not working properly"
- Not sure whats going on here. Any news on your investigations?
Unfortunately this happened before I could add more debug in the logs. The message may have been lost when I reloaded the HyperKitty config, I've been working on a system that would protect against this kind of concurrency. But I can't be sure that it's the root cause. I'll deploy the new version and see if it happens again. Since I'm keeping an original version of each message sent through the list, I could write a script that goes through all those emails and see if they are in HyperKitty's DB (and insert them if they aren't.
https://fedorahosted.org/fedora-infrastructure/ticket/5070
"Request: run MetricsGrimoire mlstats against all mailing lists monthly; provide access to database"
- Not sure where we are with this one. Does Mattdm have access to do the stats? Should we set them up to run automagically?
I sort of remember Matt saying that he found something different from MetricsGrimoire and that he didn't need the mboxes in the end. Matt?
https://fedorahosted.org/fedora-infrastructure/ticket/2079
"querying the list of all the aliases that a given FAS id is on"
- Not sure how to handle this one either. It's an old one and would require querying lots of places. It's not really mailing list related fully either.
Indeed.
Thanks for going through all those tickets!
A.
https://fedorahosted.org/fedora-infrastructure/ticket/4975 "New subscription notifications from fedora-join mailing list come from root@localhost.fedoraproject.org"
- admin emails from root@localhost. ;( There's a dupe of this I just noticed below. ;) Closing the dupe, but should we file an upstream bug? or is this a config problem locally?
This is very strange, it does not look like a software bug, but the configuration seems correct too. Those emails should be sent with the *-owner@list.fp.o address. I'll investigate.
Regardless of whethehr it should ideally come from $list-owner, root@localhost is pretty wrong and confusing in my opinion. In the meantime, I updated the site_owner config to admin@fedoraproject.org so it is at least a valid email addres, as it looks like this came from there.
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=d3863e1...
infrastructure@lists.fedoraproject.org