Hey there,
Quick update on the Mailman3 migration. I have now migrated all lists to the new mailman01 server (including fedorahosted lists). I've also migrated the SpamAssassin configuration and database and it seems to be working fine.
I had to make a small tweak to the postfix configuration though, and even if I don't think it opens a vulnerability in any way I'm pointing you to it: I've opened the submission port (587/tcp) on the loopback address only, with a bypass of the SpamAssassin check. The reason for that is that Mailman 3 now sends emails to Postfix as an SMTP client, and with the previous configuration all outgoing emails from Mailman were scanned, which made no sense and caused a huge load on the server. I had two options: 1. Set Postfix to listen on localhost:smtp without the spam checking and on its external IP with the spam checking, but this means hardcoding the server's external IP in Postfix' master.cfg file 2. Add another port for Postfix's SMTP daemon witout the SpamAssassin content filter, but keep it on localhost only for security
I chose the latter because I felt that harcoding the IP was a worse solution, but since we're using Ansible it may not matter. If you prefer the first solution I can totally make the change.
My next step is to download the old HTML archives and make them available on mailman01 through Apache to preserve existing URLs in the wild.
Then I think we'll finally be able to take the former servers out of the loop.
Aurélien
On Fri, 13 May 2016 12:41:36 +0200 Aurelien Bompard abompard@fedoraproject.org wrote:
Hey there,
Quick update on the Mailman3 migration. I have now migrated all lists to the new mailman01 server (including fedorahosted lists). I've also migrated the SpamAssassin configuration and database and it seems to be working fine.
Great. ;)
I had to make a small tweak to the postfix configuration though, and even if I don't think it opens a vulnerability in any way I'm pointing you to it: I've opened the submission port (587/tcp) on the loopback address only, with a bypass of the SpamAssassin check. The reason for that is that Mailman 3 now sends emails to Postfix as an SMTP client, and with the previous configuration all outgoing emails from Mailman were scanned, which made no sense and caused a huge load on the server. I had two options: 1. Set Postfix to listen on localhost:smtp without the spam checking and on its external IP with the spam checking, but this means hardcoding the server's external IP in Postfix' master.cfg file 2. Add another port for Postfix's SMTP daemon witout the SpamAssassin content filter, but keep it on localhost only for security
I chose the latter because I felt that harcoding the IP was a worse solution, but since we're using Ansible it may not matter. If you prefer the first solution I can totally make the change.
I think thats fine. For some reason this change is pending on all machines tho, need to sort out why it's not mailman/smtp-mm only.
My next step is to download the old HTML archives and make them available on mailman01 through Apache to preserve existing URLs in the wild.
Do we have enough space for that?
Then I think we'll finally be able to take the former servers out of the loop.
Excellent!
kevin
I think thats fine. For some reason this change is pending on all machines tho, need to sort out why it's not mailman/smtp-mm only.
Hmm, that's strange, I only changed 'roles/base/files/postfix/ master.cf/master.cf.mailman', so it should only affect hosts in the 'mailman' group. I also changed 'roles/base/tasks/postfix.yml' because only hosts starting with smtp-mm would get their master.cf from ansible before.
My next step is to download the old HTML archives and make them
available on mailman01 through Apache to preserve existing URLs in the wild.
Do we have enough space for that?
It's 106Gb, and after copy there's still 115Gb left. It'll work for now, but the disk space required on this machine is only going to grow (because of the fulltext index mainly, but also because emails are also stored unaltered in maildirs for safekeeping). What do you think we should do? Move the HTML archives elsewhere? Just increase the disk space when needed?
Aurélien
On Sat, 14 May 2016 12:34:42 +0200 Aurelien Bompard abompard@fedoraproject.org wrote:
I think thats fine. For some reason this change is pending on all machines tho, need to sort out why it's not mailman/smtp-mm only.
Hmm, that's strange, I only changed 'roles/base/files/postfix/ master.cf/master.cf.mailman', so it should only affect hosts in the 'mailman' group. I also changed 'roles/base/tasks/postfix.yml' because only hosts starting with smtp-mm would get their master.cf from ansible before.
Yeah, it looks right, but something isn't working right.
I'll dig into it in a few here...
My next step is to download the old HTML archives and make them
available on mailman01 through Apache to preserve existing URLs in the wild.
Do we have enough space for that?
It's 106Gb, and after copy there's still 115Gb left. It'll work for now, but the disk space required on this machine is only going to grow (because of the fulltext index mainly, but also because emails are also stored unaltered in maildirs for safekeeping). What do you think we should do? Move the HTML archives elsewhere? Just increase the disk space when needed?
I'm fine with just adding space for now as needed.
kevin
My next step is to download the old HTML archives and make them
available on mailman01 through Apache to preserve existing URLs in the wild.
OK, I've migrated those files, setup the proper Apache directives, and tested it.
As far as I know, we can remove the old servers now.
Aurélien
On Wed, 18 May 2016 12:17:49 +0200 Aurelien Bompard abompard@fedoraproject.org wrote:
My next step is to download the old HTML archives and make them
available on mailman01 through Apache to preserve existing URLs in the wild.
OK, I've migrated those files, setup the proper Apache directives, and tested it.
As far as I know, we can remove the old servers now.
I wish it had been that easy. ;)
I spent most of the day working on this and it's finally all done I think.
I had to setup our proxies to handle the web requests and pass them to mailman01. For both lists.fedorahosted.org and lists.fedoraproject.org. Had to sort our certs. Had to get varnish all happy. Switched transport maps. Fixed a smtp-mm that wasn't in rotation for a long time.
In any case. I think all email and web requests are going to mailman01 now and collab03 / hosted-lists01 can be turned off. I'll check back on them tomorrow or friday to confirm. ;)
The end of this journey might be worth a nice blog post... especially pointing out that even though we have done migrating we should look forward to more and more improvements.
kevin
On Wed, May 18, 2016 at 04:12:41PM -0600, Kevin Fenzi wrote:
The end of this journey might be worth a nice blog post... especially pointing out that even though we have done migrating we should look forward to more and more improvements.
I would love a step-by-step "how to convert mailman2 to mailman3" post for those using the RPM packages. Existing guides don't assume you are using the packages.
infrastructure@lists.fedoraproject.org