Hi,

so a few weeks ago my boss (Jason Brooks) asked me if I could help Fedora Magazine to be moved from the current hosting (a cloud instance on fedora cloud) to a WPengine account that is sponsored by RH, and managed by my team. He did the setup of the blog, and was discussing with Paul, but delegated to me the infra part and part of the execution.

So the plan was simple: - we set a instance, configure it and we do transfer data (using export and import) and see how it go - if that go well, we sync again, do what magic is needed fedora infra side and that's it

Obviously, a plan that simple couldn't go without issues, and indeed, went with some issues. I will skip over the Fedora infra ones we faced (like me not having my token, not having access to logs or infra, or the blogs on both side, etc), cause they got fixed fast enough, and did just delay the migration due to bad timing.

On the Wordpress side, we faced a few issues, some that could have been avoidable, some not. In retrospect, I kinda think we should have done the migration another way.

Issue 1: The curious case of duplicate buttons ==============================================

The first issue I did notice is that after the sync, the theme did start to have duplicate buttons for social media (the round one on top right, for g+, rss, facebook, etc). The fix was to go on https://fedoramagazine.org/wp-admin/themes.php and remove the duplicate. 5 buttons, not a big problem (unlike another blog I did migrate, where the whole menu structure of 30 items got duplicated and I had to remove them 1 by 1 using the not so fast web interface).

I guess that the solution would have been to test on a staging site first, and do the import on a empty site for prod.

Issue 2: Raiders of the lost sessions =====================================

Then another issue that Paul did bring was that people lost their session (so folks lost the document they had open). While the migration did happen on friday night (I was planning sooner), I didn't really anticipate that people would keep document in their browser open (while that's a valid workflow). In retrospect, I should have sent a email, but I got lost in details (and Jenkins fighting), so that's my fault for not communicating more.

Paul did poke the db, not sure if he managed to recover things.

Solution: communicate more with people, especially for a successful website as fedora magazine.

Issue 3: The hunt for red comments ==================================

Then yesterday, Paul also told me that at least one comment did disappear. Upon looking, we found that the comment was in the database and the admin UI, but didn't appear on the website. And the comment was counted on the top counter (11 comments), but not the one that bottom of the article (10 comments). For folks wanting to see, it was Colin comment on https://fedoramagazine.org/introducing-fedora-coreos/. I started to dig, and it seems that the comment was a answer to another comment, but for some reason, it was answering to a comment on another post. After trying to understand what happened, it seems that while the export xml is self consistent but ids of post/comments got changed with the 2nd export, but just for some comments. How the code did cause that problem is still a mystery. In the end, I just went in the DB and fixed the ID.

So if you see a page with the same problem, please tell me and I will take a look. If a comment did disappear, please tell too.

Solution: restarting the sync from scratch would have likely mitigated the problem.

Issue 4: With a Friend Like misc ================================

Then after migrating commblog, I did notice that a few posts (around 10) had their author field changed to be my account. I couldn't find a common denominator for that, so I did fixed them one by one. It was faster once I found "quick edit" in the UI, which, as the name indicate, is quicker.

I was then able to eliminate all the problematic posts.

So I started to look at fedora magazine, and it seems that there is a issue https://fedoramagazine.org/author/misc/ cause I got one post:

https://fedoramagazine.org/try-tilix-new-terminal-emulator-fedora-2/

But that's a copy of: https://fedoramagazine.org/try-tilix-new-terminal-emulator-fedora/

I guess I will have to clean that. Interestingly, WP UI say there is 14 posts for me. Not sure what happen to the 13 others, since it only show 1.

Restarting the sync from scratch would have likely prevented the issue.

Issue 5: They link ==================

Ryan Lerch did found today that among the post, there is some who are different, and that at least 1 post had wrong links:

https://lists.fedorahosted.org/archives/list/magazine@lists.fedoraproject.or...

Upon looking at the export file, the link was correct, but it seems that WP is fixing things on import, or something like that. I need to see if that's in the DB or a setting somewhere else.

I suspect the domain name was different at the time of the migration, in the wordpress configuration, so it tried to make things correct. I guess we need to make sure the server is 100% correct before we import posts. or it could also be a caching issue, but we did reset the cache twice.

I will see how many post are impacted with a sql query and fix stuff.

Issue 6: Invasion of the post snatchers =======================================

On irc, Gregbartholomew did signal that one on the article they wrote got duplicated, the same for Ryan. I guess that was a side effect of the 2nd sync, especially if the name of the post got changed (so it got created as a new post). Given that the windows was quite small (2 days), people can just get ride of the postpeop^W posts.

Again, likely a side effect of not cleaning the db before syncing.

Missing features ================

1) revisions ------------ As Paul told, the revision are disabled. I will see with Jason to enable them. The reasoning of wpengine is on

https://wpengine.com/support/i-noticed-revisions-are-disabled-why-also-how-d...

2) direct db access -------------------

As a admin on the main wpengine account, I do have phpmyadmin access for the various instances. But I do not have enough access to the main account to give others people account or access. Again, this will wait for Jason.

3) warning about "doing a backup" ---------------------------------

If a admin try to upgrade plugins, the UX will ask them if they want to do a backup, or say that they did. Again, that require elevated access that we can't give for now. And there is automated backup, so in doubt people can say "yes I did". And if things go wrong, we can try to help. But we can't disable the warning.

Conclusion ==========

I think while that went well, a lot of issues could have been avoided by using a separate test install, then ask to people to check, and then do the migration in prod.

Unfortunately, the timing wasn't great, between some unexpected interruptions, some roadblocks and summertime (so people taking PTO).

There was also mix-up regarding communication, as I thought that was taken care of, but I didn't verify, so my fault for not telling people in advance.

And while I speak of "restarting from scratch", I am not sure exactly how to do that in practice with a hosted solution like wpengine (if it was self hosted, the answer would have been "ansible", which is a answer to a lot of stuff). I can surely have erased all posts, all comments, all users, but maybe I am not removing everything, or removing too much. Removing the whole site would have been another solution, but then, I had no idea on how it was setup in the first place, not what config was done (especially for openid integration, or how to import a theme).

Anyway, if people see any more issues, please tell us on the list, we will take a look. (no need to name the issues, just tell what is wrong)