Our MirrorManager setup exports the current state of all mirrors every hour at :30 to a protobuf based file which is then used by the mirrorlist servers to answer the requests from yum and dnf.
The Python script requires up to 10GB of memory and takes between 35 and 50 minutes. The script does a lot of SQL queries and also some really big SQL queries joining up to 6 large MirrorManager tables.
I have rewritten this Python script in Rust and now it only needs around 1 minute instead of 35 to 50 minutes and only 600MB instead of 10GB.
I think the biggest difference is that I am almost not doing any joins in my SQL request. I download all the tables once and then I do a lot of loops over the downloaded tables and this seems to be massively faster.
As the mirrorlist-server in Rust has proven to be extremely stable over the last months we have been using it I would also like to replace the mirrorlist protbuf input generation with my new Rust based code.
I am planing to try out the new protobuf file in staging in the next days and would then try to get my new protobuf generation program into Fedora. Once it is packaged I would discuss here how and if we want to deploy in Fedora's infrastructure.
Having the possibility to generate the mirrorlist input data in about a minute would significantly reduce the load on the database server and enable us to react much faster if broken protobuf data has been synced to the mirrorlist servers on the proxies.
Adrian
On Mon, Jun 01, 2020 at 04:18:35PM +0200, Adrian Reber wrote:
Our MirrorManager setup exports the current state of all mirrors every hour at :30 to a protobuf based file which is then used by the mirrorlist servers to answer the requests from yum and dnf.
The Python script requires up to 10GB of memory and takes between 35 and 50 minutes. The script does a lot of SQL queries and also some really big SQL queries joining up to 6 large MirrorManager tables.
I have rewritten this Python script in Rust and now it only needs around 1 minute instead of 35 to 50 minutes and only 600MB instead of 10GB.
Wow. nice!
I think the biggest difference is that I am almost not doing any joins in my SQL request. I download all the tables once and then I do a lot of loops over the downloaded tables and this seems to be massively faster.
As the mirrorlist-server in Rust has proven to be extremely stable over the last months we have been using it I would also like to replace the mirrorlist protbuf input generation with my new Rust based code.
I am planing to try out the new protobuf file in staging in the next days and would then try to get my new protobuf generation program into Fedora. Once it is packaged I would discuss here how and if we want to deploy in Fedora's infrastructure.
Cool. You will need to hurry as staging goes off on monday, and back in a few weeks. :)
Having the possibility to generate the mirrorlist input data in about a minute would significantly reduce the load on the database server and enable us to react much faster if broken protobuf data has been synced to the mirrorlist servers on the proxies.
Yeah, and I wonder if it would let us revisit the entire sequence from 'update push finished' to updated mirrorlist server.
Great stuff! Thanks for working on it!
kevin
On Thu, Jun 04, 2020 at 07:02:41PM -0700, Kevin Fenzi wrote:
On Mon, Jun 01, 2020 at 04:18:35PM +0200, Adrian Reber wrote:
Our MirrorManager setup exports the current state of all mirrors every hour at :30 to a protobuf based file which is then used by the mirrorlist servers to answer the requests from yum and dnf.
The Python script requires up to 10GB of memory and takes between 35 and 50 minutes. The script does a lot of SQL queries and also some really big SQL queries joining up to 6 large MirrorManager tables.
I have rewritten this Python script in Rust and now it only needs around 1 minute instead of 35 to 50 minutes and only 600MB instead of 10GB.
Wow. nice!
I think the biggest difference is that I am almost not doing any joins in my SQL request. I download all the tables once and then I do a lot of loops over the downloaded tables and this seems to be massively faster.
As the mirrorlist-server in Rust has proven to be extremely stable over the last months we have been using it I would also like to replace the mirrorlist protbuf input generation with my new Rust based code.
I am planing to try out the new protobuf file in staging in the next days and would then try to get my new protobuf generation program into Fedora. Once it is packaged I would discuss here how and if we want to deploy in Fedora's infrastructure.
Cool. You will need to hurry as staging goes off on monday, and back in a few weeks. :)
Then I just have to wait a bit. No problem.
Having the possibility to generate the mirrorlist input data in about a minute would significantly reduce the load on the database server and enable us to react much faster if broken protobuf data has been synced to the mirrorlist servers on the proxies.
Yeah, and I wonder if it would let us revisit the entire sequence from 'update push finished' to updated mirrorlist server.
Probably. As the new code will not run on the current RHEL 7 based mm-backend01 would it make sense to run a short running service like this on Fedora's OpenShift? We could also create a new read-only (SELECT only) database account for this.
Adrian
On Wed, 10 Jun 2020 at 05:10, Adrian Reber adrian@lisas.de wrote:
On Thu, Jun 04, 2020 at 07:02:41PM -0700, Kevin Fenzi wrote:
On Mon, Jun 01, 2020 at 04:18:35PM +0200, Adrian Reber wrote:
Our MirrorManager setup exports the current state of all mirrors every hour at :30 to a protobuf based file which is then used by the mirrorlist servers to answer the requests from yum and dnf.
The Python script requires up to 10GB of memory and takes between 35
and
50 minutes. The script does a lot of SQL queries and also some really big SQL queries joining up to 6 large MirrorManager tables.
I have rewritten this Python script in Rust and now it only needs
around
1 minute instead of 35 to 50 minutes and only 600MB instead of 10GB.
Wow. nice!
I think the biggest difference is that I am almost not doing any joins in my SQL request. I download all the tables once and then I do a lot
of
loops over the downloaded tables and this seems to be massively faster.
As the mirrorlist-server in Rust has proven to be extremely stable over the last months we have been using it I would also like to replace the mirrorlist protbuf input generation with my new Rust based code.
I am planing to try out the new protobuf file in staging in the next days and would then try to get my new protobuf generation program into Fedora. Once it is packaged I would discuss here how and if we want to deploy in Fedora's infrastructure.
Cool. You will need to hurry as staging goes off on monday, and back in a few weeks. :)
Then I just have to wait a bit. No problem.
Having the possibility to generate the mirrorlist input data in about a minute would significantly reduce the load on the database server and enable us to react much faster if broken protobuf data has been synced to the mirrorlist servers on the proxies.
Yeah, and I wonder if it would let us revisit the entire sequence from 'update push finished' to updated mirrorlist server.
Probably. As the new code will not run on the current RHEL 7 based mm-backend01 would it make sense to run a short running service like this on Fedora's OpenShift? We could also create a new read-only (SELECT only) database account for this.
We can do that or we can make a mm-backend02 which is el8 for this
On Wed, Jun 10, 2020 at 11:09:49AM +0200, Adrian Reber wrote:
Then I just have to wait a bit. No problem.
Having the possibility to generate the mirrorlist input data in about a minute would significantly reduce the load on the database server and enable us to react much faster if broken protobuf data has been synced to the mirrorlist servers on the proxies.
Yeah, and I wonder if it would let us revisit the entire sequence from 'update push finished' to updated mirrorlist server.
This would help us with the case of: - updates push/rawhide finishes, master mirror is updated. - openqa/other internal thing tries to get images or updates in that change and gets a metalink with the old checksum so it can't get the new stuff. - mm-backend01 generates and pushes out a new protobuf.
Probably. As the new code will not run on the current RHEL 7 based mm-backend01 would it make sense to run a short running service like this on Fedora's OpenShift? We could also create a new read-only (SELECT only) database account for this.
We could, or as smooge suggests make a mm-backend02?
But I guess now mm-backend02 just generates new proobuf files and copies them to mirrorlists? If thats all it's doing, perhaps we could indeed replace it with an openshift project.
kevin
On Mon, Jun 15, 2020 at 03:36:23PM -0700, Kevin Fenzi wrote:
On Wed, Jun 10, 2020 at 11:09:49AM +0200, Adrian Reber wrote:
Then I just have to wait a bit. No problem.
Having the possibility to generate the mirrorlist input data in about a minute would significantly reduce the load on the database server and enable us to react much faster if broken protobuf data has been synced to the mirrorlist servers on the proxies.
Yeah, and I wonder if it would let us revisit the entire sequence from 'update push finished' to updated mirrorlist server.
This would help us with the case of:
- updates push/rawhide finishes, master mirror is updated.
- openqa/other internal thing tries to get images or updates in that change and gets a metalink with the old checksum so it can't get the new stuff.
- mm-backend01 generates and pushes out a new protobuf.
Probably. As the new code will not run on the current RHEL 7 based mm-backend01 would it make sense to run a short running service like this on Fedora's OpenShift? We could also create a new read-only (SELECT only) database account for this.
We could, or as smooge suggests make a mm-backend02?
But I guess now mm-backend02 just generates new proobuf files and copies them to mirrorlists? If thats all it's doing, perhaps we could indeed replace it with an openshift project.
We need a system to run the tool and copy the data to all proxies.
I would like to see a new MirrorManager database user who can only do selects as that is all we need.
Currently we copy the files via SSH to the proxies, if we continue doing it that way, then we would also need the existing SSH key to copy the data to the proxies.
Easiest would probably be a small Fedora 32 based VM with 2GB of memory.
Adrian
On Tue, Jun 30, 2020 at 02:17:30PM +0200, Adrian Reber wrote:
On Mon, Jun 15, 2020 at 03:36:23PM -0700, Kevin Fenzi wrote:
On Wed, Jun 10, 2020 at 11:09:49AM +0200, Adrian Reber wrote:
Then I just have to wait a bit. No problem.
Having the possibility to generate the mirrorlist input data in about a minute would significantly reduce the load on the database server and enable us to react much faster if broken protobuf data has been synced to the mirrorlist servers on the proxies.
Yeah, and I wonder if it would let us revisit the entire sequence from 'update push finished' to updated mirrorlist server.
This would help us with the case of:
- updates push/rawhide finishes, master mirror is updated.
- openqa/other internal thing tries to get images or updates in that change and gets a metalink with the old checksum so it can't get the new stuff.
- mm-backend01 generates and pushes out a new protobuf.
Probably. As the new code will not run on the current RHEL 7 based mm-backend01 would it make sense to run a short running service like this on Fedora's OpenShift? We could also create a new read-only (SELECT only) database account for this.
We could, or as smooge suggests make a mm-backend02?
But I guess now mm-backend02 just generates new proobuf files and copies them to mirrorlists? If thats all it's doing, perhaps we could indeed replace it with an openshift project.
We need a system to run the tool and copy the data to all proxies.
I would like to see a new MirrorManager database user who can only do selects as that is all we need.
Currently we copy the files via SSH to the proxies, if we continue doing it that way, then we would also need the existing SSH key to copy the data to the proxies.
Easiest would probably be a small Fedora 32 based VM with 2GB of memory.
I'm not sure f32 will work with 2gb memory anymore. I dont think it installs at any rate.
I do like the idea of just making it an openshift pod. Perhaps this could even fit with pingous 'toddlers' setup. ie:
* listen for message saying a repo has updated * update the db * create the protobuf * push out to proxies
The only weird part of putting it in openshift is that we would need to have fedora_ftp (ro) there available as a volume, but that is doable...
kevin
On Wed, Jul 01, 2020 at 09:16:05AM -0700, Kevin Fenzi wrote:
On Tue, Jun 30, 2020 at 02:17:30PM +0200, Adrian Reber wrote:
On Mon, Jun 15, 2020 at 03:36:23PM -0700, Kevin Fenzi wrote:
On Wed, Jun 10, 2020 at 11:09:49AM +0200, Adrian Reber wrote:
Then I just have to wait a bit. No problem.
Having the possibility to generate the mirrorlist input data in about a minute would significantly reduce the load on the database server and enable us to react much faster if broken protobuf data has been synced to the mirrorlist servers on the proxies.
Yeah, and I wonder if it would let us revisit the entire sequence from 'update push finished' to updated mirrorlist server.
This would help us with the case of:
- updates push/rawhide finishes, master mirror is updated.
- openqa/other internal thing tries to get images or updates in that change and gets a metalink with the old checksum so it can't get the new stuff.
- mm-backend01 generates and pushes out a new protobuf.
Probably. As the new code will not run on the current RHEL 7 based mm-backend01 would it make sense to run a short running service like this on Fedora's OpenShift? We could also create a new read-only (SELECT only) database account for this.
We could, or as smooge suggests make a mm-backend02?
But I guess now mm-backend02 just generates new proobuf files and copies them to mirrorlists? If thats all it's doing, perhaps we could indeed replace it with an openshift project.
We need a system to run the tool and copy the data to all proxies.
I would like to see a new MirrorManager database user who can only do selects as that is all we need.
Currently we copy the files via SSH to the proxies, if we continue doing it that way, then we would also need the existing SSH key to copy the data to the proxies.
Easiest would probably be a small Fedora 32 based VM with 2GB of memory.
I'm not sure f32 will work with 2gb memory anymore. I dont think it installs at any rate.
I do like the idea of just making it an openshift pod. Perhaps this could even fit with pingous 'toddlers' setup. ie:
- listen for message saying a repo has updated
- update the db
- create the protobuf
- push out to proxies
The only weird part of putting it in openshift is that we would need to have fedora_ftp (ro) there available as a volume, but that is doable...
No, this part only needs to talk read-only to the database. This is not touching anything on the disk besides writing the output. I guess you were thinking about the umdl (update-master-directory-listing) part. That would need ro access to the file-system.
The part I am talking about just reads the database and creates a protobuf snapshot of the database which is then used by the mirrorlist servers on the proxies.
Currently it runs once every hour. Which works pretty good so far. Triggering it on a message makes only limited sense as it depends on the results of the crawler. We could run it twice an hour to have newer database snapshots on the proxies.
How can I prepare it for running in openshift. Can I use the configuration for toddlers? Where can I find that?
Adrian
On Wed, Jul 01, 2020 at 06:25:47PM +0200, Adrian Reber wrote:
On Wed, Jul 01, 2020 at 09:16:05AM -0700, Kevin Fenzi wrote:
On Tue, Jun 30, 2020 at 02:17:30PM +0200, Adrian Reber wrote:
On Mon, Jun 15, 2020 at 03:36:23PM -0700, Kevin Fenzi wrote:
On Wed, Jun 10, 2020 at 11:09:49AM +0200, Adrian Reber wrote:
Then I just have to wait a bit. No problem.
> Having the possibility to generate the mirrorlist input data in about a > minute would significantly reduce the load on the database server and > enable us to react much faster if broken protobuf data has been synced > to the mirrorlist servers on the proxies.
Yeah, and I wonder if it would let us revisit the entire sequence from 'update push finished' to updated mirrorlist server.
This would help us with the case of:
- updates push/rawhide finishes, master mirror is updated.
- openqa/other internal thing tries to get images or updates in that change and gets a metalink with the old checksum so it can't get the new stuff.
- mm-backend01 generates and pushes out a new protobuf.
Probably. As the new code will not run on the current RHEL 7 based mm-backend01 would it make sense to run a short running service like this on Fedora's OpenShift? We could also create a new read-only (SELECT only) database account for this.
We could, or as smooge suggests make a mm-backend02?
But I guess now mm-backend02 just generates new proobuf files and copies them to mirrorlists? If thats all it's doing, perhaps we could indeed replace it with an openshift project.
We need a system to run the tool and copy the data to all proxies.
I would like to see a new MirrorManager database user who can only do selects as that is all we need.
Currently we copy the files via SSH to the proxies, if we continue doing it that way, then we would also need the existing SSH key to copy the data to the proxies.
Easiest would probably be a small Fedora 32 based VM with 2GB of memory.
I'm not sure f32 will work with 2gb memory anymore. I dont think it installs at any rate.
I do like the idea of just making it an openshift pod. Perhaps this could even fit with pingous 'toddlers' setup. ie:
- listen for message saying a repo has updated
- update the db
- create the protobuf
- push out to proxies
The only weird part of putting it in openshift is that we would need to have fedora_ftp (ro) there available as a volume, but that is doable...
No, this part only needs to talk read-only to the database. This is not touching anything on the disk besides writing the output. I guess you were thinking about the umdl (update-master-directory-listing) part. That would need ro access to the file-system.
The part I am talking about just reads the database and creates a protobuf snapshot of the database which is then used by the mirrorlist servers on the proxies.
Currently it runs once every hour. Which works pretty good so far. Triggering it on a message makes only limited sense as it depends on the results of the crawler. We could run it twice an hour to have newer database snapshots on the proxies.
How can I prepare it for running in openshift. Can I use the configuration for toddlers? Where can I find that?
Toddlers has recently gained the possibility to trigger at certain time. Well we're actually sort of abusing it, we've added a script (playtime) that will send a fedora-messaging message and that should be called from a cron job. That message should then trigger the corresponding toddler.
This (playtime) is not yet deployed in production so I can't point you to an example, but the project itself is at: https://pagure.io/fedora-infra/toddlers and the openshift config is at: https://pagure.io/fedora-infra/ansible/blob/master/f/playbooks/openshift-app...
Pierre
On Thu, Jul 02, 2020 at 09:44:10AM +0200, Pierre-Yves Chibon wrote:
On Wed, Jul 01, 2020 at 06:25:47PM +0200, Adrian Reber wrote:
On Wed, Jul 01, 2020 at 09:16:05AM -0700, Kevin Fenzi wrote:
On Tue, Jun 30, 2020 at 02:17:30PM +0200, Adrian Reber wrote:
On Mon, Jun 15, 2020 at 03:36:23PM -0700, Kevin Fenzi wrote:
On Wed, Jun 10, 2020 at 11:09:49AM +0200, Adrian Reber wrote:
Then I just have to wait a bit. No problem.
> > Having the possibility to generate the mirrorlist input data in about a > > minute would significantly reduce the load on the database server and > > enable us to react much faster if broken protobuf data has been synced > > to the mirrorlist servers on the proxies. > > Yeah, and I wonder if it would let us revisit the entire sequence from > 'update push finished' to updated mirrorlist server.
This would help us with the case of:
- updates push/rawhide finishes, master mirror is updated.
- openqa/other internal thing tries to get images or updates in that change and gets a metalink with the old checksum so it can't get the new stuff.
- mm-backend01 generates and pushes out a new protobuf.
Probably. As the new code will not run on the current RHEL 7 based mm-backend01 would it make sense to run a short running service like this on Fedora's OpenShift? We could also create a new read-only (SELECT only) database account for this.
We could, or as smooge suggests make a mm-backend02?
But I guess now mm-backend02 just generates new proobuf files and copies them to mirrorlists? If thats all it's doing, perhaps we could indeed replace it with an openshift project.
We need a system to run the tool and copy the data to all proxies.
I would like to see a new MirrorManager database user who can only do selects as that is all we need.
Currently we copy the files via SSH to the proxies, if we continue doing it that way, then we would also need the existing SSH key to copy the data to the proxies.
Easiest would probably be a small Fedora 32 based VM with 2GB of memory.
I'm not sure f32 will work with 2gb memory anymore. I dont think it installs at any rate.
I do like the idea of just making it an openshift pod. Perhaps this could even fit with pingous 'toddlers' setup. ie:
- listen for message saying a repo has updated
- update the db
- create the protobuf
- push out to proxies
The only weird part of putting it in openshift is that we would need to have fedora_ftp (ro) there available as a volume, but that is doable...
No, this part only needs to talk read-only to the database. This is not touching anything on the disk besides writing the output. I guess you were thinking about the umdl (update-master-directory-listing) part. That would need ro access to the file-system.
The part I am talking about just reads the database and creates a protobuf snapshot of the database which is then used by the mirrorlist servers on the proxies.
Currently it runs once every hour. Which works pretty good so far. Triggering it on a message makes only limited sense as it depends on the results of the crawler. We could run it twice an hour to have newer database snapshots on the proxies.
How can I prepare it for running in openshift. Can I use the configuration for toddlers? Where can I find that?
Toddlers has recently gained the possibility to trigger at certain time. Well we're actually sort of abusing it, we've added a script (playtime) that will send a fedora-messaging message and that should be called from a cron job. That message should then trigger the corresponding toddler.
This (playtime) is not yet deployed in production so I can't point you to an example, but the project itself is at: https://pagure.io/fedora-infra/toddlers and the openshift config is at: https://pagure.io/fedora-infra/ansible/blob/master/f/playbooks/openshift-app...
Thanks for the pointer. I had a look at it, but I do not think I understand it. So many sections where I have no idea what they do
I would need some help setting this up.
Adrian
On Fri, Jul 03, 2020 at 08:03:16AM +0200, Adrian Reber wrote:
On Thu, Jul 02, 2020 at 09:44:10AM +0200, Pierre-Yves Chibon wrote:
On Wed, Jul 01, 2020 at 06:25:47PM +0200, Adrian Reber wrote:
On Wed, Jul 01, 2020 at 09:16:05AM -0700, Kevin Fenzi wrote:
On Tue, Jun 30, 2020 at 02:17:30PM +0200, Adrian Reber wrote:
On Mon, Jun 15, 2020 at 03:36:23PM -0700, Kevin Fenzi wrote:
On Wed, Jun 10, 2020 at 11:09:49AM +0200, Adrian Reber wrote: > > Then I just have to wait a bit. No problem. > > > > Having the possibility to generate the mirrorlist input data in about a > > > minute would significantly reduce the load on the database server and > > > enable us to react much faster if broken protobuf data has been synced > > > to the mirrorlist servers on the proxies. > > > > Yeah, and I wonder if it would let us revisit the entire sequence from > > 'update push finished' to updated mirrorlist server.
This would help us with the case of:
- updates push/rawhide finishes, master mirror is updated.
- openqa/other internal thing tries to get images or updates in that change and gets a metalink with the old checksum so it can't get the new stuff.
- mm-backend01 generates and pushes out a new protobuf.
> > Probably. As the new code will not run on the current RHEL 7 based > mm-backend01 would it make sense to run a short running service like > this on Fedora's OpenShift? We could also create a new read-only (SELECT > only) database account for this.
We could, or as smooge suggests make a mm-backend02?
But I guess now mm-backend02 just generates new proobuf files and copies them to mirrorlists? If thats all it's doing, perhaps we could indeed replace it with an openshift project.
We need a system to run the tool and copy the data to all proxies.
I would like to see a new MirrorManager database user who can only do selects as that is all we need.
Currently we copy the files via SSH to the proxies, if we continue doing it that way, then we would also need the existing SSH key to copy the data to the proxies.
Easiest would probably be a small Fedora 32 based VM with 2GB of memory.
I'm not sure f32 will work with 2gb memory anymore. I dont think it installs at any rate.
I do like the idea of just making it an openshift pod. Perhaps this could even fit with pingous 'toddlers' setup. ie:
- listen for message saying a repo has updated
- update the db
- create the protobuf
- push out to proxies
The only weird part of putting it in openshift is that we would need to have fedora_ftp (ro) there available as a volume, but that is doable...
No, this part only needs to talk read-only to the database. This is not touching anything on the disk besides writing the output. I guess you were thinking about the umdl (update-master-directory-listing) part. That would need ro access to the file-system.
The part I am talking about just reads the database and creates a protobuf snapshot of the database which is then used by the mirrorlist servers on the proxies.
Currently it runs once every hour. Which works pretty good so far. Triggering it on a message makes only limited sense as it depends on the results of the crawler. We could run it twice an hour to have newer database snapshots on the proxies.
How can I prepare it for running in openshift. Can I use the configuration for toddlers? Where can I find that?
Toddlers has recently gained the possibility to trigger at certain time. Well we're actually sort of abusing it, we've added a script (playtime) that will send a fedora-messaging message and that should be called from a cron job. That message should then trigger the corresponding toddler.
This (playtime) is not yet deployed in production so I can't point you to an example, but the project itself is at: https://pagure.io/fedora-infra/toddlers and the openshift config is at: https://pagure.io/fedora-infra/ansible/blob/master/f/playbooks/openshift-app...
Thanks for the pointer. I had a look at it, but I do not think I understand it. So many sections where I have no idea what they do
I would need some help setting this up.
Most happy to help with this. Just poke me on IRC and block a slot in my calendar and we'll go through it :)
Pierre
On Wed, Jul 01, 2020 at 06:25:47PM +0200, Adrian Reber wrote:
No, this part only needs to talk read-only to the database. This is not touching anything on the disk besides writing the output. I guess you were thinking about the umdl (update-master-directory-listing) part. That would need ro access to the file-system.
Right. I was actually thinking of both.
ie, at some point down the road it might be a nice thing to get all of mirrormanager in openshift. Each part could be a seperate project or under one or all as various toddlers, but I think in the end it would make it more maintainable and also open up some improvements on workflow... not to say we couldn't also do this without moving everything to openshift, but I openshift has some nice advantages from the operations side anyhow.
kevin
On Wed, Jul 01, 2020 at 09:16:05AM -0700, Kevin Fenzi wrote:
On Tue, Jun 30, 2020 at 02:17:30PM +0200, Adrian Reber wrote:
On Mon, Jun 15, 2020 at 03:36:23PM -0700, Kevin Fenzi wrote:
On Wed, Jun 10, 2020 at 11:09:49AM +0200, Adrian Reber wrote:
Then I just have to wait a bit. No problem.
Having the possibility to generate the mirrorlist input data in about a minute would significantly reduce the load on the database server and enable us to react much faster if broken protobuf data has been synced to the mirrorlist servers on the proxies.
Yeah, and I wonder if it would let us revisit the entire sequence from 'update push finished' to updated mirrorlist server.
This would help us with the case of:
- updates push/rawhide finishes, master mirror is updated.
- openqa/other internal thing tries to get images or updates in that change and gets a metalink with the old checksum so it can't get the new stuff.
- mm-backend01 generates and pushes out a new protobuf.
Probably. As the new code will not run on the current RHEL 7 based mm-backend01 would it make sense to run a short running service like this on Fedora's OpenShift? We could also create a new read-only (SELECT only) database account for this.
We could, or as smooge suggests make a mm-backend02?
But I guess now mm-backend02 just generates new proobuf files and copies them to mirrorlists? If thats all it's doing, perhaps we could indeed replace it with an openshift project.
We need a system to run the tool and copy the data to all proxies.
I would like to see a new MirrorManager database user who can only do selects as that is all we need.
Currently we copy the files via SSH to the proxies, if we continue doing it that way, then we would also need the existing SSH key to copy the data to the proxies.
Easiest would probably be a small Fedora 32 based VM with 2GB of memory.
I'm not sure f32 will work with 2gb memory anymore. I dont think it installs at any rate.
I do like the idea of just making it an openshift pod. Perhaps this could even fit with pingous 'toddlers' setup. ie:
I tried to create a toddler, but that setup is too complicated for me. Especially if something is not working it will be almost impossible for me to debug it if it is running somewhere I cannot reach via SSH.
I just tried to build the generate-mirrorlist-cache on RHEL 7 (using Rust from EPEL) and it works fine. Instead of 20 minutes it needs 30 seconds to generate the mirrorlist cache file on mm-backend01.
Although a RPM is available in Fedora I am not sure the RPM can be made available in EPEL 7.
RPM Fusion is using the Rust based generate-mirrorlist-cache for some months already and I do not see any problems with it.
I would like to use this also for Fedora and a dedicated Fedora based VM or building from source on RHEL 7 is both possible. Building from source does not sound like something used in Fedora Infrastructure so setting up a Fedora based VM would be necessary. Would that be possible?
Adrian
On Mon, 5 Oct 2020 at 02:24, Adrian Reber adrian@lisas.de wrote:
On Wed, Jul 01, 2020 at 09:16:05AM -0700, Kevin Fenzi wrote:
On Tue, Jun 30, 2020 at 02:17:30PM +0200, Adrian Reber wrote:
On Mon, Jun 15, 2020 at 03:36:23PM -0700, Kevin Fenzi wrote:
On Wed, Jun 10, 2020 at 11:09:49AM +0200, Adrian Reber wrote:
Then I just have to wait a bit. No problem.
> Having the possibility to generate the mirrorlist input data
in about a
> minute would significantly reduce the load on the database
server and
> enable us to react much faster if broken protobuf data has
been synced
> to the mirrorlist servers on the proxies.
Yeah, and I wonder if it would let us revisit the entire
sequence from
'update push finished' to updated mirrorlist server.
This would help us with the case of:
- updates push/rawhide finishes, master mirror is updated.
- openqa/other internal thing tries to get images or updates in that change and gets a metalink with the old checksum so it can't get
the
new stuff.
- mm-backend01 generates and pushes out a new protobuf.
Probably. As the new code will not run on the current RHEL 7 based mm-backend01 would it make sense to run a short running service
like
this on Fedora's OpenShift? We could also create a new read-only
(SELECT
only) database account for this.
We could, or as smooge suggests make a mm-backend02?
But I guess now mm-backend02 just generates new proobuf files and
copies
them to mirrorlists? If thats all it's doing, perhaps we could indeed replace it with an openshift project.
We need a system to run the tool and copy the data to all proxies.
I would like to see a new MirrorManager database user who can only do selects as that is all we need.
Currently we copy the files via SSH to the proxies, if we continue
doing
it that way, then we would also need the existing SSH key to copy the data to the proxies.
Easiest would probably be a small Fedora 32 based VM with 2GB of
memory.
I'm not sure f32 will work with 2gb memory anymore. I dont think it installs at any rate.
I do like the idea of just making it an openshift pod. Perhaps this could even fit with pingous 'toddlers' setup. ie:
I tried to create a toddler, but that setup is too complicated for me. Especially if something is not working it will be almost impossible for me to debug it if it is running somewhere I cannot reach via SSH.
I just tried to build the generate-mirrorlist-cache on RHEL 7 (using Rust from EPEL) and it works fine. Instead of 20 minutes it needs 30 seconds to generate the mirrorlist cache file on mm-backend01.
Although a RPM is available in Fedora I am not sure the RPM can be made available in EPEL 7.
RPM Fusion is using the Rust based generate-mirrorlist-cache for some months already and I do not see any problems with it.
We are not wanting to deploy new EL7 systems but would probably install an EL8 box for this. Does this change the need for moving to Fedora on it?
I would like to use this also for Fedora and a dedicated Fedora based VM or building from source on RHEL 7 is both possible. Building from source does not sound like something used in Fedora Infrastructure so setting up a Fedora based VM would be necessary. Would that be possible?
Adrian
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Mon, Oct 05, 2020 at 08:30:12AM -0400, Stephen John Smoogen wrote:
On Mon, 5 Oct 2020 at 02:24, Adrian Reber adrian@lisas.de wrote:
On Wed, Jul 01, 2020 at 09:16:05AM -0700, Kevin Fenzi wrote:
On Tue, Jun 30, 2020 at 02:17:30PM +0200, Adrian Reber wrote:
On Mon, Jun 15, 2020 at 03:36:23PM -0700, Kevin Fenzi wrote:
On Wed, Jun 10, 2020 at 11:09:49AM +0200, Adrian Reber wrote:
Then I just have to wait a bit. No problem.
> > Having the possibility to generate the mirrorlist input data
in about a
> > minute would significantly reduce the load on the database
server and
> > enable us to react much faster if broken protobuf data has
been synced
> > to the mirrorlist servers on the proxies. > > Yeah, and I wonder if it would let us revisit the entire
sequence from
> 'update push finished' to updated mirrorlist server.
This would help us with the case of:
- updates push/rawhide finishes, master mirror is updated.
- openqa/other internal thing tries to get images or updates in that change and gets a metalink with the old checksum so it can't get
the
new stuff.
- mm-backend01 generates and pushes out a new protobuf.
Probably. As the new code will not run on the current RHEL 7 based mm-backend01 would it make sense to run a short running service
like
this on Fedora's OpenShift? We could also create a new read-only
(SELECT
only) database account for this.
We could, or as smooge suggests make a mm-backend02?
But I guess now mm-backend02 just generates new proobuf files and
copies
them to mirrorlists? If thats all it's doing, perhaps we could indeed replace it with an openshift project.
We need a system to run the tool and copy the data to all proxies.
I would like to see a new MirrorManager database user who can only do selects as that is all we need.
Currently we copy the files via SSH to the proxies, if we continue
doing
it that way, then we would also need the existing SSH key to copy the data to the proxies.
Easiest would probably be a small Fedora 32 based VM with 2GB of
memory.
I'm not sure f32 will work with 2gb memory anymore. I dont think it installs at any rate.
I do like the idea of just making it an openshift pod. Perhaps this could even fit with pingous 'toddlers' setup. ie:
I tried to create a toddler, but that setup is too complicated for me. Especially if something is not working it will be almost impossible for me to debug it if it is running somewhere I cannot reach via SSH.
I just tried to build the generate-mirrorlist-cache on RHEL 7 (using Rust from EPEL) and it works fine. Instead of 20 minutes it needs 30 seconds to generate the mirrorlist cache file on mm-backend01.
Although a RPM is available in Fedora I am not sure the RPM can be made available in EPEL 7.
RPM Fusion is using the Rust based generate-mirrorlist-cache for some months already and I do not see any problems with it.
We are not wanting to deploy new EL7 systems but would probably install an EL8 box for this. Does this change the need for moving to Fedora on it?
I just asked on #fedora-rust, but it seems it is not easily possible to build the Fedora Rust packages for EL8. If I am understanding it correctly it seems we need to run the Rust based mirrorlist cache generation on a Fedora host. If we have a second mm-backend system (mm-backend02) that is Fedora based to generate the mirrorlist cache we could decrease the amount of RAM (32GB) on mm-backend01 to something like 8GB.
Adrian
On Tue, 6 Oct 2020 at 03:46, Adrian Reber adrian@lisas.de wrote:
On Mon, Oct 05, 2020 at 08:30:12AM -0400, Stephen John Smoogen wrote:
On Mon, 5 Oct 2020 at 02:24, Adrian Reber adrian@lisas.de wrote:
We are not wanting to deploy new EL7 systems but would probably install
an
EL8 box for this. Does this change the need for moving to Fedora on it?
I just asked on #fedora-rust, but it seems it is not easily possible to build the Fedora Rust packages for EL8. If I am understanding it correctly it seems we need to run the Rust based mirrorlist cache generation on a Fedora host. If we have a second mm-backend system (mm-backend02) that is Fedora based to generate the mirrorlist cache we could decrease the amount of RAM (32GB) on mm-backend01 to something like 8GB.
OK that makes sense. This will be something that needs upgrading every 6 months like our proxies, but it is what it is.
Adrian
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Tue, Oct 06, 2020 at 10:38:40AM -0400, Stephen John Smoogen wrote:
On Tue, 6 Oct 2020 at 03:46, Adrian Reber adrian@lisas.de wrote:
On Mon, Oct 05, 2020 at 08:30:12AM -0400, Stephen John Smoogen wrote:
On Mon, 5 Oct 2020 at 02:24, Adrian Reber adrian@lisas.de wrote:
We are not wanting to deploy new EL7 systems but would probably install
an
EL8 box for this. Does this change the need for moving to Fedora on it?
I just asked on #fedora-rust, but it seems it is not easily possible to build the Fedora Rust packages for EL8. If I am understanding it correctly it seems we need to run the Rust based mirrorlist cache generation on a Fedora host. If we have a second mm-backend system (mm-backend02) that is Fedora based to generate the mirrorlist cache we could decrease the amount of RAM (32GB) on mm-backend01 to something like 8GB.
OK that makes sense. This will be something that needs upgrading every 6 months like our proxies, but it is what it is.
I have also seen that countme is deployed from ansible directly from git. I could do that also for mirrorlist cache generation code on RHEL 7 which would mean we do not need any other hosts. We could just run it directly on mm-backend01 as it is right now and switch between the existing Python based code and the new Rust based code just as we need it.
Adrian
On Mon, Nov 02, 2020 at 07:34:12AM +0100, Adrian Reber wrote:
On Tue, Oct 06, 2020 at 10:38:40AM -0400, Stephen John Smoogen wrote:
On Tue, 6 Oct 2020 at 03:46, Adrian Reber adrian@lisas.de wrote:
On Mon, Oct 05, 2020 at 08:30:12AM -0400, Stephen John Smoogen wrote:
On Mon, 5 Oct 2020 at 02:24, Adrian Reber adrian@lisas.de wrote:
We are not wanting to deploy new EL7 systems but would probably install
an
EL8 box for this. Does this change the need for moving to Fedora on it?
I just asked on #fedora-rust, but it seems it is not easily possible to build the Fedora Rust packages for EL8. If I am understanding it correctly it seems we need to run the Rust based mirrorlist cache generation on a Fedora host. If we have a second mm-backend system (mm-backend02) that is Fedora based to generate the mirrorlist cache we could decrease the amount of RAM (32GB) on mm-backend01 to something like 8GB.
OK that makes sense. This will be something that needs upgrading every 6 months like our proxies, but it is what it is.
I have also seen that countme is deployed from ansible directly from git. I could do that also for mirrorlist cache generation code on RHEL 7 which would mean we do not need any other hosts. We could just run it directly on mm-backend01 as it is right now and switch between the existing Python based code and the new Rust based code just as we need it.
Sure, thats ok with me, as long as in ansible we pin to a specific ref on the git repo (ie, don't just say HEAD and get changes on ansible runs).
kevin
infrastructure@lists.fedoraproject.org
-
Adrian Reber -
Kevin Fenzi -
Pierre-Yves Chibon -
Stephen John Smoogen