Good Morning Everyone,
Our infrastructure is mostly a python store, meaning almost all our apps are
written in python and most using wsgi.
However in python we are using a number of framework:
* flask for most
* pyramid for some of the biggest (bodhi, FAS3)
* Django (askbot, Hyperkitty)
* TurboGears2 (fedora-packages)
* aiohttp (python3, async app: mdapi)
While this makes sometime things difficult, these are fairly standard framework
and most of our developers are able to help on all.
However, as I see us starting to look at JS for some of our apps (fedora-hubs,
wartaa...), I wonder if we could start the discussion early about the different
framework and eventually see if we can unify around one.
This would also allow those of us not familiar with any JS framework to look at
the recommended one instead of picking one up semi-randomly.
So has anyone experience with one or more JS framework? Do you have one that
would you recommend? Why?
Thanks for your inputs,
Our MirrorManager setup exports the current state of all mirrors every
hour at :30 to a protobuf based file which is then used by the
mirrorlist servers to answer the requests from yum and dnf.
The Python script requires up to 10GB of memory and takes between 35 and
50 minutes. The script does a lot of SQL queries and also some really
big SQL queries joining up to 6 large MirrorManager tables.
I have rewritten this Python script in Rust and now it only needs around
1 minute instead of 35 to 50 minutes and only 600MB instead of 10GB.
I think the biggest difference is that I am almost not doing any joins
in my SQL request. I download all the tables once and then I do a lot of
loops over the downloaded tables and this seems to be massively faster.
As the mirrorlist-server in Rust has proven to be extremely stable over
the last months we have been using it I would also like to replace the
mirrorlist protbuf input generation with my new Rust based code.
I am planing to try out the new protobuf file in staging in the next
days and would then try to get my new protobuf generation program into
Fedora. Once it is packaged I would discuss here how and if we want to
deploy in Fedora's infrastructure.
Having the possibility to generate the mirrorlist input data in about a
minute would significantly reduce the load on the database server and
enable us to react much faster if broken protobuf data has been synced
to the mirrorlist servers on the proxies.
we are now in the infrastructure freeze leading up to the Fedora 33
Final release. This is a final release freeze.
We do this to ensure that our infrastructure is stable and ready to
release Fedora 33 when it's available.
You can see a list of hosts that do not freeze by checking out the
ansible repo and running the freezelist script:
ansible/scripts/freezelist -i inventory
Any hosts listed as freezes is frozen until 2020-10-21 (or later if
release slips). Frozen hosts should have no changes made to them without
a sign-off on the change from at least 2 sysadmin-main or rel-eng
members, along with (in most cases) a patch of the exact change to be
made to this list.
You are kindly invited to the meeting:
Fedora Infrastructure on 2020-10-29 from 15:00:00 to 16:00:00 UTC
The meeting will be about:
Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before.
To test authentication with the new AAA system I'd like to deploy a couple
very basic apps that do nothing but auth in staging's openshift. It
shouldn't touch any configuration besides the reverse proxies and the new
project in openshift. And it's staging only.
Is it OK?
On Mon, Oct 26, 2020 at 04:00:06PM +0100, Jan Kasprzak wrote:
> Kevin Fenzi wrote:
> : On Mon, Oct 26, 2020 at 11:12:32AM +0100, Jan Kasprzak wrote:
> : > Hello,
> : >
> : >
> : > Kevin Fenzi wrote:
> : > : As some of you may know, there's been reports of slow sync times for
> : > : content from the master mirrors last week.
> : > : (see: https://pagure.io/fedora-infrastructure/issue/9392 )
> : >
> : > ftp.linux.cz Tier1 mirror admin here. Thanks for the heads up.
> : >
> : > : * Our two mirrors located in other datacenters:
> : > : download-ib01.fedoraproject.org
> : > : and
> : > : download-cc-rdu01.fedoraproject.org
> : >
> : > None of the above contain the fedora-enchilada0 rsync package,
> : > and in the fedora-releases package there is no 33 subdirectory.
> : > Which package should I use for rsyncing?
> : Odd. They do have it defined. Just like the main master mirrors. ;(
> : It could be something to do with them having ipv6 addresses and you
> : coming from a different ipv6 address? We have
> : ftp.linux.cz
> : in rsyncd.conf... so it should reverse lookup the ip and allow it, just
> : like the iad2 masters. ;(
> OK, the reverse lookup would be a problem then. Adding -4 to the
> rsync command line fixes the problem - I can now download from
> Can you add 2001:718:801:230::cd to the ACL? Unfortunately, I have
> many different FQDNs pointing to my server, so PTRs can be inconsistent
> (my fault, I know).
We can. Would it be ok to add it in a few days?
Or do you need it now?
On Mon, Oct 26, 2020 at 11:12:32AM +0100, Jan Kasprzak wrote:
> Kevin Fenzi wrote:
> : As some of you may know, there's been reports of slow sync times for
> : content from the master mirrors last week.
> : (see: https://pagure.io/fedora-infrastructure/issue/9392 )
> ftp.linux.cz Tier1 mirror admin here. Thanks for the heads up.
> : * Our two mirrors located in other datacenters:
> : download-ib01.fedoraproject.org
> : and
> : download-cc-rdu01.fedoraproject.org
> None of the above contain the fedora-enchilada0 rsync package,
> and in the fedora-releases package there is no 33 subdirectory.
> Which package should I use for rsyncing?
Odd. They do have it defined. Just like the main master mirrors. ;(
It could be something to do with them having ipv6 addresses and you
coming from a different ipv6 address? We have
in rsyncd.conf... so it should reverse lookup the ip and allow it, just
like the iad2 masters. ;(
If you can drop by #fedora-admin on irc.freenode.net we can try and
debug it? Or I can just add some hard coded IPv4/IPv6 addresses for you?
> : may provide faster speeds than the dl-tier01.fedoraproject.org mirrors
> : right now depending on your location/routing, so you may want to at
> : least temporarily sync against them.
> So far I have manually hardlinked releases/test/33_Beta tree to
> releases/33, and I am running rsync of
> on top of it. The download speed around 50 Mbit/s.
> Is there a faster way how to get my mirror in sync? Thanks,
Well, fedora-quick-mirror will only copy those specific files you still
need and won't cause our end to have to stat the entire tree so you can
compare it (like bare rsync does). But otherwise that should work...
As some of you may know, there's been reports of slow sync times for
content from the master mirrors last week.
(see: https://pagure.io/fedora-infrastructure/issue/9392 )
This was caused by many routes prefering one of our upstream connections
and saturating it. We are working on increasing BW and rebalancing
routes, but these might take a while to fully take effect.
In the mean time, since Fedora 33 is due out tuesday:
* I have disabled rsync on dl01/02/03. These are our 'non tier1' rsync
servers. Likely we will enable them again tomorrow night to allow
additional mirrors to sync. Note however that if you are not a tier1
mirror, you should sync from one of those:
* Our two mirrors located in other datacenters:
may provide faster speeds than the dl-tier01.fedoraproject.org mirrors
right now depending on your location/routing, so you may want to at
least temporarily sync against them.
* Mirrors are strongly advised to use the https://pagure.io/quick-fedora-mirror/
script to sync if it all possible. This saves you and us lots of I/O and
just syncs the content you need.
Please do check your mirrors and make sure they are syncing up the
Fedora 33 content correctly before tuesdays release.