-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Hi,
I am working on a project designed to mirror large, changing archives of software in a manner that ensures data integrity and atomic updates using a peer-to-peer protocol.
My team and I would like to design this software such that it provides a suitable replacement for rsync. As Fedora is among the leading distributions, I would like to solicit your opinions on the implementation. We have designed much of the architecture, but have not implemented anything yet. One issue we wish to address currently is that of interpreted vs. compiled languages.
My team and I would like to know whether the community would be accepting of such a project (which includes a daemon) if it were written in Python rather than C or C++. If so, it would greatly simplify the implementation and allow it to be more robust. Python's built-in libraries and facilities provide much of the path and network manipulation that the daemon requires. Using the Python standard libraries allows us to rely upon a well-tested base and focus on higher-level issues.
What are your opinions, as system administrators, on using Python for long-running daemon processes when the developers are explicitly mindful of memory considerations?
- --Ben Boeckel
On Sun, 25 Oct 2009, Ben Boeckel wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Hi,
I am working on a project designed to mirror large, changing archives of software in a manner that ensures data integrity and atomic updates using a peer-to-peer protocol.
My team and I would like to design this software such that it provides a suitable replacement for rsync. As Fedora is among the leading distributions, I would like to solicit your opinions on the implementation. We have designed much of the architecture, but have not implemented anything yet. One issue we wish to address currently is that of interpreted vs. compiled languages.
My team and I would like to know whether the community would be accepting of such a project (which includes a daemon) if it were written in Python rather than C or C++. If so, it would greatly simplify the implementation and allow it to be more robust. Python's built-in libraries and facilities provide much of the path and network manipulation that the daemon requires. Using the Python standard libraries allows us to rely upon a well-tested base and focus on higher-level issues.
What are your opinions, as system administrators, on using Python for long-running daemon processes when the developers are explicitly mindful of memory considerations?
I've generally had better luck with C/C++ based daemons from a systems admin point of view. But we've used plenty of python based daemons that worked just fine. I think the problem is the lower barrier to using python means it's easier for less experienced developers to create python daemons.
Really though, if it's done right, python daemons can be just as good as C daemons. And extending Python is very easy as you mentioned with using pythons built in libraries. If you're concerned about performance though it shouldn't take much to do the basics of what you want and compare C to python. Sometimes they're identical, sometimes C wins but I don't think I've seen python win yet (with performance). But it is much easier to work with python :)
-Mike
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Mike McGrath wrote:
On Sun, 25 Oct 2009, Ben Boeckel wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Hi,
I am working on a project designed to mirror large, changing archives of software in a manner that ensures data integrity and atomic updates using a peer-to-peer protocol.
My team and I would like to design this software such that it provides a suitable replacement for rsync. As Fedora is among the leading distributions, I would like to solicit your opinions on the implementation. We have designed much of the architecture, but have not implemented anything yet. One issue we wish to address currently is that of interpreted vs. compiled languages.
My team and I would like to know whether the community would be accepting of such a project (which includes a daemon) if it were written in Python rather than C or C++. If so, it would greatly simplify the implementation and allow it to be more robust. Python's built-in libraries and facilities provide much of the path and network manipulation that the daemon requires. Using the Python standard libraries allows us to rely upon a well-tested base and focus on higher-level issues.
What are your opinions, as system administrators, on using Python for long-running daemon processes when the developers are explicitly mindful of memory considerations?
I've generally had better luck with C/C++ based daemons from a systems admin point of view. But we've used plenty of python based daemons that worked just fine. I think the problem is the lower barrier to using python means it's easier for less experienced developers to create python daemons.
Really though, if it's done right, python daemons can be just as good as C daemons. And extending Python is very easy as you mentioned with using pythons built in libraries. If you're concerned about performance though it shouldn't take much to do the basics of what you want and compare C to python. Sometimes they're identical, sometimes C wins but I don't think I've seen python win yet (with performance). But it is much easier to work with python :)
-Mike
Thanks for your reply.
Seeing as it is a mirroring daemon, the network is the bottleneck. If it isn't then either you're sitting next door, our implementation is bad, or the hardware shouldn't be a mirror in the first place.
- --Ben
On Sun, 25 Oct 2009, Ben Boeckel wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Mike McGrath wrote:
On Sun, 25 Oct 2009, Ben Boeckel wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Hi,
I am working on a project designed to mirror large, changing archives of software in a manner that ensures data integrity and atomic updates using a peer-to-peer protocol.
My team and I would like to design this software such that it provides a suitable replacement for rsync. As Fedora is among the leading distributions, I would like to solicit your opinions on the implementation. We have designed much of the architecture, but have not implemented anything yet. One issue we wish to address currently is that of interpreted vs. compiled languages.
My team and I would like to know whether the community would be accepting of such a project (which includes a daemon) if it were written in Python rather than C or C++. If so, it would greatly simplify the implementation and allow it to be more robust. Python's built-in libraries and facilities provide much of the path and network manipulation that the daemon requires. Using the Python standard libraries allows us to rely upon a well-tested base and focus on higher-level issues.
What are your opinions, as system administrators, on using Python for long-running daemon processes when the developers are explicitly mindful of memory considerations?
I've generally had better luck with C/C++ based daemons from a systems admin point of view. But we've used plenty of python based daemons that worked just fine. I think the problem is the lower barrier to using python means it's easier for less experienced developers to create python daemons.
Really though, if it's done right, python daemons can be just as good as C daemons. And extending Python is very easy as you mentioned with using pythons built in libraries. If you're concerned about performance though it shouldn't take much to do the basics of what you want and compare C to python. Sometimes they're identical, sometimes C wins but I don't think I've seen python win yet (with performance). But it is much easier to work with python :)
-Mike
Thanks for your reply.
Seeing as it is a mirroring daemon, the network is the bottleneck. If it isn't then either you're sitting next door, our implementation is bad, or the hardware shouldn't be a mirror in the first place.
With all my babbling I forgot to mention we do already run python daemons in Fedora Infrastructure. Func is one, TurboGears (though it's wrapped in mod_wsgi) and one that we wrote ourselves[1] is our mirrorlist server. It's the backend that powers:
http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-11&arch=i386
-Mike
[1] By 'ourselves' I mean Matt Domsch who, quite honestly, has designed the most robust, fault tolerant dynamic content system we run.
On 10/25/2009 11:51 PM, Mike McGrath wrote:
With all my babbling I forgot to mention we do already run python daemons in Fedora Infrastructure. Func is one, TurboGears (though it's wrapped in mod_wsgi) and one that we wrote ourselves[1] is our mirrorlist server. It's the backend that powers:
http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-11&arch=i386
And koji.fedoraproject.org, no?
And translation is going through Django these days?
-- Jeroen
On Mon, 26 Oct 2009, Jeroen van Meeuwen wrote:
On 10/25/2009 11:51 PM, Mike McGrath wrote:
With all my babbling I forgot to mention we do already run python daemons in Fedora Infrastructure. Func is one, TurboGears (though it's wrapped in mod_wsgi) and one that we wrote ourselves[1] is our mirrorlist server. It's the backend that powers:
http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-11&arch=i386
And koji.fedoraproject.org, no?
Yes but I think it's more along the lines of a python script that gets called when a page gets loaded, it doesn't actually run statefully like our tg apps do.
And translation is going through Django these days?
That's true, django is another one we run. I bet there's a few more sitting around that we haven't thought of yet :)
-Mike
On 10/26/2009 12:37 AM, Mike McGrath wrote:
On Mon, 26 Oct 2009, Jeroen van Meeuwen wrote:
On 10/25/2009 11:51 PM, Mike McGrath wrote:
With all my babbling I forgot to mention we do already run python daemons in Fedora Infrastructure. Func is one, TurboGears (though it's wrapped in mod_wsgi) and one that we wrote ourselves[1] is our mirrorlist server. It's the backend that powers:
http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-11&arch=i386
And koji.fedoraproject.org, no?
Yes but I think it's more along the lines of a python script that gets called when a page gets loaded, it doesn't actually run statefully like our tg apps do.
But the backend does run python foo, right?
And translation is going through Django these days?
That's true, django is another one we run. I bet there's a few more sitting around that we haven't thought of yet :)
I bet too ;-))
-- Jeroen
On Sunday 25 October 2009 06:26:49 pm Jeroen van Meeuwen wrote:
On 10/25/2009 11:51 PM, Mike McGrath wrote:
With all my babbling I forgot to mention we do already run python daemons in Fedora Infrastructure. Func is one, TurboGears (though it's wrapped in mod_wsgi) and one that we wrote ourselves[1] is our mirrorlist server. It's the backend that powers:
http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-11&arch=i386
And koji.fedoraproject.org, no?
koji is a mod_python app. it doesnt run as a daemon at all. but it it all python.
On Mon, Oct 26, 2009 at 12:41 AM, Dennis Gilmore dennis@ausil.us wrote:
On Sunday 25 October 2009 06:26:49 pm Jeroen van Meeuwen wrote:
And koji.fedoraproject.org, no?
koji is a mod_python app. it doesnt run as a daemon at all. but it it all python.
There are python daemons in the system though. The builders run kojid, which is a daemon, and the process that triggers repo regeneration (kojira) is a daemon. Of course, these are not public facing -- they really only talk to the hub.
The daemon distinction might be wrong thing to fixate on here. There is nothing in that distinction that should exclude python (or most any language). I think the real factors of concern are: size, complexity, speed, system load, robustness, and security.
It's entirely possible to create large and complex systems with Python. It's expressiveness and interpreted nature are a big help here. Robustness is probably more a factor of design and good coding than the language itself. That being said, the relative ease of programming in Python, coupled with its exception handling, likely give it an advantage over C in this department (over the same development time at least). That's my opinion; others may disagree (and of course Python is not the only rapid development language with good exception handling).
Security threats are everywhere. With Python at least you have to worry much less about buffer overflows, but of course you can introduce security flaws in any language.
Obviously, as an interpreted language, system load and speed are where Python suffers. There are optimization tricks, but you'll never get close to execution speed of C. You can rewrite crucial portions in C, though I rarely see that actually done.
On Wed, Oct 28, 2009 at 05:08:03PM -0400, Mike McLean wrote:
On Mon, Oct 26, 2009 at 12:41 AM, Dennis Gilmore dennis@ausil.us wrote:
The daemon distinction might be wrong thing to fixate on here. There is nothing in that distinction that should exclude python (or most any language). I think the real factors of concern are: size, complexity, speed, system load, robustness, and security.
<nod> By and large I agree with you. One thing further to think about is that becoming dependent on a tool written in an interpreted language means that you need to install that language on your system and may become tied to a specific version of that language. In theory, this shouldn't be worse than tying yourself to a specific C library but in practice I've found that parallel installing interpreters and their libraries is a lot less supported than parallel installing a C lib.
Using python in Fedora Infrastructure probably isn't too big a deal as we have abundant python programmers here to port things forward if the main developers don't but it is something to consider with other languages or in other environments.
-Toshio
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Toshio Kuratomi wrote:
On Wed, Oct 28, 2009 at 05:08:03PM -0400, Mike McLean wrote:
On Mon, Oct 26, 2009 at 12:41 AM, Dennis Gilmore <dennis-
OMEdqnBlyiw@public.gmane.org> wrote:
The daemon distinction might be wrong thing to fixate on here. There is nothing in that distinction that should exclude python (or most any language). I think the real factors of concern are: size, complexity, speed, system load, robustness, and security.
<nod> By and large I agree with you. One thing further to think about is that becoming dependent on a tool written in an interpreted language means that you need to install that language on your system and may become tied to a specific version of that language. In theory, this shouldn't be worse than tying yourself to a specific C library but in practice I've found that parallel installing interpreters and their libraries is a lot less supported than parallel installing a C lib.
Hopefully we have 2.4 as the minimum version. I'm pretty sure 2.4 is ubiquitous even on the enterprise-y distributions, but if we need older, we can try for that as well.
Using python in Fedora Infrastructure probably isn't too big a deal as we have abundant python programmers here to port things forward if the main developers don't but it is something to consider with other languages or in other environments.
We're aiming to get it to work on RHEL/CentOS 4 (base, not + EPEL) and Debian stable so that current mirrors running systems that old don't need to upgrade the OS to start using our software (I'm pretty sure we have the hardware to run test instances of the software with various setups). If the libraries we end up using just aren't available, I think RHEL 5 will be a suitable minimum. When Python 3 comes looming over any distribution that starts using our software, we should be able to help port things over. I'd expect Fedora to be the first for this to happen to, but I could see Gentoo having howtos for getting 3.0 as the main Python before that (not that servers would be running such a setup anyways).
-Toshio
- --Ben
On Thu, Oct 29, 2009 at 12:04:08AM -0400, Ben Boeckel wrote:
Hopefully we have 2.4 as the minimum version. I'm pretty sure 2.4 is ubiquitous even on the enterprise-y distributions, but if we need older, we can try for that as well.
Just a note. If you're targeting RHEL/CentOS4 as your minimum, you need to target python-2.3 as your minimum python version.
-Toshio
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Mike McLean wrote:
On Mon, Oct 26, 2009 at 12:41 AM, Dennis Gilmore <dennis-
OMEdqnBlyiw@public.gmane.org> wrote:
On Sunday 25 October 2009 06:26:49 pm Jeroen van Meeuwen wrote:
And koji.fedoraproject.org, no?
koji is a mod_python app. it doesnt run as a daemon at all. but it it all python.
There are python daemons in the system though. The builders run kojid, which is a daemon, and the process that triggers repo regeneration (kojira) is a daemon. Of course, these are not public facing -- they really only talk to the hub.
The daemon distinction might be wrong thing to fixate on here. There is nothing in that distinction that should exclude python (or most any language). I think the real factors of concern are: size, complexity, speed, system load, robustness, and security.
True. I think the speed at which we can deliver with Python makes it much more attractive, even if not just at first. We could something similar to what git did with bash and replace the Python with C incrementally if necessary, but I think that if we go about it right, it shouldn't be much of an issue.
It's entirely possible to create large and complex systems with Python. It's expressiveness and interpreted nature are a big help here. Robustness is probably more a factor of design and good coding than the language itself. That being said, the relative ease of programming in Python, coupled with its exception handling, likely give it an advantage over C in this department (over the same development time at least). That's my opinion; others may disagree (and of course Python is not the only rapid development language with good exception handling).
I agree here. This thread is a fact-finding mission and research. We wanted to make sure we weren't headed down the wrong road before we headed out.
Security threats are everywhere. With Python at least you have to worry much less about buffer overflows, but of course you can introduce security flaws in any language.
Of course.
Obviously, as an interpreted language, system load and speed are where Python suffers. There are optimization tricks, but you'll never get close to execution speed of C. You can rewrite crucial portions in C, though I rarely see that actually done.
One place where we had concerns were when dealing with the "shopping list" I referred to elsewhere in the thread. This list is 60MB on one of the Ubuntu mirror directories we have here (our Fedora mirror has been decommissioned while we await more disk space). I imagine Fedora's will be much larger due to the drpm files and secondary architectures put even more into it. We have code that is able to extract information without loading the entire file into our process space. It's very close to what it would be if it were C except for a .split() call on a string (but it's simple enough that a sscanf could do it as well if needed) and the memory/strcmp stuff Python has baked in. Hopefully we will have unit tests and timing on our code this weekend.
- --Ben
On Sun, Oct 25, 2009 at 06:46:05PM -0400, Ben Boeckel wrote:
Seeing as it is a mirroring daemon, the network is the bottleneck. If it isn't then either you're sitting next door, our implementation is bad, or the hardware shouldn't be a mirror in the first place.
Speaking from experience, the network isn't always the bottleneck. I/O performance is often a performance problem, especially when walking the directory tree to build filelists. CPU performance can come into play if you are performing hashes or compression of the data to be transferred. I suggest you post your message to the Fedora mirror-list-d where I'm sure you'll get lots of feedback.
On Mon, 26 Oct 2009, Chuck Anderson wrote:
On Sun, Oct 25, 2009 at 06:46:05PM -0400, Ben Boeckel wrote:
Seeing as it is a mirroring daemon, the network is the bottleneck. If it isn't then either you're sitting next door, our implementation is bad, or the hardware shouldn't be a mirror in the first place.
Speaking from experience, the network isn't always the bottleneck. I/O performance is often a performance problem, especially when walking the directory tree to build filelists. CPU performance can come into play if you are performing hashes or compression of the data to be transferred. I suggest you post your message to the Fedora mirror-list-d where I'm sure you'll get lots of feedback.
Very true, if this behaves similarly to rsync. Reading over a large change set to transmit only small changes is very resource intensive everywhere but the network.
-Mike
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Mike McGrath wrote:
On Mon, 26 Oct 2009, Chuck Anderson wrote:
On Sun, Oct 25, 2009 at 06:46:05PM -0400, Ben Boeckel wrote:
Seeing as it is a mirroring daemon, the network is the bottleneck. If it
isn't
then either you're sitting next door, our implementation is bad, or the hardware shouldn't be a mirror in the first place.
Speaking from experience, the network isn't always the bottleneck. I/O performance is often a performance problem, especially when walking the directory tree to build filelists. CPU performance can come into play if you are performing hashes or compression of the data to be transferred. I suggest you post your message to the Fedora mirror-list-d where I'm sure you'll get lots of feedback.
Very true, if this behaves similarly to rsync. Reading over a large change set to transmit only small changes is very resource intensive everywhere but the network.
-Mike
The way we are planning to handle updates should keep this small. The network side knows nothing of the on-disk file structure and the tree-creator part knows nothing of the network other than "it's where the things I need come from", so updates won't be "what's changed in this directory?" so much as "what do I not have that is on my shopping list?" which is then asked for of the network.
- --Ben
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Chuck Anderson wrote:
On Sun, Oct 25, 2009 at 06:46:05PM -0400, Ben Boeckel wrote:
Seeing as it is a mirroring daemon, the network is the bottleneck. If it
isn't
then either you're sitting next door, our implementation is bad, or the hardware shouldn't be a mirror in the first place.
Speaking from experience, the network isn't always the bottleneck. I/O performance is often a performance problem, especially when walking the directory tree to build filelists.
AIUI, the tree will only be walked over during update composition and applying an update to the mirror (updates are done out-of-tree so that there is no catching a server "with its pants down" per se.
CPU performance can come into play if you are performing hashes or compression of the data to be transferred.
Yes, we'll have to keep an eye on this. As things come down from the tubes, the hashes should be working so they'll most likely be waiting on the network to get done with its files.
I suggest you post your message to the Fedora mirror-list-d where I'm sure you'll get lots of feedback.
Will do. Thanks for the tip.
- --Ben
On Sun, Oct 25, 2009 at 06:14:14PM -0400, Ben Boeckel wrote:
My team and I would like to know whether the community would be accepting of such a project (which includes a daemon) if it were written in Python rather than C or C++.
No problems here, MirrorManager includes several daemon processes written in python.
The only real problem I have with python is the memory manager. I have to constantly think "gee, if I read this file line by line and do something with it, will my RAM needs grow to 10-20x the size of the file?" And I love python on x86_64 - it makes people buy twice as much RAM from Dell! *
So, go for it. If it turns out that python is inefficient for parts of the problem, you can always split those parts out. I had to split out the mirrorlist_server bit from the mod_wsgi app, to get the benefits of shared pages (fork() on a 100+MB process) w/o killing Apache with duplicate processes but no sharing.
* (no kidding: the mirrorlist_server process takes 125MB on x86_32, 250MB on x86_64. ugh.)
infrastructure@lists.fedoraproject.org
-
Ben Boeckel -
Chuck Anderson -
Dennis Gilmore -
Jeroen van Meeuwen -
Matt Domsch -
Mike McGrath -
Mike McLean -
Toshio Kuratomi