Phoronix is known to perform various tests on Linux distribution, which is IMHO very interesting, as there were not such a real metrics in past - just user feelings. But Fedora is not collecting the medals. You may say "It's rawhide" or "We do't care", but users (and potential future developers, translators etc.) do care: http://www.phoronix.com/vr.php?view=14045
Therefore I think, this kind of tests should become at least a part of QA procedure for a release, to make sure we do not loose too much while beeing "first".
Adam Pribyl
On 07/28/2009 08:13 AM, Adam Pribyl wrote: <snip> Removed mentioning of phoronix and their extraordinaire benchmarking techniques... </snip>
Therefore I think, this kind of tests should become at least a part of QA procedure for a release, to make sure we do not loose too much while beeing "first".
You cannot reliably "benchmark" rawhide performance nor should we even try. Identifying and fixing bugs comes first, *tuning* comes later.
However actively monitor supported release performance and improving it sounds more a likely candidate but you will need to contact infrastructure to see if they have resources to spare etc..
JBG
On Jul 28, 2009, at 3:08, Karel Volný kvolny@redhat.com wrote:
...
You cannot reliably "benchmark" rawhide performance nor should we even try. Identifying and fixing bugs comes first, *tuning* comes later.
pathetic performance *is* a *bug*
Rawhide has lots of debugging turned on in various places so it is not suitable for benchmarking until this is removed. Target beta or the final release.
-- Jes
On Tue, 28 Jul 2009, Jesse Keating wrote:
On Jul 28, 2009, at 3:08, Karel Volný kvolny@redhat.com wrote:
...
You cannot reliably "benchmark" rawhide performance nor should we even try. Identifying and fixing bugs comes first, *tuning* comes later.
pathetic performance *is* a *bug*
Rawhide has lots of debugging turned on in various places so it is not suitable for benchmarking until this is removed. Target beta or the final release.
Even thou I never looked for a list of what's enabled in rawhide, I'd be surprised if other distros do not have something similar. They may be in different development stage, right (e.g. already beta).
-- Jes
From the other comments: I understand the benchmarking is not only firing
some test suite, but about comparing and tuning. I may try to arange one old PC with "out of the box" rawnhide to run periodicaly e.g. the Phoronix test suite and publish the results - this will provide comparison. Let's see, if it'll be useful.
Adam Pribyl
On Thu, Jul 30, 2009 at 12:14:02PM +0200, Adam Pribyl wrote:
Rawhide has lots of debugging turned on in various places so it is not suitable for benchmarking until this is removed. Target beta or the final release.
Even thou I never looked for a list of what's enabled in rawhide, I'd be surprised if other distros do not have something similar. They may be in different development stage, right (e.g. already beta).
To the best of my knowledge, none of the other popular distros enable the level of debugging we do in the Fedora kernel. There's a slightly dated list detailing the extent of what we enable at https://fedoraproject.org/wiki/KernelDebugStrategy
We're also the only distro that leaves certain debug options enabled post-release.
You can see this by noting the lack of bug reports mentioning 'lockdep' 'slab debug' or other debug options on other distros bug trackers.
Dave
On Thu, 2009-07-30 at 14:21 -0400, Dave Jones wrote:
On Thu, Jul 30, 2009 at 12:14:02PM +0200, Adam Pribyl wrote:
Rawhide has lots of debugging turned on in various places so it is not suitable for benchmarking until this is removed. Target beta or the final release.
Even thou I never looked for a list of what's enabled in rawhide, I'd be surprised if other distros do not have something similar. They may be in different development stage, right (e.g. already beta).
To the best of my knowledge, none of the other popular distros enable the level of debugging we do in the Fedora kernel. There's a slightly dated list detailing the extent of what we enable at https://fedoraproject.org/wiki/KernelDebugStrategy
We're also the only distro that leaves certain debug options enabled post-release.
You can see this by noting the lack of bug reports mentioning 'lockdep' 'slab debug' or other debug options on other distros bug trackers.
Is there some way we can autogenerate some message on the lines of:
"This kernel has the following debugging settings enabled: foo bar baz Don't use it for performance testing (or be aware that it will suck if you do)"
or some similar message, perhaps even displaying it on logon/motd, or something similar?
Are these debug options for the running kernel queryable e.g. via /proc ?
For that matter, do we already have a "this is a pre-release version of Fedora; rawhide will eat your brains" message?
Thoughts?
Dave
On Thu, Jul 30, 2009 at 02:33:38PM -0400, David Malcolm wrote:
Is there some way we can autogenerate some message on the lines of:
"This kernel has the following debugging settings enabled: foo bar baz Don't use it for performance testing (or be aware that it will suck if you do)"
or some similar message, perhaps even displaying it on logon/motd, or something similar?
Are these debug options for the running kernel queryable e.g. via /proc ?
if [ -f /proc/lockdep ]
should do it.
Dave
On 07/31/2009 12:03 AM, David Malcolm wrote:
For that matter, do we already have a "this is a pre-release version of Fedora; rawhide will eat your brains" message?
Thoughts?
Added some boiler plate text to
https://fedoraproject.org/wiki/Fedora_12_Alpha_release_notes#Debugging_Infor...
Can be carried over on all release notes for test releases.
Rahul
On Thu, Jul 30, 2009 at 02:21:27PM -0400, Dave Jones wrote:
On Thu, Jul 30, 2009 at 12:14:02PM +0200, Adam Pribyl wrote:
Rawhide has lots of debugging turned on in various places so it is not suitable for benchmarking until this is removed. Target beta or the final release.
To the best of my knowledge, none of the other popular distros enable the level of debugging we do in the Fedora kernel. There's a slightly dated list detailing the extent of what we enable at https://fedoraproject.org/wiki/KernelDebugStrategy
For what it's worth, in the comments, many people calmly (of course, there were a few angry ones) about this--that Fedora, unlike the others tested, has more debugging code in Rawhide.
The best thing to do--for someone who had the time and inclination, would be to get the phoronix test suite and run it using F11 or perhaps a debugged F12 if they have the ability to redo it, and run it against either of the other two. (I'd go for Ubuntu, since it's on one CD.)
However, I think it's the proverbial tempest in a teapot--I haven't seeen any mention of it on the forums. It's probably one of those things that excites people for a day, but is unlikely to turn the distro hoppers away from trying F12 when it comes out.
Jesse Keating wrote:
On Jul 28, 2009, at 3:08, Karel Volný kvolny@redhat.com wrote:
...
You cannot reliably "benchmark" rawhide performance nor should we even try. Identifying and fixing bugs comes first, *tuning* comes later.
pathetic performance *is* a *bug*
Rawhide has lots of debugging turned on in various places so it is not suitable for benchmarking until this is removed. Target beta or the final release.
Isn't that way too late? I can't imagine any developer taking time away from getting new exciting features in the release just to make the system usably fast again. And testing with debug on can mask functional problems, been there, done that.
On Mon, 2009-08-03 at 16:15 -0400, Bill Davidsen wrote:
Rawhide has lots of debugging turned on in various places so it is not suitable for benchmarking until this is removed. Target beta or the final release.
Isn't that way too late? I can't imagine any developer taking time away from getting new exciting features in the release just to make the system usably fast again.
After beta freeze, you're not allowed to introduce any more new exciting features.
On 28/07/09 10:49, "Jóhann B. Guðmundsson" wrote:
On 07/28/2009 08:13 AM, Adam Pribyl wrote:
<snip> Removed mentioning of phoronix and their extraordinaire benchmarking techniques... </snip>
The fact that they are out there, and that people will read them. Should be enough to warrant some attention. Let's not play Ostrich.
On Tue, Jul 28, 2009 at 11:12:31AM +0100, Frank Murphy wrote:
On 28/07/09 10:49, "Jóhann B. Guðmundsson" wrote:
On 07/28/2009 08:13 AM, Adam Pribyl wrote:
<snip> Removed mentioning of phoronix and their extraordinaire benchmarking techniques... </snip>
The fact that they are out there, and that people will read them. Should be enough to warrant some attention. Let's not play Ostrich.
Have to agree with that one. Ignoring it won't make it go away and, to the general public, whoever that may be, makes Fedora look foolish, IMHO, the equivalent of putting fingers in ears and going "Lalala I can't hear you." :)
However, as Rawhide is alpha/beta--I would think that it would be full of debugging code that would slow it down, making it a poor choice for comparision testing.
For example, in FreeBSD's CURRENT, their UPDATING section always explains that it's got a great deal of debugging code and that if one wants to use it, they are advised to recompile the kernel, removing the debuggging options.
On Tue, Jul 28, 2009 at 1:13 PM, Caolán McNamaracaolanm@redhat.com wrote:
On Tue, 2009-07-28 at 07:07 -0400, Scott Robbins wrote:
However, as Rawhide is alpha/beta--I would think that it would be full of debugging code that would slow it down, making it a poor choice for comparision testing.
What debugging code are we talking about here ?
The kernel has various debug features enabled that can have an (depending on the workload) impact on performance, they get disabled for release kernels and separate -debug kernels are created with those options turned on.
2009/7/28 drago01 drago01@gmail.com
On Tue, Jul 28, 2009 at 1:13 PM, Caolán McNamaracaolanm@redhat.com wrote:
On Tue, 2009-07-28 at 07:07 -0400, Scott Robbins wrote:
However, as Rawhide is alpha/beta--I would think that it would be full of debugging code that would slow it down, making it a poor choice for comparision testing.
What debugging code are we talking about here ?
The kernel has various debug features enabled that can have an (depending on the workload) impact on performance, they get disabled for release kernels and separate -debug kernels are created with those options turned on.
of course, if we wanna test performance, we can test the code with debugging features disabled, like after release.
-- fedora-test-list mailing list fedora-test-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-test-list
You all do realize that this is not as simple as installing your favorite benchmarking application and execute it on a all debugged turned off rawhide right. The benchmarking needs to be run for a certain amount of time to notice potential trends. The output needs to be analyzed from the benchmarking test(s) and the tests need to be repeat with several different compile options and performed on several different architecture again repeated if application(s) that are being benchmarked have various tuning knobs. Then when it has been correctly identify which application(s) are performing *badly* the code needs to be profiled, improving areas in the code correctly identified, suggestion or a patch proposed to upstream etc etc etc ...
If there is enough will and manpower to deploy a proper *performance testing* environment and infra has enough resource to host it. Get together create a Performance SIG and ask infra for resource. If not I suggest you continue to watch the outcome from those phoronix tests, stare at system monitor or create your own graphs in calc.
JBG
On Tue, Jul 28, 2009 at 12:13:28PM +0100, Caolán McNamara wrote:
On Tue, 2009-07-28 at 07:07 -0400, Scott Robbins wrote:
However, as Rawhide is alpha/beta--I would think that it would be full of debugging code that would slow it down, making it a poor choice for comparision testing.
What debugging code are we talking about here ?
I don't know. That's why I said "I would think." :)
I do know that at times, I've had issues that I've posted here, and been given an rpm of say, a kernel that had debugging code removed which then worked.
Sorry if it came off as knowledge, rather than a question, I blame it on the early hour. Rereading my message, I see it was phrased poorly.
Thanks for pointing that out. Let me rephrase it.
I know that in many alpha/beta releases of Linux and other distributions, performance is slowed by a great deal of debugging code?
Is that the case in Rawhide?
On Tue, Jul 28, 2009 at 10:13:19AM +0200, Adam Pribyl wrote:
Phoronix is known to perform various tests on Linux distribution, which is IMHO very interesting, as there were not such a real metrics in past - just user feelings. But Fedora is not collecting the medals. You may say "It's rawhide" or "We do't care", but users (and potential future developers, translators etc.) do care: http://www.phoronix.com/vr.php?view=14045
Therefore I think, this kind of tests should become at least a part of QA procedure for a release, to make sure we do not loose too much while beeing "first".
Did you just volunteer to do that?
josh