Hi
mattdm suggested the upcoming scheduler change should be discussed here.[1] I might not have enough time to talk details at the moment, but I noticed this is coming up relatively soon. This is what I understand so far about the decision.
Thanks for all the software, Alan
[1] https://unix.stackexchange.com/questions/483881/what-does-fedora-workstation...
## CFQ is scheduled for removal
Jens Axboe is planning to remove the CFQ I/O scheduler in 4.21. That is, CFQ is part of the "legacy" single-queue block layer, which is going to be removed for ease of maintenance.[2]
Axboes last comment on this timing was made *after* the fix for data corruption on MQ. I.e. the data corruption covered by the recent thread on this list.
*Obligatory disclaimer*. Read the paragraph above, and consider waiting for the next stable kernel update, before you test MQ (including BFQ) on your own machine :-).
[2] "It's definitely still going" - Jens Abxoe. https://bugzilla.kernel.org/show_bug.cgi?id=201685#c279
## The kernel wants us to choose our new default
For devices which have only one hardware queue, the new upstream default is mq-deadline. Going from CFQ to mq-deadline is a significant change. For example, the deadline scheduler does not support ionice.
The alternative to the MQ deadline scheduler will be BFQ.[3][4] Upstream discussed this, and the powers that be (mostly Axboe :-) are explicitly expecting us (downstreams) to make this decision.[5]
[3] BFQ: http://algo.ing.unimo.it/people/paolo/disk_sched/description.php
[4] https://unix.stackexchange.com/questions/375600/how-to-enable-and-use-the-bf...
[5] "I'd prefer if the distros would lead the way on this, as theyare the ones that will most likely see the most bug reports" - Jens Axboe, https://www.spinics.net/lists/linux-block/msg31062.html
## Arguments for (or against) BFQ?
Paolo Valente kindly wrote an informative response, which I will copy in to a separate message. The following is my very limited first impression.
Personally I lean towards BFQ by default. It appears nominated as the successor to CFQ, I think it's worthy as such, and it makes distinct improvements of it's own. I would recommend it with more confidence if I understood how the improvements work :-).
The deadline scheduler probably isn't a *complete* disaster. Ubuntu ran with the deadline scheduler for a while.[6] I haven't checked whether they changed the tuning knobs though!
RHEL7 defaults to CFQ for SATA drives. This is notable given that it recommends avoiding (or tuning) CFQ on basically any other server hardware (and specifically to avoid it on hardware RAID).[7]
I've tried BFQ on my laptops hard drive (not SSD). It has some associated tests for responsiveness (the "S" tests). I don't have a real-world feel for it, but I agree the test numbers are an impressive improvement over CFQ.
I don't have results for the S tests on the deadline scheduler. I did note the eponymous "deadline" for sync reads has a default of 500 ms.
The other test I have, is that "deadline" doesn't match CFQ's level of fairness for reads v.s. writes, even with the recent addition of WBT. Neither approached what I would actually call fairness. BFQ did.[8] This is due to BFQ's "compensations" for device writeback caching and NCQ. And allegedly for Linux writeback. These extra compensations are the part I don't understand, so far.[9]
[6] https://askubuntu.com/questions/784442/why-does-ubuntu-16-04-set-all-drive-i...
[7] RHEL7 IO schedulers:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...
[8] I tried to closely match the test from the cover letters from the WBT patch series. I don't have detailed statistics, but I believe deadline+WBT was less fair than CFQ. It was definitely not more fair.
https://unix.stackexchange.com/questions/483269/determine-the-specific-benef...
[9] https://groups.google.com/forum/#!topic/bfq-iosched/yjZDn_HnLIc
Paolo Valente kindly wrote a response on my StackExchange question. It is possible StackExchange will remove it, since it was not strictly an answer to my question. So I am re-posting his response on this mailing list.
-------- Forwarded Message --------
From: Paolo Valente paolo.valente@linaro.org
Some information that might be useful for your choice
I'm one of the authors of BFQ, so I'm all but a disinterested party :) But I'll report only numbers obtained with repeatable test.
We have been testing BFQ on SD Cards, eMMC, HDDs, SATA SSDs, and NVMe SSDs. As for HDDs and SSDs, we have run tests with both single-disk and RAID configurations.
In terms of throughput, results can be summarized as follows. With SD Cards, eMMC and HDDs (single and RAID), there is no regression in terms of throughput. In contrast, with HDDs, there is a gain around 20-30% with some workload.
On SSDs, there is a loss of throughput only
- with random sync I/O: around 2-3 % on average SSDs, up to 10-15% on very fast NVMe SSDs. With a workload meant to put BFQ in the most difficult condition, we reached a loss of 18% [1], but in any other third-party test the loss is around 10% in the worst case. This loss is mainly due to the fact that BFQ is not a minimal I/O scheduler. We are working on this. It is not easy; we will need time to fill this gap. - with only-write I/O on very fast SSDs: around 5-10%. This is due to a problem with I/O-request tags. We have already found a solution. Since we do not consider this issue critical, we are giving more priority to other items in our TODO list. If you think otherwise, we are willing to change our priorities.
Because of the above overhead, BFQ cannot process more than 400-500 KIOPS on a commodity CPU.
In terms of responsiveness and latency for time-sensitive applications (such as audio/video players), results are simply incomparable. For example, regardless of the I/O workload in the background, with BFQ applications start as quickly as if the drive was idle. With any of the other schedulers, applications may take ten times as long, or even not start at all (until the background workload is over) [1].
In addition, as for server-like workloads, BFQ enable, e.g., the desired fraction of the I/O bandwidth to be guaranteed to each client (or container, VM, or any other kind of entity sharing storage), while reaching a total throughput not comparable to that reached by any other solution for controlling I/O [2].
Finally, if you are in doubt about some particular workload, we will be glad to test it.
[1] http://algo.ing.unimo.it/people/paolo/disk_sched/results.php
On Tue, 11 Dec 2018 11:48:29 -0000 "Alan Jenkins" alan.christopher.jenkins@gmail.com wrote:
[3] BFQ: http://algo.ing.unimo.it/people/paolo/disk_sched/description.php
[4] https://unix.stackexchange.com/questions/375600/how-to-enable-and-use-the-bf...
[5] "I'd prefer if the distros would lead the way on this, as theyare the ones that will most likely see the most bug reports" - Jens Axboe, https://www.spinics.net/lists/linux-block/msg31062.html
I compiled a custom kernel from the fedora src.rpm for 4.19.8. I turned off all schedulers except NOOP and BFQ. But there was no way in the configuration process (make menuconfig) to set BFQ as default. I tried setting it in kernel-local, but the build process errored because it said NOOP is the default and that disagreed with my choice. I'm running the kernel and it is using noop. And there is no way to change it in the /sys hierarchy.
So, how do I get a fedora kernel to run BFQ?
Readding devel@
On Wed, Dec 12, 2018 at 11:08 AM stan stanl-fedorauser@vfemail.net wrote:
On Tue, 11 Dec 2018 11:48:29 -0000 "Alan Jenkins" alan.christopher.jenkins@gmail.com wrote:
[3] BFQ: http://algo.ing.unimo.it/people/paolo/disk_sched/description.php
[4] https://unix.stackexchange.com/questions/375600/how-to-enable-and-use-the-bf...
[5] "I'd prefer if the distros would lead the way on this, as theyare the ones that will most likely see the most bug reports" - Jens Axboe, https://www.spinics.net/lists/linux-block/msg31062.html
I compiled a custom kernel from the fedora src.rpm for 4.19.8. I turned off all schedulers except NOOP and BFQ. But there was no way in the configuration process (make menuconfig) to set BFQ as default. I tried setting it in kernel-local, but the build process errored because it said NOOP is the default and that disagreed with my choice. I'm running the kernel and it is using noop. And there is no way to change it in the /sys hierarchy.
So, how do I get a fedora kernel to run BFQ?
Short version: Yep, so far in 4.20 there is neither a CONFIG_DEFAULT_BFQ or CONFIG_DEFAULT_IOSCHED="bfq" near as I can tell. Maybe it's different for 4.21.
I used two boot params: scsi_mod.use_blk_mq=1 elevator=bfq. I don't think that's a good way for a distribution to set the default though.
Icky version:
I set the following in /etc/default/grub and then ran grub2-mkconfig (not on Rawhide!) GRUB_CMDLINE_LINUX="scsi_mod.use_blk_mq=1 elevator=bfq zswap.enabled=1 zswap.max_pool_percent=25 zswap.compressor=lz4"
I also created /etc/dracut.conf.d/bfq.conf containing: add_drivers+=" bfq " yes, with spaces, and rebuilt the initramfs
But upon reboot, total implosion. Piles of USB errors and disconnects (the boot device is a Samsung FIT USB stick which fits flush in an Intel NUC). I didn't have time to troubleshoot what's causing this problem, other than to plug the USB stick into another computer to verify the stick is good and hasn't been corrupted. It's possibly related the mq-blk bug in 4.19.0 through 4.19.7 - so I've since upgraded to 4.19.8 which as those patches, but I haven't had a chance to retest.
Am 12.12.18 um 20:28 schrieb Chris Murphy:
But upon reboot, total implosion. Piles of USB errors and disconnects (the boot device is a Samsung FIT USB stick which fits flush in an Intel NUC). I didn't have time to troubleshoot what's causing this problem, other than to plug the USB stick into another computer to verify the stick is good and hasn't been corrupted. It's possibly related the mq-blk bug in 4.19.0 through 4.19.7 - so I've since upgraded to 4.19.8 which as those patches, but I haven't had a chance to retest.
this was NOT fixed in 4.19.7 Fedora had a patch for the distro kernel
Really Readding devel@ this time...
On Wed, Dec 12, 2018 at 11:08 AM stan stanl-fedorauser@vfemail.net wrote:
On Tue, 11 Dec 2018 11:48:29 -0000 "Alan Jenkins" alan.christopher.jenkins@gmail.com wrote:
[3] BFQ: http://algo.ing.unimo.it/people/paolo/disk_sched/description.php
[4] https://unix.stackexchange.com/questions/375600/how-to-enable-and-use-the-bf...
[5] "I'd prefer if the distros would lead the way on this, as theyare the ones that will most likely see the most bug reports" - Jens Axboe, https://www.spinics.net/lists/linux-block/msg31062.html
I compiled a custom kernel from the fedora src.rpm for 4.19.8. I turned off all schedulers except NOOP and BFQ. But there was no way in the configuration process (make menuconfig) to set BFQ as default. I tried setting it in kernel-local, but the build process errored because it said NOOP is the default and that disagreed with my choice. I'm running the kernel and it is using noop. And there is no way to change it in the /sys hierarchy.
So, how do I get a fedora kernel to run BFQ?
Short version: Yep, so far in 4.20 there is neither a CONFIG_DEFAULT_BFQ or CONFIG_DEFAULT_IOSCHED="bfq" near as I can tell. Maybe it's different for 4.21.
I used two boot params: scsi_mod.use_blk_mq=1 elevator=bfq. I don't think that's a good way for a distribution to set the default though.
Icky version:
I set the following in /etc/default/grub and then ran grub2-mkconfig (not on Rawhide!) GRUB_CMDLINE_LINUX="scsi_mod.use_blk_mq=1 elevator=bfq zswap.enabled=1 zswap.max_pool_percent=25 zswap.compressor=lz4"
I also created /etc/dracut.conf.d/bfq.conf containing: add_drivers+=" bfq " yes, with spaces, and rebuilt the initramfs
But upon reboot, total implosion. Piles of USB errors and disconnects (the boot device is a Samsung FIT USB stick which fits flush in an Intel NUC). I didn't have time to troubleshoot what's causing this problem, other than to plug the USB stick into another computer to verify the stick is good and hasn't been corrupted. It's possibly related the mq-blk bug in 4.19.0 through 4.19.7 - so I've since upgraded to 4.19.8 which as those patches, but I haven't had a chance to retest.
Il giorno 12 dic 2018, alle ore 20:30, Chris Murphy lists@colorremedies.com ha scritto:
Really Readding devel@ this time...
On Wed, Dec 12, 2018 at 11:08 AM stan stanl-fedorauser@vfemail.net wrote:
On Tue, 11 Dec 2018 11:48:29 -0000 "Alan Jenkins" alan.christopher.jenkins@gmail.com wrote:
[3] BFQ: http://algo.ing.unimo.it/people/paolo/disk_sched/description.php
[4] https://unix.stackexchange.com/questions/375600/how-to-enable-and-use-the-bf...
[5] "I'd prefer if the distros would lead the way on this, as theyare the ones that will most likely see the most bug reports" - Jens Axboe, https://www.spinics.net/lists/linux-block/msg31062.html
I compiled a custom kernel from the fedora src.rpm for 4.19.8. I turned off all schedulers except NOOP and BFQ. But there was no way in the configuration process (make menuconfig) to set BFQ as default. I tried setting it in kernel-local, but the build process errored because it said NOOP is the default and that disagreed with my choice. I'm running the kernel and it is using noop. And there is no way to change it in the /sys hierarchy.
So, how do I get a fedora kernel to run BFQ?
Short version: Yep, so far in 4.20 there is neither a CONFIG_DEFAULT_BFQ or CONFIG_DEFAULT_IOSCHED="bfq" near as I can tell. Maybe it's different for 4.21.
Yep, hundreds of kilobytes of email have been spent trying to convince Jens Axboe that a default option had to be kept for blk-mq too. At no avail. If anyone wants to restart this fight, I'll be on your side.
I used two boot params: scsi_mod.use_blk_mq=1
Yes, this is needed to switch to blk-mq, until 4.21, when there will be no legacy block any longer. One can also set the corresponding option in the kernel configuration.
elevator=bfq.
I'm not sure this option works with blkmq. IIRC doing this may even cause system failures with blk-mq (but maybe this has been fixed). AFAIK, the only way is to set bfq manually or through, e.g., dev rules, but YMMV.
I don't think that's a good way for a distribution to set the default though.
Icky version:
I set the following in /etc/default/grub and then ran grub2-mkconfig (not on Rawhide!) GRUB_CMDLINE_LINUX="scsi_mod.use_blk_mq=1 elevator=bfq zswap.enabled=1 zswap.max_pool_percent=25 zswap.compressor=lz4"
I also created /etc/dracut.conf.d/bfq.conf containing: add_drivers+=" bfq " yes, with spaces, and rebuilt the initramfs
But upon reboot, total implosion. Piles of USB errors and disconnects (the boot device is a Samsung FIT USB stick which fits flush in an Intel NUC). I didn't have time to troubleshoot what's causing this problem, other than to plug the USB stick into another computer to verify the stick is good and hasn't been corrupted. It's possibly related the mq-blk bug in 4.19.0 through 4.19.7 - so I've since upgraded to 4.19.8 which as those patches, but I haven't had a chance to retest.
I doubt this is bfq fault, as bfq is rather frequently used for USB too. Probably bfq has not been activated at all.
Paolo
-- Chris Murphy _______________________________________________ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
[stan] So, how do I get a fedora kernel to run BFQ?
[paolo] AFAIK, the only way is to set bfq manually or through, e.g., dev rules, but YMMV.
I used a udev rule. I linked to instructions in my original post, the link is:
https://unix.stackexchange.com/questions/375600/how-to-enable-and-use-the-bf...
I anticipated this question and tried writing a convenient script before I posted. But I ended up writing an Ansible role. Then I realised this was useful for me, but not so useful to post as a quick script for others.
https://github.com/sourcejedi/ansible-site/tree/master/roles/bfq
Regards Alan
On Thu, 13 Dec 2018 13:07:14 -0000 "Alan Jenkins" alan.christopher.jenkins@gmail.com wrote:
[stan] So, how do I get a fedora kernel to run BFQ?
[paolo] AFAIK, the only way is to set bfq manually or through, e.g., dev rules, but YMMV.
I used a udev rule. I linked to instructions in my original post, the link is:
https://unix.stackexchange.com/questions/375600/how-to-enable-and-use-the-bf...
I anticipated this question and tried writing a convenient script before I posted. But I ended up writing an Ansible role. Then I realised this was useful for me, but not so useful to post as a quick script for others.
https://github.com/sourcejedi/ansible-site/tree/master/roles/bfq
Thanks, I'll give this a whirl.
On 12/11/18 3:48 AM, Alan Jenkins wrote:
Hi
mattdm suggested the upcoming scheduler change should be discussed here.[1] I might not have enough time to talk details at the moment, but I noticed this is coming up relatively soon. This is what I understand so far about the decision.
Thanks for all the software, Alan
[1] https://unix.stackexchange.com/questions/483881/what-does-fedora-workstation...
## CFQ is scheduled for removal
Jens Axboe is planning to remove the CFQ I/O scheduler in 4.21. That is, CFQ is part of the "legacy" single-queue block layer, which is going to be removed for ease of maintenance.[2]
Axboes last comment on this timing was made *after* the fix for data corruption on MQ. I.e. the data corruption covered by the recent thread on this list.
*Obligatory disclaimer*. Read the paragraph above, and consider waiting for the next stable kernel update, before you test MQ (including BFQ) on your own machine :-).
[2] "It's definitely still going" - Jens Abxoe. https://bugzilla.kernel.org/show_bug.cgi?id=201685#c279
## The kernel wants us to choose our new default
For devices which have only one hardware queue, the new upstream default is mq-deadline. Going from CFQ to mq-deadline is a significant change. For example, the deadline scheduler does not support ionice.
The alternative to the MQ deadline scheduler will be BFQ.[3][4] Upstream discussed this, and the powers that be (mostly Axboe :-) are explicitly expecting us (downstreams) to make this decision.[5]
[3] BFQ: http://algo.ing.unimo.it/people/paolo/disk_sched/description.php
[4] https://unix.stackexchange.com/questions/375600/how-to-enable-and-use-the-bf...
[5] "I'd prefer if the distros would lead the way on this, as theyare the ones that will most likely see the most bug reports" - Jens Axboe, https://www.spinics.net/lists/linux-block/msg31062.html
## Arguments for (or against) BFQ?
Paolo Valente kindly wrote an informative response, which I will copy in to a separate message. The following is my very limited first impression.
Personally I lean towards BFQ by default. It appears nominated as the successor to CFQ, I think it's worthy as such, and it makes distinct improvements of it's own. I would recommend it with more confidence if I understood how the improvements work :-).
The deadline scheduler probably isn't a *complete* disaster. Ubuntu ran with the deadline scheduler for a while.[6] I haven't checked whether they changed the tuning knobs though!
RHEL7 defaults to CFQ for SATA drives. This is notable given that it recommends avoiding (or tuning) CFQ on basically any other server hardware (and specifically to avoid it on hardware RAID).[7]
I've tried BFQ on my laptops hard drive (not SSD). It has some associated tests for responsiveness (the "S" tests). I don't have a real-world feel for it, but I agree the test numbers are an impressive improvement over CFQ.
I don't have results for the S tests on the deadline scheduler. I did note the eponymous "deadline" for sync reads has a default of 500 ms.
The other test I have, is that "deadline" doesn't match CFQ's level of fairness for reads v.s. writes, even with the recent addition of WBT. Neither approached what I would actually call fairness. BFQ did.[8] This is due to BFQ's "compensations" for device writeback caching and NCQ. And allegedly for Linux writeback. These extra compensations are the part I don't understand, so far.[9]
[6] https://askubuntu.com/questions/784442/why-does-ubuntu-16-04-set-all-drive-i...
[7] RHEL7 IO schedulers:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...
[8] I tried to closely match the test from the cover letters from the WBT patch series. I don't have detailed statistics, but I believe deadline+WBT was less fair than CFQ. It was definitely not more fair.
https://unix.stackexchange.com/questions/483269/determine-the-specific-benef...
[9] https://groups.google.com/forum/#!topic/bfq-iosched/yjZDn_HnLIc
Thanks for starting this discussion. Based on this thread and other discussions, I'm going to see about enabling BFQ for 4.21. Once 4.21-rc1 comes out (i.e. almost all configs should be in) we can review the settings and see if anything needs to be adjusted. If I get some time before then (i.e. before I leave for holiday) I'll see if I can change the defaults.
Laura
On 13/12/2018, Laura Abbott labbott@redhat.com wrote:
Thanks for starting this discussion. Based on this thread and other discussions, I'm going to see about enabling BFQ for 4.21. Once 4.21-rc1 comes out (i.e. almost all configs should be in) we can review the settings and see if anything needs to be adjusted. If I get some time before then (i.e. before I leave for holiday) I'll see if I can change the defaults.
Laura
Are there any more up-to-date details on this? Specifically I am wondering if there will be any separate package providing udev rules, that people should be aware of, if they want to provide testing of 5.0 kernels for Fedora.
Thanks Alan
On Tue, 5 Mar 2019 14:07:15 +0000 Alan Jenkins alan.christopher.jenkins@gmail.com wrote:
On 13/12/2018, Laura Abbott labbott@redhat.com wrote:
Thanks for starting this discussion. Based on this thread and other discussions, I'm going to see about enabling BFQ for 4.21. Once 4.21-rc1 comes out (i.e. almost all configs should be in) we can review the settings and see if anything needs to be adjusted. If I get some time before then (i.e. before I leave for holiday) I'll see if I can change the defaults.
Laura
Are there any more up-to-date details on this? Specifically I am wondering if there will be any separate package providing udev rules, that people should be aware of, if they want to provide testing of 5.0 kernels for Fedora.
I've been using the 5.0 kernels locally compiled and optimized for my hardware with bfq enabled. I don't have multi thread hardware. The performance has been comparable to cfq, or slightly worse. Subjective. And there could be other software changes affecting this.
I've recently also started using the gcc kernel-stack plugin to clean stack on return from kernel calls, and didn't really notice any effect on performance in normal usage.
Il giorno 5 mar 2019, alle ore 17:18, stan upaitag@zoho.com ha scritto:
On Tue, 5 Mar 2019 14:07:15 +0000 Alan Jenkins alan.christopher.jenkins@gmail.com wrote:
On 13/12/2018, Laura Abbott labbott@redhat.com wrote:
Thanks for starting this discussion. Based on this thread and other discussions, I'm going to see about enabling BFQ for 4.21. Once 4.21-rc1 comes out (i.e. almost all configs should be in) we can review the settings and see if anything needs to be adjusted. If I get some time before then (i.e. before I leave for holiday) I'll see if I can change the defaults.
Laura
Are there any more up-to-date details on this? Specifically I am wondering if there will be any separate package providing udev rules, that people should be aware of, if they want to provide testing of 5.0 kernels for Fedora.
I've been using the 5.0 kernels locally compiled and optimized for my hardware with bfq enabled. I don't have multi thread hardware. The performance has been comparable to cfq, or slightly worse. Subjective.
I started testing 5.0 today too, with incredibly bad performance, on an old Intel Core i7-2760QM@2.40GHz. I found out the bottleneck to be the CPU (about 7x slowdown). After reporting this issue to some Linaro guys (in CC), I've been suggested a tiresome bisection w.r.t. 4.20 (4.20 which works with no issue). I hope someone will show up with the cause of the problem, relieving me of this burden :)
At any rate, let me take this opportunity for updating you guys on what happened in the last months.
First, server-side, I discovered that the techniques used to guarantee I/O bandwidth to clients, containers and virtual machines easily result in throughput losses of up to 90%! So I improved BFQ so as to make it an alternative solution that brings this loss down to just 10%. Full details in this very recent (today :) ) short article: http://ow.ly/vsrW50mBAGl
Second, PC-side, I've pushed new commits for the dev version of BFQ (I'll submit these commits for the production version, probably tomorrow; so they'll probably be all available from 5.2). These commits provide the following, measurable performance boost: - up to ~80% faster application start-up times in the presence of background workloads - ~150% throughput boost in one of the nastiest workloads for BFQ the one generated by dbench. The throughput is finally on pr with any other I/O scheduler, and most likely equal to the maximum possible throughput reachable with this test - elimination of the 18% loss of throughput occurring with only random reads, w.r.t. to none as I/O scheduler; there is no loss any more;
For any question, I'll be glad to answer, if I can, Paolo
And there could be other software changes affecting this.
I've recently also started using the gcc kernel-stack plugin to clean stack on return from kernel calls, and didn't really notice any effect on performance in normal usage. _______________________________________________ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
On Tue, 5 Mar 2019 17:42:12 +0100 Paolo Valente paolo.valente@linaro.org wrote:
At any rate, let me take this opportunity for updating you guys on what happened in the last months.
First, server-side, I discovered that the techniques used to guarantee I/O bandwidth to clients, containers and virtual machines easily result in throughput losses of up to 90%! So I improved BFQ so as to make it an alternative solution that brings this loss down to just 10%. Full details in this very recent (today :) ) short article: http://ow.ly/vsrW50mBAGl
Second, PC-side, I've pushed new commits for the dev version of BFQ (I'll submit these commits for the production version, probably tomorrow; so they'll probably be all available from 5.2). These commits provide the following, measurable performance boost:
- up to ~80% faster application start-up times in the presence of background workloads
- ~150% throughput boost in one of the nastiest workloads for BFQ the one generated by dbench. The throughput is finally on pr with any other I/O scheduler, and most likely equal to the maximum possible throughput reachable with this test
- elimination of the 18% loss of throughput occurring with only random reads, w.r.t. to none as I/O scheduler; there is no loss any more;
This sounds great! Thanks for the update. Looking forward to 5.2.
Il giorno 5 mar 2019, alle ore 21:13, stan upaitag@zoho.com ha scritto:
On Tue, 5 Mar 2019 17:42:12 +0100 Paolo Valente paolo.valente@linaro.org wrote:
At any rate, let me take this opportunity for updating you guys on what happened in the last months.
First, server-side, I discovered that the techniques used to guarantee I/O bandwidth to clients, containers and virtual machines easily result in throughput losses of up to 90%! So I improved BFQ so as to make it an alternative solution that brings this loss down to just 10%. Full details in this very recent (today :) ) short article: http://ow.ly/vsrW50mBAGl
Second, PC-side, I've pushed new commits for the dev version of BFQ (I'll submit these commits for the production version, probably tomorrow; so they'll probably be all available from 5.2). These commits provide the following, measurable performance boost:
- up to ~80% faster application start-up times in the presence of
background workloads
- ~150% throughput boost in one of the nastiest workloads for BFQ the
one generated by dbench. The throughput is finally on pr with any other I/O scheduler, and most likely equal to the maximum possible throughput reachable with this test
- elimination of the 18% loss of throughput occurring with only
random reads, w.r.t. to none as I/O scheduler; there is no loss any more;
This sounds great! Thanks for the update. Looking forward to 5.2.
If you are curious, here's the performance you will enjoy from 5.2, and, partially, already from 5.1 (tested through the dev version of bfq on top of a 4.19): https://algo.ing.unimo.it/people/paolo/disk_sched/results.php
Unless 5.2 will be as broken as 5.0 currently is! ;)
Thanks, Paolo
kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Il giorno 5 mar 2019, alle ore 17:42, Paolo Valente paolo.valente@linaro.org ha scritto:
Il giorno 5 mar 2019, alle ore 17:18, stan upaitag@zoho.com ha scritto:
On Tue, 5 Mar 2019 14:07:15 +0000 Alan Jenkins alan.christopher.jenkins@gmail.com wrote:
On 13/12/2018, Laura Abbott labbott@redhat.com wrote:
Thanks for starting this discussion. Based on this thread and other discussions, I'm going to see about enabling BFQ for 4.21. Once 4.21-rc1 comes out (i.e. almost all configs should be in) we can review the settings and see if anything needs to be adjusted. If I get some time before then (i.e. before I leave for holiday) I'll see if I can change the defaults.
Laura
Are there any more up-to-date details on this? Specifically I am wondering if there will be any separate package providing udev rules, that people should be aware of, if they want to provide testing of 5.0 kernels for Fedora.
I've been using the 5.0 kernels locally compiled and optimized for my hardware with bfq enabled. I don't have multi thread hardware. The performance has been comparable to cfq, or slightly worse. Subjective.
I started testing 5.0 today too, with incredibly bad performance, on an old Intel Core i7-2760QM@2.40GHz. I found out the bottleneck to be the CPU (about 7x slowdown). After reporting this issue to some Linaro guys (in CC), I've been suggested a tiresome bisection w.r.t. 4.20 (4.20 which works with no issue). I hope someone will show up with the cause of the problem, relieving me of this burden :)
At any rate, let me take this opportunity for updating you guys on what happened in the last months.
First, server-side, I discovered that the techniques used to guarantee I/O bandwidth to clients, containers and virtual machines easily result in throughput losses of up to 90%! So I improved BFQ so as to make it an alternative solution that brings this loss down to just 10%. Full details in this very recent (today :) ) short article: http://ow.ly/vsrW50mBAGl
One question, hoping that it is not off topic here. As a consequence of the above article, I've been asked what distros already use bfq, on Kubernetes forums: https://discuss.kubernetes.io/t/a-solution-to-the-severe-throughput-loss-cau...
What can I say about Fedora?
Thanks, Paolo
Second, PC-side, I've pushed new commits for the dev version of BFQ (I'll submit these commits for the production version, probably tomorrow; so they'll probably be all available from 5.2). These commits provide the following, measurable performance boost:
- up to ~80% faster application start-up times in the presence of
background workloads
- ~150% throughput boost in one of the nastiest workloads for BFQ the
one generated by dbench. The throughput is finally on pr with any other I/O scheduler, and most likely equal to the maximum possible throughput reachable with this test
- elimination of the 18% loss of throughput occurring with only
random reads, w.r.t. to none as I/O scheduler; there is no loss any more;
For any question, I'll be glad to answer, if I can, Paolo
And there could be other software changes affecting this.
I've recently also started using the gcc kernel-stack plugin to clean stack on return from kernel calls, and didn't really notice any effect on performance in normal usage. _______________________________________________ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
kernel@lists.fedoraproject.org