https://fedoraproject.org/wiki/Changes/LTOBuildImprovements
== Summary == Currently all packages that are not opted out of LTO include -ffat-lto-objects in their build flags. This proposal would remove -ffat-lto-objects from the default LTO flags and only use it for packages that actually need it.
== Owner == * Name: [[User:law | Jeff Law]] * Email: law@redhat.com
== Detailed Description == -ffat-lto-objects was added to the default LTO flags to ensure that any installed .o/.a files included actual compiled code rather than just LTO bytecodes (which are stripped after the install phase). However, that is wasteful from a compile-time standpoint as few packages actually install any .o/.a files.
This proposal would remove -ffat-lto-objects from the default LTO flags and packages that actually need the option would have to opt-in via an RPM macro in their .spec file. This should significantly improve build times for most packages in Fedora.
To ensure that we can identify packages that need the opt-in now and in the future, the plan is to pass to brp-strip-lto a flag indicating whether or not the package has opted into -ffat-lto-objects. If brp-strip-lto finds .o/.a files, but the package has not opted into -ffat-lto-objects, then brp-strip-lto would signal an error.
== Benefit to Fedora == The key benefit to Fedora is improved package build times and lower load on the builders.
== Scope == * Proposal owners: The feature owner (Jeff Law) will need to settle on a suitable RPM macro to indicate an opt-in to -ffat-lto-objects, implement the necessary tests in brp-strip-lto and opt-in the initial set of packages. This will be accomplished by doing the prototype implementation locally, building all the Fedora packages to generate the opt-in set. Committing the necessary opt-ins, then committing the necessary changes to the RPM macros.
* Other developers: There should be minimal work for other developers. The most likely scenarios where other developers will need to get involved would include: # Packages which are excluded from x86_64 builds and which need the opt-in will need the appropriate package owners to add the opt-in. # Packages which are FTBFS when the builds are run to find the set of packages that need to opt-in and which need to opt-in will need packager attention. # It is possible that the faster builds may trigger build failures in packages that have missing dependencies in their Makefiles. We saw a few of these during the initial LTO work and those packages were either fixed or -j parallelism removed. This work may expose more of these problems.
I expect these all to be relatively rare occurences, but with 9000+ packages in Fedora I wouldn't be surprised if we see a few of each of these issues.
* Release engineering: [https://pagure.io/releng/issues #Releng issue number] (a check of an impact with Release Engineering is needed) This should have no release engineering impacts. * Policies and guidelines: The packaging guidelines will need to be updated to document the new macro. * Trademark approval: N/A (not needed for this Change) * Alignment with Objectives: This proposal does not align with any current Fedora Objectives.
== Upgrade/compatibility impact == This change should have zero impact on upgrade/compatibility. In fact, the change should have no user visible impacts.
== How To Test == No special testing is needed. Any issues with this proposal will show up as FTBFS issues.
== User Experience == Users should see no changes to the user experience.
== Dependencies == Packages which need to opt-in to -ffat-lto-objects will need their .spec files updated to include the new macro.
== Contingency Plan == If this can not be completed by final development freeze, then the RPM macro changes would not be installed and the change could defer to Fedora 35. * Contingency mechanism: Proposal owner will only commit the RPM macro changes once the opt-in package set has been identified and opt-ins added to those package's spec files. So no special contingency mechanism should be needed * Contingency deadline: It is most beneficial to have this completed before the mass rebuild; however, the drop dead date should be beta freeze. * Blocks release? No * Blocks product? No
== Documentation == No upstream documentation. Packaging guidelines will need a minor update.
== Release Notes == I do not expect this change to require any release notes.
On Wed, Dec 30, 2020 at 7:54 PM Ben Cotton bcotton@redhat.com wrote:
https://fedoraproject.org/wiki/Changes/LTOBuildImprovements
== Summary == Currently all packages that are not opted out of LTO include -ffat-lto-objects in their build flags. This proposal would remove -ffat-lto-objects from the default LTO flags and only use it for packages that actually need it.
== Owner ==
- Name: [[User:law | Jeff Law]]
- Email: law@redhat.com
== Detailed Description == -ffat-lto-objects was added to the default LTO flags to ensure that any installed .o/.a files included actual compiled code rather than just LTO bytecodes (which are stripped after the install phase). However, that is wasteful from a compile-time standpoint as few packages actually install any .o/.a files.
This proposal would remove -ffat-lto-objects from the default LTO flags and packages that actually need the option would have to opt-in via an RPM macro in their .spec file. This should significantly improve build times for most packages in Fedora.
Does this mean that packages that are explicitly shipping a static library to the end user need to enable this macro to allow the installed static library to be usable by an end-user's compiler? If this is the case, then the packaging guidelines should be updated to reflect this.
To ensure that we can identify packages that need the opt-in now and in the future, the plan is to pass to brp-strip-lto a flag indicating whether or not the package has opted into -ffat-lto-objects. If brp-strip-lto finds .o/.a files, but the package has not opted into -ffat-lto-objects, then brp-strip-lto would signal an error.
== Benefit to Fedora == The key benefit to Fedora is improved package build times and lower load on the builders.
== Scope ==
- Proposal owners: The feature owner (Jeff Law) will need to settle on
a suitable RPM macro to indicate an opt-in to -ffat-lto-objects, implement the necessary tests in brp-strip-lto and opt-in the initial set of packages. This will be accomplished by doing the prototype implementation locally, building all the Fedora packages to generate the opt-in set. Committing the necessary opt-ins, then committing the necessary changes to the RPM macros.
- Other developers: There should be minimal work for other
developers. The most likely scenarios where other developers will need to get involved would include: # Packages which are excluded from x86_64 builds and which need the opt-in will need the appropriate package owners to add the opt-in. # Packages which are FTBFS when the builds are run to find the set of packages that need to opt-in and which need to opt-in will need packager attention. # It is possible that the faster builds may trigger build failures in packages that have missing dependencies in their Makefiles. We saw a few of these during the initial LTO work and those packages were either fixed or -j parallelism removed. This work may expose more of these problems.
I expect these all to be relatively rare occurences, but with 9000+ packages in Fedora I wouldn't be surprised if we see a few of each of these issues.
- Release engineering: [https://pagure.io/releng/issues #Releng issue
number] (a check of an impact with Release Engineering is needed) This should have no release engineering impacts.
- Policies and guidelines: The packaging guidelines will need to be
updated to document the new macro.
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives: This proposal does not align with any
current Fedora Objectives.
== Upgrade/compatibility impact == This change should have zero impact on upgrade/compatibility. In fact, the change should have no user visible impacts.
== How To Test == No special testing is needed. Any issues with this proposal will show up as FTBFS issues.
== User Experience == Users should see no changes to the user experience.
== Dependencies == Packages which need to opt-in to -ffat-lto-objects will need their .spec files updated to include the new macro.
== Contingency Plan == If this can not be completed by final development freeze, then the RPM macro changes would not be installed and the change could defer to Fedora 35.
- Contingency mechanism: Proposal owner will only commit the RPM macro
changes once the opt-in package set has been identified and opt-ins added to those package's spec files. So no special contingency mechanism should be needed
- Contingency deadline: It is most beneficial to have this completed
before the mass rebuild; however, the drop dead date should be beta freeze.
- Blocks release? No
- Blocks product? No
== Documentation == No upstream documentation. Packaging guidelines will need a minor update.
== Release Notes == I do not expect this change to require any release notes.
-- Ben Cotton He / Him / His Senior Program Manager, Fedora & CentOS Stream Red Hat TZ=America/Indiana/Indianapolis _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On 12/30/20 3:48 PM, Ian McInerney wrote:
On Wed, Dec 30, 2020 at 7:54 PM Ben Cotton <bcotton@redhat.com mailto:bcotton@redhat.com> wrote:
https://fedoraproject.org/wiki/Changes/LTOBuildImprovements <https://fedoraproject.org/wiki/Changes/LTOBuildImprovements> == Summary == Currently all packages that are not opted out of LTO include -ffat-lto-objects in their build flags. This proposal would remove -ffat-lto-objects from the default LTO flags and only use it for packages that actually need it. == Owner == * Name: [[User:law | Jeff Law]] * Email: law@redhat.com <mailto:law@redhat.com> == Detailed Description == -ffat-lto-objects was added to the default LTO flags to ensure that any installed .o/.a files included actual compiled code rather than just LTO bytecodes (which are stripped after the install phase). However, that is wasteful from a compile-time standpoint as few packages actually install any .o/.a files. This proposal would remove -ffat-lto-objects from the default LTO flags and packages that actually need the option would have to opt-in via an RPM macro in their .spec file. This should significantly improve build times for most packages in Fedora.Does this mean that packages that are explicitly shipping a static library to the end user need to enable this macro to allow the installed static library to be usable by an end-user's compiler? If this is the case, then the packaging guidelines should be updated to reflect this.
Yes and the change request reflects that an update to the packaging guidelines is necessary.
Jeff
On Sat, Jan 2, 2021 at 5:33 PM Jeff Law law@redhat.com wrote:
On 12/30/20 3:48 PM, Ian McInerney wrote:
On Wed, Dec 30, 2020 at 7:54 PM Ben Cotton <bcotton@redhat.com mailto:bcotton@redhat.com> wrote:
https://fedoraproject.org/wiki/Changes/LTOBuildImprovements <https://fedoraproject.org/wiki/Changes/LTOBuildImprovements> == Summary == Currently all packages that are not opted out of LTO include -ffat-lto-objects in their build flags. This proposal would remove -ffat-lto-objects from the default LTO flags and only use it for packages that actually need it. == Owner == * Name: [[User:law | Jeff Law]] * Email: law@redhat.com <mailto:law@redhat.com> == Detailed Description == -ffat-lto-objects was added to the default LTO flags to ensure that any installed .o/.a files included actual compiled code rather than just LTO bytecodes (which are stripped after the install phase). However, that is wasteful from a compile-time standpoint as few packages actually install any .o/.a files. This proposal would remove -ffat-lto-objects from the default LTO flags and packages that actually need the option would have to opt-in via an RPM macro in their .spec file. This should significantly improve build times for most packages in Fedora.Does this mean that packages that are explicitly shipping a static library to the end user need to enable this macro to allow the installed static library to be usable by an end-user's compiler? If this is the case, then the packaging guidelines should be updated to reflect this.
Yes and the change request reflects that an update to the packaging guidelines is necessary.
The proposal only says the macro will be documented, but doesn't go into any more detail. I think it would be beneficial if the proposal mentioned the macro would become required for static library packages and that policy needs to be added - since that is adding a new MUST section to the packaging guidelines.
-Ian
On 1/2/21 10:39 AM, Ian McInerney wrote:
On Sat, Jan 2, 2021 at 5:33 PM Jeff Law <law@redhat.com mailto:law@redhat.com> wrote:
On 12/30/20 3:48 PM, Ian McInerney wrote: > > > On Wed, Dec 30, 2020 at 7:54 PM Ben Cotton <bcotton@redhat.com <mailto:bcotton@redhat.com> > <mailto:bcotton@redhat.com <mailto:bcotton@redhat.com>>> wrote: > > https://fedoraproject.org/wiki/Changes/LTOBuildImprovements <https://fedoraproject.org/wiki/Changes/LTOBuildImprovements> > <https://fedoraproject.org/wiki/Changes/LTOBuildImprovements <https://fedoraproject.org/wiki/Changes/LTOBuildImprovements>> > > > == Summary == > Currently all packages that are not opted out of LTO include > -ffat-lto-objects in their build flags. This proposal would remove > -ffat-lto-objects from the default LTO flags and only use it for > packages that actually need it. > > == Owner == > * Name: [[User:law | Jeff Law]] > * Email: law@redhat.com <mailto:law@redhat.com> <mailto:law@redhat.com <mailto:law@redhat.com>> > > > == Detailed Description == > -ffat-lto-objects was added to the default LTO flags to ensure that > any installed .o/.a files included actual compiled code rather than > just LTO bytecodes (which are stripped after the install phase). > However, that is wasteful from a compile-time standpoint as few > packages actually install any .o/.a files. > > This proposal would remove -ffat-lto-objects from the default LTO > flags and packages that actually need the option would have to opt-in > via an RPM macro in their .spec file. This should significantly > improve build times for most packages in Fedora. > > > Does this mean that packages that are explicitly shipping a static > library to the end user need to enable this macro to allow the > installed static library to be usable by an end-user's compiler? If > this is the case, then the packaging guidelines should be updated to > reflect this. Yes and the change request reflects that an update to the packaging guidelines is necessary.The proposal only says the macro will be documented, but doesn't go into any more detail. I think it would be beneficial if the proposal mentioned the macro would become required for static library packages and that policy needs to be added - since that is adding a new MUST section to the packaging guidelines.
ACK. I'll add some additional text to the proposal.
jeff
* Ben Cotton:
To ensure that we can identify packages that need the opt-in now and in the future, the plan is to pass to brp-strip-lto a flag indicating whether or not the package has opted into -ffat-lto-objects. If brp-strip-lto finds .o/.a files, but the package has not opted into -ffat-lto-objects, then brp-strip-lto would signal an error.
And presumably fail the build?
A lot of the existing RPM post-processing steps detect, report, and ignore errors because the generated RPM package might still be partially useful.
Thanks, Florian
On 1/2/21 3:13 AM, Florian Weimer wrote:
- Ben Cotton:
To ensure that we can identify packages that need the opt-in now and in the future, the plan is to pass to brp-strip-lto a flag indicating whether or not the package has opted into -ffat-lto-objects. If brp-strip-lto finds .o/.a files, but the package has not opted into -ffat-lto-objects, then brp-strip-lto would signal an error.
And presumably fail the build?
Yes. The point here is to ensure that if the build installs .o/.a files that it must have opted into -ffat-lto-objects so that we don't end up with useless .o/.a files in our binary RPM packages.
A lot of the existing RPM post-processing steps detect, report, and ignore errors because the generated RPM package might still be partially useful.
True, but ignoring the error in this case runs the very real risk that a package could install a .o/.a file with no code/symbols. That in turn can cause downstream FTBFS errors in other packages and all kinds of headaches on developer systems. Failing the build here seems much safer.
Contrast to ignoring a dwz error. The resulting binary RPMs are still very much usable, the debuginfo packages are just larger than is strictly necessary.
jeff
* Jeff Law:
On 1/2/21 3:13 AM, Florian Weimer wrote:
- Ben Cotton:
To ensure that we can identify packages that need the opt-in now and in the future, the plan is to pass to brp-strip-lto a flag indicating whether or not the package has opted into -ffat-lto-objects. If brp-strip-lto finds .o/.a files, but the package has not opted into -ffat-lto-objects, then brp-strip-lto would signal an error.
And presumably fail the build?
Yes. The point here is to ensure that if the build installs .o/.a files that it must have opted into -ffat-lto-objects so that we don't end up with useless .o/.a files in our binary RPM packages.
Makes sense.
A lot of the existing RPM post-processing steps detect, report, and ignore errors because the generated RPM package might still be partially useful.
True, but ignoring the error in this case runs the very real risk that a package could install a .o/.a file with no code/symbols. That in turn can cause downstream FTBFS errors in other packages and all kinds of headaches on developer systems. Failing the build here seems much safer.
Contrast to ignoring a dwz error. The resulting binary RPMs are still very much usable, the debuginfo packages are just larger than is strictly necessary.
The downside is that toolchain bugs tend to go unnoticed and aren't fixed as a result.
Thanks, Florian
On 1/4/21 2:27 AM, Florian Weimer wrote:
A lot of the existing RPM post-processing steps detect, report, and ignore errors because the generated RPM package might still be partially useful.
True, but ignoring the error in this case runs the very real risk that a package could install a .o/.a file with no code/symbols. That in turn can cause downstream FTBFS errors in other packages and all kinds of headaches on developer systems. Failing the build here seems much safer.
Contrast to ignoring a dwz error. The resulting binary RPMs are still very much usable, the debuginfo packages are just larger than is strictly necessary.
The downside is that toolchain bugs tend to go unnoticed and aren't fixed as a result.
Right. And I think that's been a real problem. I really dislike ignoring errors :(
jeff
On 12/30/20 11:52 AM, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/LTOBuildImprovements
== Summary == Currently all packages that are not opted out of LTO include -ffat-lto-objects in their build flags. This proposal would remove -ffat-lto-objects from the default LTO flags and only use it for packages that actually need it.
== Owner ==
- Name: [[User:law | Jeff Law]]
- Email: law@redhat.com
== Detailed Description == -ffat-lto-objects was added to the default LTO flags to ensure that any installed .o/.a files included actual compiled code rather than just LTO bytecodes (which are stripped after the install phase). However, that is wasteful from a compile-time standpoint as few packages actually install any .o/.a files.
This proposal would remove -ffat-lto-objects from the default LTO flags and packages that actually need the option would have to opt-in via an RPM macro in their .spec file. This should significantly improve build times for most packages in Fedora.
What is this macro going to be called? I would like to get an early start on updating my packages.
-Tom
To ensure that we can identify packages that need the opt-in now and in the future, the plan is to pass to brp-strip-lto a flag indicating whether or not the package has opted into -ffat-lto-objects. If brp-strip-lto finds .o/.a files, but the package has not opted into -ffat-lto-objects, then brp-strip-lto would signal an error.
== Benefit to Fedora == The key benefit to Fedora is improved package build times and lower load on the builders.
== Scope ==
- Proposal owners: The feature owner (Jeff Law) will need to settle on
a suitable RPM macro to indicate an opt-in to -ffat-lto-objects, implement the necessary tests in brp-strip-lto and opt-in the initial set of packages. This will be accomplished by doing the prototype implementation locally, building all the Fedora packages to generate the opt-in set. Committing the necessary opt-ins, then committing the necessary changes to the RPM macros.
- Other developers: There should be minimal work for other
developers. The most likely scenarios where other developers will need to get involved would include: # Packages which are excluded from x86_64 builds and which need the opt-in will need the appropriate package owners to add the opt-in. # Packages which are FTBFS when the builds are run to find the set of packages that need to opt-in and which need to opt-in will need packager attention. # It is possible that the faster builds may trigger build failures in packages that have missing dependencies in their Makefiles. We saw a few of these during the initial LTO work and those packages were either fixed or -j parallelism removed. This work may expose more of these problems.
I expect these all to be relatively rare occurences, but with 9000+ packages in Fedora I wouldn't be surprised if we see a few of each of these issues.
- Release engineering: [https://pagure.io/releng/issues #Releng issue
number] (a check of an impact with Release Engineering is needed) This should have no release engineering impacts.
- Policies and guidelines: The packaging guidelines will need to be
updated to document the new macro.
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives: This proposal does not align with any
current Fedora Objectives.
== Upgrade/compatibility impact == This change should have zero impact on upgrade/compatibility. In fact, the change should have no user visible impacts.
== How To Test == No special testing is needed. Any issues with this proposal will show up as FTBFS issues.
== User Experience == Users should see no changes to the user experience.
== Dependencies == Packages which need to opt-in to -ffat-lto-objects will need their .spec files updated to include the new macro.
== Contingency Plan == If this can not be completed by final development freeze, then the RPM macro changes would not be installed and the change could defer to Fedora 35.
- Contingency mechanism: Proposal owner will only commit the RPM macro
changes once the opt-in package set has been identified and opt-ins added to those package's spec files. So no special contingency mechanism should be needed
- Contingency deadline: It is most beneficial to have this completed
before the mass rebuild; however, the drop dead date should be beta freeze.
- Blocks release? No
- Blocks product? No
== Documentation == No upstream documentation. Packaging guidelines will need a minor update.
== Release Notes == I do not expect this change to require any release notes.
On 1/4/21 12:22 PM, Tom Stellard wrote:
On 12/30/20 11:52 AM, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/LTOBuildImprovements
== Summary == Currently all packages that are not opted out of LTO include -ffat-lto-objects in their build flags. This proposal would remove -ffat-lto-objects from the default LTO flags and only use it for packages that actually need it.
== Owner ==
- Name: [[User:law | Jeff Law]]
- Email: law@redhat.com
== Detailed Description == -ffat-lto-objects was added to the default LTO flags to ensure that any installed .o/.a files included actual compiled code rather than just LTO bytecodes (which are stripped after the install phase). However, that is wasteful from a compile-time standpoint as few packages actually install any .o/.a files.
This proposal would remove -ffat-lto-objects from the default LTO flags and packages that actually need the option would have to opt-in via an RPM macro in their .spec file. This should significantly improve build times for most packages in Fedora.
What is this macro going to be called? I would like to get an early start on updating my packages.
No idea. I'm open to suggestions.
IIRC we had kicked around the idea of having the clang/llvm path instantiate the LTO bytecodes in .o/.a files, which would avoid these issues for packages using clang/llvm. Is that something we still want to pursue? I vaguely recall discussing this with David B. and he came up with a reason why that needed to happen before brp-strip-lto, but I can't remember any of the details at this point.
Jeff
On 1/4/21 11:28 AM, Jeff Law wrote:
On 1/4/21 12:22 PM, Tom Stellard wrote:
On 12/30/20 11:52 AM, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/LTOBuildImprovements
== Summary == Currently all packages that are not opted out of LTO include -ffat-lto-objects in their build flags. This proposal would remove -ffat-lto-objects from the default LTO flags and only use it for packages that actually need it.
== Owner ==
- Name: [[User:law | Jeff Law]]
- Email: law@redhat.com
== Detailed Description == -ffat-lto-objects was added to the default LTO flags to ensure that any installed .o/.a files included actual compiled code rather than just LTO bytecodes (which are stripped after the install phase). However, that is wasteful from a compile-time standpoint as few packages actually install any .o/.a files.
This proposal would remove -ffat-lto-objects from the default LTO flags and packages that actually need the option would have to opt-in via an RPM macro in their .spec file. This should significantly improve build times for most packages in Fedora.
What is this macro going to be called? I would like to get an early start on updating my packages.
No idea. I'm open to suggestions.
IIRC we had kicked around the idea of having the clang/llvm path instantiate the LTO bytecodes in .o/.a files, which would avoid these issues for packages using clang/llvm. Is that something we still want to pursue? I vaguely recall discussing this with David B. and he came up with a reason why that needed to happen before brp-strip-lto, but I can't remember any of the details at this point.
Yes, David is still working on a solution for clang/llvm. I'm not sure, though, at what point in the RPM build process the LTO bytecode will get codegen'd.
-Tom
Jeff _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mon, Jan 4, 2021 at 7:28 PM Jeff Law law@redhat.com wrote:
snip...
What is this macro going to be called? I would like to get an early start on updating my packages.
No idea. I'm open to suggestions.
Any update on the name of this macro/its implementation? I am working on revising a spec file that has a static library included and if this macro is available to use already I would like to add it as long as I am working in the spec.
-Ian
On 2/2/21 5:03 PM, Ian McInerney wrote:
On Mon, Jan 4, 2021 at 7:28 PM Jeff Law <law@redhat.com mailto:law@redhat.com> wrote:
snip... > > What is this macro going to be called? I would like to get an early > start on updating my packages. No idea. I'm open to suggestions.Any update on the name of this macro/its implementation? I am working on revising a spec file that has a static library included and if this macro is available to use already I would like to add it as long as I am working in the spec.
No movement yet. I had to push this (and other) stuff down to deal with a variety of F34 firedrills. I expect to come back to it during the F35 cycle.
jeff
Just wonder, have somebody inherited this feature after Jeff? And is anybody actually working on the LTO?
~~~
$ grep -R lto | grep -E 'global|define' | grep nil | wc -l 190
~~~
Vít
Dne 30. 12. 20 v 20:52 Ben Cotton napsal(a):
https://fedoraproject.org/wiki/Changes/LTOBuildImprovements
== Summary == Currently all packages that are not opted out of LTO include -ffat-lto-objects in their build flags. This proposal would remove -ffat-lto-objects from the default LTO flags and only use it for packages that actually need it.
== Owner ==
- Name: [[User:law | Jeff Law]]
- Email: law@redhat.com
== Detailed Description == -ffat-lto-objects was added to the default LTO flags to ensure that any installed .o/.a files included actual compiled code rather than just LTO bytecodes (which are stripped after the install phase). However, that is wasteful from a compile-time standpoint as few packages actually install any .o/.a files.
This proposal would remove -ffat-lto-objects from the default LTO flags and packages that actually need the option would have to opt-in via an RPM macro in their .spec file. This should significantly improve build times for most packages in Fedora.
To ensure that we can identify packages that need the opt-in now and in the future, the plan is to pass to brp-strip-lto a flag indicating whether or not the package has opted into -ffat-lto-objects. If brp-strip-lto finds .o/.a files, but the package has not opted into -ffat-lto-objects, then brp-strip-lto would signal an error.
== Benefit to Fedora == The key benefit to Fedora is improved package build times and lower load on the builders.
== Scope ==
- Proposal owners: The feature owner (Jeff Law) will need to settle on
a suitable RPM macro to indicate an opt-in to -ffat-lto-objects, implement the necessary tests in brp-strip-lto and opt-in the initial set of packages. This will be accomplished by doing the prototype implementation locally, building all the Fedora packages to generate the opt-in set. Committing the necessary opt-ins, then committing the necessary changes to the RPM macros.
- Other developers: There should be minimal work for other
developers. The most likely scenarios where other developers will need to get involved would include: # Packages which are excluded from x86_64 builds and which need the opt-in will need the appropriate package owners to add the opt-in. # Packages which are FTBFS when the builds are run to find the set of packages that need to opt-in and which need to opt-in will need packager attention. # It is possible that the faster builds may trigger build failures in packages that have missing dependencies in their Makefiles. We saw a few of these during the initial LTO work and those packages were either fixed or -j parallelism removed. This work may expose more of these problems.
I expect these all to be relatively rare occurences, but with 9000+ packages in Fedora I wouldn't be surprised if we see a few of each of these issues.
- Release engineering: [https://pagure.io/releng/issues #Releng issue
number] (a check of an impact with Release Engineering is needed) This should have no release engineering impacts.
- Policies and guidelines: The packaging guidelines will need to be
updated to document the new macro.
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives: This proposal does not align with any
current Fedora Objectives.
== Upgrade/compatibility impact == This change should have zero impact on upgrade/compatibility. In fact, the change should have no user visible impacts.
== How To Test == No special testing is needed. Any issues with this proposal will show up as FTBFS issues.
== User Experience == Users should see no changes to the user experience.
== Dependencies == Packages which need to opt-in to -ffat-lto-objects will need their .spec files updated to include the new macro.
== Contingency Plan == If this can not be completed by final development freeze, then the RPM macro changes would not be installed and the change could defer to Fedora 35.
- Contingency mechanism: Proposal owner will only commit the RPM macro
changes once the opt-in package set has been identified and opt-ins added to those package's spec files. So no special contingency mechanism should be needed
- Contingency deadline: It is most beneficial to have this completed
before the mass rebuild; however, the drop dead date should be beta freeze.
- Blocks release? No
- Blocks product? No
== Documentation == No upstream documentation. Packaging guidelines will need a minor update.
== Release Notes == I do not expect this change to require any release notes.
On 8/12/2021 10:09 AM, Vít Ondruch wrote:
Just wonder, have somebody inherited this feature after Jeff? And is anybody actually working on the LTO?
$ grep -R lto | grep -E 'global|define' | grep nil | wc -l 190
I don't think anyone is actively working on reducing that set. But that to me would be quite low priority.
Addressing the -ffat-lto-objects would seem to be more important as it would significantly reduce build times. I did a test run a while back and identified all the packages that would need to get fixed, but changed jobs before I could start fixing them.
We're doing some light Fedora work here, which I expect to ramp up again in several months. Reducing build times is something I can make a case to mgt for as it'll affect some of the work we're doing rather significantly.
Jeff
On Thu, Aug 12 2021 at 11:41:12 AM -0600, Jeff Law jeffreyalaw@gmail.com wrote:
I don't think anyone is actively working on reducing that set. But that to me would be quite low priority.
Actually Marek Polacek has been reporting bugs! See e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1990111 for a recent success case where I was able to get LTO working for WebKitGTK thanks to some investigation by Jakub Jelinek.
Michael
devel@lists.stg.fedoraproject.org
-
Ben Cotton -
Florian Weimer -
Ian McInerney -
Jeff Law -
Jeff Law -
Michael Catanzaro -
Tom Stellard -
Vít Ondruch