Note from Change Wrangler: This Change Proposal requires mass rebuild. However, two weeks ago (June 19th), we have already passed the deadline for Change proposals requiring mass rebuild. I will leave the decision whether this Change proposal is accepted or not to RelEng and FESCo teams.
= Proposed System Wide Change: Remove Excessive Linking = https://fedoraproject.org/wiki/Changes/RemoveExcessiveLinking
Owner(s): * Igor Gnatenko <ignatenkobrain at fedoraproject dot org> * Neal Gompa <ngompa13 at gmail dot com>
Pass "--as-needed" flag the linker through default system-wide LDFLAGS.
== Detailed description == The flag ("--as-needed") tells the linker to link in the produced binary only the libraries containing symbols actually used by the binary itself. This binary can be either a final executale or another library. The use of the "--as-needed" flag allows the linker to avoid linking extra libraries in a binary. This not only improves startup times (as the loader does not have to load all the libraries for every step) but might avoid the full initialization of big frameworks.
== Scope == * Proposal owners: Add "-Wl,--as-needed" into RPM_LD_FLAGS (in redhat-rpm-config).
* Other developers: Nothing should break, but immediate work-around would be to disable this flag (will be provided in redhat-rpm-config) and fix real issue later.
* Release engineering: #7604 [https://pagure.io/releng/issue/7604] (mass rebuild is desired after this change).
** List of deliverables: N/A (not a System Wide Change)
* Policies and guidelines: Add information how to turn it off (TODO link to FPC ticket).
* Trademark approval: N/A (not needed for this Change)
On Tue, Jul 3, 2018 at 4:23 AM Jan Kurik jkurik@redhat.com wrote:
Note from Change Wrangler: This Change Proposal requires mass rebuild. However, two weeks ago (June 19th), we have already passed the deadline for Change proposals requiring mass rebuild. I will leave the decision whether this Change proposal is accepted or not to RelEng and FESCo teams.
= Proposed System Wide Change: Remove Excessive Linking = https://fedoraproject.org/wiki/Changes/RemoveExcessiveLinking
Owner(s):
- Igor Gnatenko <ignatenkobrain at fedoraproject dot org>
- Neal Gompa <ngompa13 at gmail dot com>
Pass "--as-needed" flag the linker through default system-wide LDFLAGS.
== Detailed description == The flag ("--as-needed") tells the linker to link in the produced binary only the libraries containing symbols actually used by the binary itself. This binary can be either a final executale or another library. The use of the "--as-needed" flag allows the linker to avoid linking extra libraries in a binary. This not only improves startup times (as the loader does not have to load all the libraries for every step) but might avoid the full initialization of big frameworks.
== Scope ==
- Proposal owners:
Add "-Wl,--as-needed" into RPM_LD_FLAGS (in redhat-rpm-config).
- Other developers:
Nothing should break, but immediate work-around would be to disable this flag (will be provided in redhat-rpm-config) and fix real issue later.
- Release engineering:
#7604 [https://pagure.io/releng/issue/7604] (mass rebuild is desired after this change).
** List of deliverables: N/A (not a System Wide Change)
- Policies and guidelines:
Add information how to turn it off (TODO link to FPC ticket).
- Trademark approval:
N/A (not needed for this Change)
Jan Kuřík JBoss EAP Program Manager Red Hat Czech s.r.o., Purkynova 99/71 https://maps.google.com/?q=Purkynova+99/71&entry=gmail&source=g, 612 45 Brno, Czech Republic
Changes that require or "desire" a mass-rebuild should have been submitted by June 19th.
I can't find the reference offhand, but I feel like I remember that FESCo has ruled in the past that any changes to the default buildroot or compiler flags must be implemented as part of a mass-rebuild, to catch the fallout all at once. So with my FESCo hat on, I'm -1 to this change for F29, but I'm a strong +1 to implementing it for F30 in Rawhide immediately after we branch.
On Tue, Jul 03, 2018 at 10:21:42AM +0200, Jan Kurik wrote:
The use of the "--as-needed" flag allows the linker to avoid linking extra libraries in a binary. This not only improves startup times (as the loader does not have to load all the libraries for every step) but might avoid the full initialization of big frameworks.
This might reduce auto-generated dependencies, too, right? Generally that's a good thing, but of course there's always the off chance that something that someone expects to be there because it was always pulled into an install by a dep now won't be....
On Sat, Jul 7, 2018 at 5:18 AM Matthew Miller mattdm@fedoraproject.org wrote:
On Tue, Jul 03, 2018 at 10:21:42AM +0200, Jan Kurik wrote:
The use of the "--as-needed" flag allows the linker to avoid linking extra libraries in a binary. This not only improves startup times (as the loader does not have to load all the libraries for every step) but might avoid the full initialization of big frameworks.
This might reduce auto-generated dependencies, too, right? Generally that's a good thing, but of course there's always the off chance that something that someone expects to be there because it was always pulled into an install by a dep now won't be....
This has actually been the default in Mageia for quite a long time, since the times of Mandriva. We even have a wiki page describing the problems of overlinking[1].
[1]: https://wiki.mageia.org/en/Overlinking_issues_in_packaging
On Sat, Jul 7, 2018 at 11:27 AM Matthew Miller mattdm@fedoraproject.org wrote:
On Tue, Jul 03, 2018 at 10:21:42AM +0200, Jan Kurik wrote:
The use of the "--as-needed" flag allows the linker to avoid linking extra libraries in a binary. This not only improves startup times (as the loader does not have to load all the libraries for every step) but might avoid the full initialization of big frameworks.
This might reduce auto-generated dependencies, too, right? Generally that's a good thing, but of course there's always the off chance that something that someone expects to be there because it was always pulled into an install by a dep now won't be....
Yes, RPM generates dependencies based on what binary is linked against.
With spec below, change produces this result: ⋊> ~/r/SPECS rpm -qpR /home/brain/rpmbuild/RPMS/x86_64/hello-1-1.fc29.no_as_needed.x86_64.rpm libatk-1.0.so.0()(64bit) libc.so.6()(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libcairo-gobject.so.2()(64bit) libcairo.so.2()(64bit) libfribidi.so.0()(64bit) libgdk-3.so.0()(64bit) libgdk_pixbuf-2.0.so.0()(64bit) libgio-2.0.so.0()(64bit) libglib-2.0.so.0()(64bit) libgobject-2.0.so.0()(64bit) libgtk-3.so.0()(64bit) libpango-1.0.so.0()(64bit) libpangocairo-1.0.so.0()(64bit) libpthread.so.0()(64bit) rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(PayloadIsXz) <= 5.2-1 rtld(GNU_HASH)
⋊> ~/r/SPECS rpm -qpR /home/brain/rpmbuild/RPMS/x86_64/hello-1-1.fc29.as_needed.x86_64.rpm libc.so.6()(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libgtk-3.so.0()(64bit) libpthread.so.0()(64bit) rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(PayloadIsXz) <= 5.2-1 rtld(GNU_HASH)
And here is the spec: Name: hello Version: 1 Release: 1%{?dist} Summary: Hello
License: Public Domain URL: https://fedoraproject.org
BuildRequires: gcc BuildRequires: pkgconfig(gtk+-3.0)
%description %{summary}.
%prep %setup -c -T cat > hello.c << EOF #include <stdio.h> #include <gtk/gtk.h> int main (int argc, char *argv[]) { gtk_init(&argc, &argv);
printf("Hello!\n");
return 0; } EOF
%build gcc %{build_cflags} $(pkg-config --cflags gtk+-3.0) hello.c -o hello %{build_ldflags} $(pkg-config --libs gtk+-3.0)
%install install -Dpm0755 -t %{buildroot}%{_bindir} hello
%files %{_bindir}/hello
%changelog
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
On Tue, Jul 03, 2018 at 10:21:42AM +0200, Jan Kurik wrote:
- Other developers:
Nothing should break, but immediate work-around would be to disable this flag (will be provided in redhat-rpm-config) and fix real issue later.
That is not true, this option is quite dangerous and breaks a lot of stuff, ask SUSE people or other distros on how many times they need to add workarounds for this.
Jakub
On Tue, Jul 03, 2018 at 10:21:42AM +0200, Jan Kurik wrote:
Note from Change Wrangler: This Change Proposal requires mass rebuild. However, two weeks ago (June 19th), we have already passed the deadline for Change proposals requiring mass rebuild. I will leave the decision whether this Change proposal is accepted or not to RelEng and FESCo teams.
= Proposed System Wide Change: Remove Excessive Linking = https://fedoraproject.org/wiki/Changes/RemoveExcessiveLinking
Owner(s):
- Igor Gnatenko <ignatenkobrain at fedoraproject dot org>
- Neal Gompa <ngompa13 at gmail dot com>
Pass "--as-needed" flag the linker through default system-wide LDFLAGS.
== Detailed description == The flag ("--as-needed") tells the linker to link in the produced binary only the libraries containing symbols actually used by the binary itself. This binary can be either a final executale or another library. The use of the "--as-needed" flag allows the linker to avoid linking extra libraries in a binary. This not only improves startup times (as the loader does not have to load all the libraries for every step) but might avoid the full initialization of big frameworks.
Well, --as-needed is workaround and nothing else. The real problem is mess in makefiles and .pc (pkg-config) files.
It would be better to use --as-needed for testing purpose only, and ask maintainers why the result with --as-needed is different.
Karel
Karel Zak wrote on Wed, Aug 01, 2018 at 01:17:29PM +0200:
Well, --as-needed is workaround and nothing else. The real problem is mess in makefiles and .pc (pkg-config) files.
It would be better to use --as-needed for testing purpose only, and ask maintainers why the result with --as-needed is different.
Wearing a lib developer hat, I don't see how you can make a .pc that doesn't overlink if you provide something a bit entangled with other libs. The problem is that if your headers use any sub-lib type you need to add that lib in Requires: so that --cflags will pull that lib's include path; and that will in turn add that sub-lib to the linked libs even if the program likely doesn't care about it (either because they didn't even use that part of your lib, or because all uses of that lib will be done through your own anyway so it wasn't required in the first place)
This isn't obvious if you only support systems like fedora where most packages include dirs are standard to /usr/include but if you want to start supporting people working with dependencies in subdirs or distros like nixos that will have one include directory per dependency, you really need to put every external headers your depend on as 'Requires'.
Unless you can tell pkgconfig "take all of these dependencies for --cflags but not for --libs" I do not see how this can be improved, and that's where --as-needed is helpful.
It's not the ideal solution, but I don't have anything better right now, and pkg-config is by far not the worst solution to handling dependencies...
On Thu, Aug 02, 2018 at 11:45:00AM +0200, Dominique Martinet wrote:
Wearing a lib developer hat, I don't see how you can make a .pc that doesn't overlink if you provide something a bit entangled with other libs. The problem is that if your headers use any sub-lib type you need to add that lib in Requires: so that --cflags will pull that lib's include path; and that will in turn add that sub-lib to the linked libs even if the program likely doesn't care about it (either because they didn't even use that part of your lib, or because all uses of that lib will be done through your own anyway so it wasn't required in the first place)
Then it is clearly a task for pkg-config (and libtool) to handle it better. If the --libs it provides contains some mandatory and some optionally needed libraries, then it should differentiate between them and use -Wl,--push-state,--as-needed ... -Wl,--pop-state around those that are optionally needed. If all libraries from those tools are optional, perhaps it should use it always.
Changing the behavior of say -lpthread on the command line is a bad idea, many projects really expect it to mean that the mentioned library is linked in and if it no longer does, it causes silent breakage. Forcing users to do -Wl,--push-state,--no-as-needed ... -Wl,--pop-state whenever they really mean to link some library is too hostile.
Jakub
On Thu, Aug 2, 2018 at 4:57 AM, Jakub Jelinek jakub@redhat.com wrote:
Changing the behavior of say -lpthread on the command line is a bad idea, many projects really expect it to mean that the mentioned library is linked in and if it no longer does, it causes silent breakage. Forcing users to do -Wl,--push-state,--no-as-needed ... -Wl,--pop-state whenever they really mean to link some library is too hostile.
My understanding is that at least Debian and SUSE have defaulted to --as-needed for years, and probably more distros do as well... so presumably most problems should have been shaken out already. Is anyone familiar with the status of --as-needed in other distributions?
If Fedora and RHEL are the only major distros that are different, then that argues in favor of adopting this change, so that developers using Fedora don't accidentally write programs that are broken on other distributions. I know I've been stung by this in the past, when an application I developed on Fedora failed to link on Debian....
Michael
On Sat, Aug 4, 2018 at 12:53 PM mcatanzaro@gnome.org wrote:
On Thu, Aug 2, 2018 at 4:57 AM, Jakub Jelinek jakub@redhat.com wrote:
Changing the behavior of say -lpthread on the command line is a bad idea, many projects really expect it to mean that the mentioned library is linked in and if it no longer does, it causes silent breakage. Forcing users to do -Wl,--push-state,--no-as-needed ... -Wl,--pop-state whenever they really mean to link some library is too hostile.
My understanding is that at least Debian and SUSE have defaulted to --as-needed for years, and probably more distros do as well... so presumably most problems should have been shaken out already. Is anyone familiar with the status of --as-needed in other distributions?
If Fedora and RHEL are the only major distros that are different, then that argues in favor of adopting this change, so that developers using Fedora don't accidentally write programs that are broken on other distributions. I know I've been stung by this in the past, when an application I developed on Fedora failed to link on Debian....
Mageia/Mandriva has defaulted to --as-needed for many years (at least since 2010), and SUSE changed to this in openSUSE 13.1 (I think). Debian did it around the same time as SUSE, I believe.
Today, Fedora is the outlier for this in not having it switched on.
On Aug 2, 2018, at 2:57 AM, Jakub Jelinek jakub@redhat.com wrote:
On Thu, Aug 02, 2018 at 11:45:00AM +0200, Dominique Martinet wrote: Wearing a lib developer hat, I don't see how you can make a .pc that doesn't overlink if you provide something a bit entangled with other libs. The problem is that if your headers use any sub-lib type you need to add that lib in Requires: so that --cflags will pull that lib's include path; and that will in turn add that sub-lib to the linked libs even if the program likely doesn't care about it (either because they didn't even use that part of your lib, or because all uses of that lib will be done through your own anyway so it wasn't required in the first place)
Then it is clearly a task for pkg-config (and libtool) to handle it better. If the --libs it provides contains some mandatory and some optionally needed libraries, then it should differentiate between them and use -Wl,--push-state,--as-needed ... -Wl,--pop-state around those that are optionally needed. If all libraries from those tools are optional, perhaps it should use it always.
Changing the behavior of say -lpthread on the command line is a bad idea, many projects really expect it to mean that the mentioned library is linked in and if it no longer does, it causes silent breakage. Forcing users to do -Wl,--push-state,--no-as-needed ... -Wl,--pop-state whenever they really mean to link some library is too hostile.
Maybe pkg-config could learn to push state and use —as-needed? Some explicit options in gcc and ld like —link-as-needed=libname and —always-link=libname would be nice, too. It seems that the toolchain folks (like you) want -lpthread and the like to keep working, and the package people want a sane default where symbols are made available but the DT_NEEDED entry is only used if needed.
On the other hand, switching to as-needed linking seems like it's a step in the right direction in general, but ISTM it would be much nicer to switch to as-needed direct binding. Do glibc and binutils even support Solaris-style direct binding yet?
Dominique Martinet wrote:
Karel Zak wrote on Wed, Aug 01, 2018 at 01:17:29PM +0200:
Well, --as-needed is workaround and nothing else. The real problem is mess in makefiles and .pc (pkg-config) files.
It would be better to use --as-needed for testing purpose only, and ask maintainers why the result with --as-needed is different.
Wearing a lib developer hat, I don't see how you can make a .pc that doesn't overlink if you provide something a bit entangled with other libs. The problem is that if your headers use any sub-lib type you need to add that lib in Requires: so that --cflags will pull that lib's include path;
That's what Requires.private is for
-- Rex
Rex Dieter wrote on Thu, Aug 02, 2018 at 08:37:34AM -0500:
Wearing a lib developer hat, I don't see how you can make a .pc that doesn't overlink if you provide something a bit entangled with other libs. The problem is that if your headers use any sub-lib type you need to add that lib in Requires: so that --cflags will pull that lib's include path;
That's what Requires.private is for
I'm aware of Requires.private, but that's not how I understand things currently work. man pc(5) says this for Requires and Requires.private: --- Requires Required dependencies that must be met for the package to be usable. All dependencies must be satisfied or the pkg-config implementation must not use the package. (optional; dependency list)
Requires.private Required dependencies that must be met for the package to be usable for static linking. All dependencies must be satisfied or the pkg-config implementation must not use the package for static linking. (optional; dependency list) ---
I'm not sure I see how that would be related to what is used for compiling and what is used for linking.
In practice, `pkg-config --cflags foo` will only fetch cflags for dependencies listed in Requires, not Requires.private I haven't tested how autotools/libtool handle this but I doubt it's much different than invoking pkg-config manually, and at least meson will only add cflags for dependencies put in 'Requires' as I would have expected.
Am I missing something?
Dominique Martinet wrote:
In practice, `pkg-config --cflags foo` will only fetch cflags for dependencies listed in Requires, not Requires.private
pkg-config --cflags foo fetches cflags of Requires.private items in foo.pc for me.
I've patched many packages to use that feature, and haven't noticed any breakage (so far).
-- Rex
The real problem here is when you have complex frameworks like gtk.
pkg-config basically will link you against gdk, gdk-pixbuf, gtk and a lot of other stuff. But in practice, not every application need to link against all of them.
On Thu, Aug 2, 2018 at 6:11 PM Rex Dieter rdieter@math.unl.edu wrote:
Dominique Martinet wrote:
In practice, `pkg-config --cflags foo` will only fetch cflags for dependencies listed in Requires, not Requires.private
pkg-config --cflags foo fetches cflags of Requires.private items in foo.pc for me.
I've patched many packages to use that feature, and haven't noticed any breakage (so far).
-- Rex _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
Rex Dieter wrote on Thu, Aug 02, 2018 at 10:14:13AM -0500:
Dominique Martinet wrote:
In practice, `pkg-config --cflags foo` will only fetch cflags for dependencies listed in Requires, not Requires.private
pkg-config --cflags foo fetches cflags of Requires.private items in foo.pc for me.
I see, that indeed appears to work on fedora, but not on the nixos box I tested (because having different include paths made testing easier there, but now I'm testing on fedora it looks OK there as you say)
The difference between the two is that nixos is using the freedesktop pkg-config[1] while fedora uses pkgconf[2], so it would appear this isn't standard maybe? At which point the man page could use being more explicit about it... Oh, now I'm looking there's a (rather long) bug open about this here: https://bugs.freedesktop.org/show_bug.cgi?id=105572
[1] https://www.freedesktop.org/wiki/Software/pkg-config/ [2] https://github.com/pkgconf/pkgconf
I've patched many packages to use that feature, and haven't noticed any breakage (so far).
Well, packages can be fixed on the fedora side, but as upstream I wouldn't accept such a change until the freedesktop version behaves the same as I'd want to support users of either pkg-config version.
Igor Gnatenko wrote on Thu, Aug 02, 2018 at 06:12:23PM +0200:
The real problem here is when you have complex frameworks like gtk.
pkg-config basically will link you against gdk, gdk-pixbuf, gtk and a lot of other stuff. But in practice, not every application need to link against all of them.
Yeah, complex frameworks are difficult with that, but upstream is actually dealing with it indirectly by switching to more modern build systems (for gtk, see gnome's MesonPorting[3] page)
meson will automatically add --as-needed to the linker flags as appropriate (don't ask me what is appropriate), so we won't need to force it for them when they're done.
Eventually some autotools/libtool macro could be added for other projects to use..
[3] https://wiki.gnome.org/Initiatives/GnomeGoals/MesonPorting
In case if anybody is interested,
after mass rebuild we've dropped roughly 27709 dependencies on libraries in RPM metadata which means that size of primary.xml has decreased by 1.3M which would be something like 100K in gzip archive.. Which is not very much, but it was never primary goal and still nice to see :) Although with other changes, metadata size (in compressed format) went from 15M to 14M.
Unfortunately I don't have any way to test performance of all random binaries we have in distribution, so I can't say how faster they started to be.
On Tue, Jul 3, 2018 at 10:23 AM Jan Kurik jkurik@redhat.com wrote:
Note from Change Wrangler: This Change Proposal requires mass rebuild. However, two weeks ago (June 19th), we have already passed the deadline for Change proposals requiring mass rebuild. I will leave the decision whether this Change proposal is accepted or not to RelEng and FESCo teams.
= Proposed System Wide Change: Remove Excessive Linking = https://fedoraproject.org/wiki/Changes/RemoveExcessiveLinking
Owner(s):
- Igor Gnatenko <ignatenkobrain at fedoraproject dot org>
- Neal Gompa <ngompa13 at gmail dot com>
Pass "--as-needed" flag the linker through default system-wide LDFLAGS.
== Detailed description == The flag ("--as-needed") tells the linker to link in the produced binary only the libraries containing symbols actually used by the binary itself. This binary can be either a final executale or another library. The use of the "--as-needed" flag allows the linker to avoid linking extra libraries in a binary. This not only improves startup times (as the loader does not have to load all the libraries for every step) but might avoid the full initialization of big frameworks.
== Scope ==
- Proposal owners:
Add "-Wl,--as-needed" into RPM_LD_FLAGS (in redhat-rpm-config).
- Other developers:
Nothing should break, but immediate work-around would be to disable this flag (will be provided in redhat-rpm-config) and fix real issue later.
- Release engineering:
#7604 [https://pagure.io/releng/issue/7604] (mass rebuild is desired after this change).
** List of deliverables: N/A (not a System Wide Change)
- Policies and guidelines:
Add information how to turn it off (TODO link to FPC ticket).
- Trademark approval:
N/A (not needed for this Change)
Jan Kuřík JBoss EAP Program Manager Red Hat Czech s.r.o., Purkynova 99/71, 612 45 Brno, Czech Republic _______________________________________________ devel-announce mailing list -- devel-announce@lists.fedoraproject.org To unsubscribe send an email to devel-announce-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapro...
devel@lists.stg.fedoraproject.org
-
Andrew Lutomirski -
Dominique Martinet -
Igor Gnatenko -
Jakub Jelinek -
Jan Kurik -
Karel Zak -
Matthew Miller -
mcatanzaro＠gnome.org -
Neal Gompa -
Rex Dieter -
Stephen Gallagher