Note: This started as notes and questions from FPC members at the meeting, progressed to more notes and questions that FPC members Remi|Fedora and abadger1999 (me) put together after the meeting, and ended with a thorough rewrite by langdon (one of the SCL Guideline drafters) after listening to abadger1999 and Remi|Fedora's explanations of their notes and questions. The final rewrite of these comments mostly takes the form of things that Should be changed in the draft but it must be noted that the final form wasn't created by the FPC so it is quite possible that the FPC will disagree or want to seek other options about some of the suggestions made here. Sorry for the confusing parentage.
Comments on the SCL Draft: https://fedoraproject.org/wiki/User:Bkabrda/SCLGuidelinesDraft
* Can we assume that the implementation can be changed to address problems, add features, or follow different conventions? Can be done if backwards compat is possible? Or is implementation set in stone so we'd have to kludge around any shortcomings?
* One of the goals of SCLs is isolation from the rest of the system, correct? It seems, though, that once you run the enable script, that shell would have a different PATH with the possibility of a different language interpreter. As long as scripts can use /usr/bin/env as a shebang or scripts are using PATH-based commands these will end up using programs from the SCL. Should FPC revisit banning these? Is it okay to run with SCLs even without banning? Just something to document?
== Criteria: When is a Collection the Right Choice? ==
* What is the logical limit of the "size" of that collection? * (toshio) Criteria should state whether collections are to be "thin" or "thick". Thin collections would consist of only a specific target (for instance, ruby1.9.3) and the packages needed to implement it. Thick collections would include the collection, the packages to implement it, and packages implemented on top of it. * langdon has the concept of "Ease of use" packages that are often needed by people developing in a certain environment (for instance, adding rails to the ruby1.9.3 collection). Including ease of use packages would be a restricted variety of thick collections. * langdon and I also discussed thick collections allowing as many packages being added to it as maintainers wanted. To take a python example, if people wanted to support django and flask and pyramid and TurboGears1 and TurboGears2 on python2.6 SCL, they could have packages for all of those in the python2.6 Collection. langdon thought this would be preferable to having framework-level SCLs but I have concerns about how this plays out with a backwards compatibility policy. * thin collections would depend on inter-scl dependencies to provide people with things they want for "ease of use". For instance, there could be a rails3 scl that depends on scl-ruby1.9.3 that provides rails3 for ruby1.9.3. * Naming of scls with inter-scl dependencies needs to be worked out. You might have people who want to package: rails3 + ruby1.9.3, rails4 + ruby1.9.3, rails3 + ruby2.0, rails4 + ruby2.0 and all of those need unique names.
* What may the purpose of the Collection be? * Criteria should indicate that a "language" may be included but not a framework (Rails) or a complete platform (LAMP). However, optional, "ease of use" packages may be included but not installed by the main meta-package. Otherwise, the scls will need to change/fork too fast * (toshio): langdon is convinced this is the right path but I'm not sure. There's a lot of problems with how this interacts with backwards compat requirements. I think a whole package has to be worked out with how those shape up. * Databases and other "servers" may be included but should never "replace" the native versions
* Collections and backwards compat: * A requirement that scls should fit the same size as reasonable backwards compat defintions from upstream. For instance, if foo-X.Y.Z and every foo-X release is backwards compatible with other foo-X releases, the scl should be for fooX and not fooXY. * The "API" of an scl should never "shrink" only "grow." For example, if Rails3 is included in the Ruby1.9 SCL, and Rails4 wants to be included it does not replace Rails3. If, hypothetically, Rails4 cannot be installed "next to" Rails3, a new SCL would need to be created to support Rails4, which requires a "real" change. * (toshio) What would the new SCL be named? * (toshio) This is problematic without also specifying a lifetime for SCLs and replacement of existing SCLs. Some of the knock-on questions around this: * What are we going to do about security issues in SCLs? Does our policy say that backwards compat is more important than security issues? * Maybe Fedora simply isn't about backwards compat to this extent and so we have to allow API breaks within a release. * Maybe the policy for SCLs in The Outer Rings should have a strict backwards compat section but SCLs in Fedora Commons are more relaxed (that would make the model: Fedora Commons SCLs are primarily for enabling different versions of dependencies for software that we want to ship in Fedora Commons; Outer Ring SCLs are primarily for enabling developers to target an SCL rather than Fedora as their platform.) * Maybe our policy should only allow "thin" SCLs. ie: a ruby1.9 SCL would just have ruby1.9 and anything it needed in order to run. If someone wanted to ship rails3 and rails4 targeted for ruby1.9, those would be two separate SCLs that depended on the ruby1.9 SCL. This allows individual SCL pieces to be deprecated without interfering with their parents or neighbors. * Only partially addressing the situation, could figure out some way to mark some pieces of an SCL as having guaranteed backwards compat while others are only implementation detail. If you had an SCL that shipped ruby1.9 but it needed a newer version of OpenSSL you might ship that inside the SCL at first. But you wouldn't want people to rely on that particular version being available in the SCL in the future (once the base OS was updated to include that version.) * No matter where it is, need to have some criteria aroud how to obsolete an SCL, how to remove an SCL altogether, and whether two separate SCLs can ship largely the same thing. * (toshio) Can we make a recommendation on how or when to use scl-dependencies. For example, in the Rails4 example above, there may be scenarios that we want to allow this in Fedora. Although, the recommendation could be that frameworks and the like should be provided outside of Fedora Commons. * Should a collection be used only for things that cannot be parallel packaged (e.g. python 3.3 can be packaged natively so it's not a candidate for an SCL but ruby1.9 cannot and thus it is)
== Naming == * problem with the examples (ruby193 and examples from: http://fedorahosted.org/SoftwareCollections ) For instance, python3.3 would conflict with a python3.3 parallel stack that did not use scls. * Heard about namespace prefixes and that there were thoughts to name them after vendors (rht-ruby1.9). If we use /opt/$vendor that might make sense. * Could still have clashes with things like redhat-release, redhat-rpm-config, etc... make sure the vendors we use doesn't have any pre-existing conflicts of this sort. * scl makes sense as well. mmaslano had said that end-users wouldn't know what an scl was but remi says you'd need to use a tool called "scl" to manage which scls are active anyway. So that may not be a problem? * If we have multi-vendor for scl (fedora, rpmfusion, local admin case again), then vendor may make more sense again. * Change example naming: ruby193 => ruby1.9.3 Dots are clearer and thus preferred * Do we really want minor versions in there? or should this be ruby1.9 packages? Need to make clear that the version should be about backwards compatibility. So if ruby doesn't ensure that minor versions have compatibility, scl-ruby1.9.3 would be appropriate. OTOH, scl-python2.7 would be appropriate for an upstream that does have minor version backwards compat.
== File Location ==
* If we're going to use /opt/vendor: * scls should then obey the rest of the standard: /var/opt and /etc/opt are for host specific state and config. Is that the case? can it be fixed if not? * We'd need to get a fedora vendor string from LANANA * Might not want it to be "fedora" that might be more likely to be taken than something else. "fedoraproject"? "fdr"? * Other locations: * /usr/scl * (toshio): "Large software packages must not use a direct subdirectory under the /usr hierarchy." which could be interpreted as forbiding this location. (Although others may be more strict about defining "large sofware packages"). * %{_libdir}/scl * (toshio): I don't see an FHS problem with this although the same issue arises about putting the host-specific config in /etc and the variable state data in /var. I'm also coming around to /opt/$registered_vendor being the least-wrong place for this. * What does the filesystem layout look like inside of this prefix? /opt/$vendor/scl/rht-ruby1.9.3/{normal filesystem}? * Does this allow local sites and addon repos to make their own vendor? How? (Thinking of rhel/centos, epel, and local admin here. Or fedora/rpmfusion, and local admin) * Are scls allowed to install files outside of the scl tree? If so, what types of files (criteria) and the naming of these? (systemd unit files, rpm macro files, launchers for applications/language interpreters in the scl? Should we go with the FHS recommendation about /opt/vendor containing everything that is necessary for the package and then making copies(symlinks maybe?) to other needed locations? * (langdon) personally, i dont think it should touch anything besides opt and, potentially, symlinks * (toshio) Well, how do things like the postgres package that start a daemon work then? * (toshio) I don't think we can have symlinks into /opt work as expected but symlinks out of /opt should work. /etc/opt/scl/[...]/config and /var/opt/scl/[...]/state are both host-specific while /opt may be a shared network filesystem.
== Spec Files == * Why don't we follow the mingw model and have separate spec files (mingw is both separate spec and separate package) for scl? Advantages: * Many fewer conditionals in spec files (greater readability/maintainabilty) * Primary maintainer doesn't need to know or care about SCLs if other people do * SCL could be the only spec that is necessary or desired for a particular component * Can a SCL spec be "only" an SCL spec (will not build a non-scl rpm)? If so, we need some more content describing this scenario. * Propose moving the "style guide" to before the example (e.g. Section 3 before Section 2), so that some of the following questions don't come up for the reader * Why are there so many macros that mean the same thing? -- %scl, %pkg_name, %scl_name, %name
=== Specific comments on the draft === * Move template to the bottom of the draft. Had many questions after reading the template that were eventually answered in other parts of the draft. * Style guide is not to have a period at the end of Summary * Description in template should probably be more clear that packagers should fill in a proper description * Template skips %prep and %build. Are these empty? If so, just include the empty %prep and %build headers. * "The -build subpackage should include Requires: scl-utils-build." Why a should and not a must? * Content about the enable scriptlet implies that the usual path macros (_bindir etc) are changed in the scl package. Need to make it clear what standard macros are being overridden to what. Perhaps consider "tables" like http://fedoraproject.org/wiki/Packaging:MinGW#Filesystem_location_macros but with slightly different columns, e,g, macro | override? | Normally Modified? | example | description * "The macros.%{scl}-config in %{scl}-build package contains the %scl macro definition (name of the SCL). If you need to use some more macros for this SCL, add them into this file." <= but there's nothing in the spec file that obviously creates this. Need instructions on how to create this file for manual use and then how to include the file in the spec file and not have whatever automated macro normally creates it not overwrite it. * "essential for its runtime". "Essential" needs to be explained and hopefully it is just a link to the earlier part of the doc. * -runtime package must be arch specific => it contains %{scl_prefix}%{_libdir} so it's arch dependent. (and this means the main package must be arch specific too)
* Creating the enable file in %install -- would rather see macros like: - %scl_add_path %scl_add_ld_library_path %scl_add_manpath than the HERE-doc style thing that will cut and paste to most specs. * "set of macros that contain the values of the original RPM macros" should be expanded on. naming of these could be better => establish a common prefix for anything that is being overridden instead of %pkg_name vs %_root_. also pkg_name probably conflicts with some existing packages. Maybe something like %_prescl_name, %_prescl_libdir, etc. [can change the recommendation from: packagers should define %pkg_name to packagers should define a macro that is the value of pkg_name => %{?scl: %global srcname %pkg_name and %{?!scl: %global srcname %name} * Little more guidance needed for Requires. Maybe making the examples more real-life would clarify things. ("Since ifconfig is a binary we're going to use the system version of, do not use scl_prefix with it. rubygem-foo, otoh, has to be in the path of the ruby interpreter which this scl is to be a part of. Therefore it needs to use scl_prefix"). Also, an example of a "Versioned Requires" would be helpful. * Need more wording on Requires: of non-scl packages as they can act as conflicts. Something like: Requires: foo = 1.2.3 … where the scl package is the only thing keeping foo back at that version. So you can't update "the OS" without updating the scl. wording might be something like "yes, you can require non-scl stuff … but you have to be very general in your requirements, and if you need something that might be updated you have to either scl bundle it or be willing to rebuild the scl when it updates". * Should we list somewhere the scl-modified deps scripts? Otherwise how will someone creating a new ocaml-scl know whether an ocaml-scl dep script has already been written? * Please elaborate on the function of %scl_package_override() * In the dealing with macro files section: I think that the "Now consider a situation[...]" starts a whole new scenario? Correct? If so, needs to be better phrased right now it's associating with the previous examples but it doesn't seem to make sense in that context. * So If I've understood everything correctly, building in a standard buildroot will get you a non-scl'd package. Building with the $scl-build package installed will build the scl version? * Should say that in the first line of Building Packages * In Building Packages: Don't need to include scl-utils-build explicitly in the config_opts line (brought in via dependencies), correct?
== Design == * What directories is an SCL allowed to modify? * Naming of packages is a problem as currently designed. Liklihood of conflict with native packages is very high. Propose to include "scl-" at the begining of names. * Include "dots" in names, e.g. ruby1.9 rather than ruby19
== Conversion to SCLs == * Parts of this section are more "tips & tricks"? Should it be on a non-guideline page? * Probably best to vette the advice anyway and then move what makes sense to a separate page afterwards. * Add information about why /usr/bin/env is not a generally good thing in SCLs. If this is more general than SCLs, does FPC need to revisit a ban on /usr/bin/env in shebangs? * sed line should limit to the first line of the file as well as the shebang pattern. * how does scl deal with compiled-in paths inside of binaries? Does it depend on %configure to do the right thing with %{_libdir} etc? If so we should point out to people that old-school builds which aren't configure driven may need patches (actually, this applies to install location as well as compiled-in paths and to scripts that have file paths embedded as well as binaries) * How do auto provides and requires for elf libraries work?
== Inter-SCL Dependencies == * Example of using this with versioned Requires? * The %scl_require example just doesn't seem right somehow.... The ruby193 scl brings in rails-3.2.3? That seems like poor naming (if the metapackage brings in rails, then it should be named rails. OTOH, if the package purpose is to provide ruby193 then rails should be optional not essential.) or poor dependency management (Which would be part of the higher level questions of what can be an SCL and what can be inside an SCL).
* Are non-scl packages allowed to dep on scl packages? If so, how do they do so?
* The filters don't work in EL6 thing.... people seem more excited about SCLs in EPEL than in fedora so we probably need to document how to filter the provides and requires for EL6. However, we can point to the EPEL6 Differences page from here and document the behaviour there.
== Build & Distribution == * What are the impacts on the build system? * How are we preventing build problems from the mainline package from interfering with the scl package and vice versa? * How are we allowing for a package to be built in more than one scl? * How are we allowing for differences between scls? (ie: ruby1.9.3 scl with rubygem-foo1.0; ruby2.0 scl with rubygem-foo2.0) * How are we preventing build problems from one scl from interfering with a different scl? * Remi mentions that he thinks it would be done in a separate branch. Would we need a branch per scl that the package is built in? * How do SCLs get distributed? * Is this a "separate repo" or is it just in the normal repo(s)? * (toshio) Both. Some SCLs can go into Fedora Commons which would continue to have one repo. SCLs in the outer rings would have some separate structure. * Build section would benefit from minor sections, perhaps: "Overview," "Testing" and "Example"? Sections need a bit more color to be understandable.
== General Comments == * mingw is a similar set of guidelines (but in separate spec files rather than the same one.) So issues like what their naming solves and tracking of fedora packages could be looked at. http://fedoraproject.org/wiki/Packaging:MinGW. The mingw packaging guidelines should also be reviewed for "missing content" vs this proposal. * No definition of a SCL appears in the document * Example text: Software Collections (SCLs) give rpm packagers a means of packaging multiple versions of software for a single distribution. The version from the software collection does not interact with the system version. They can provide backwards and forwards compat versions of a package compared to what's on the system. * Include "definitions" at the beginning of the document, some example things that need definition (and suggestions) * SCL -> whole software collection? <- originally the term was "dynamic software collection" for the "whole thing" but it has fallen in to disuse... we could bring it back * SCL defining package -> the srpm-level package that defines the scl. * SCL metapackage -> built from the SCL defining package. defines the essential packages you get when you install the scl. * SCL runtime package -> built from the SCL defining package. It's the setup of the system for this SCL including registering the SCL with the system and setting up the filesystem. * SCL build package -> built from the SCL defining package. It contains macro definitions needed to build an SCL. * SCL package -> any package which can be built for an SCL. * Elaborate on what "belongs" in the "-runtime" package * Extracting one of these from the sample repository seems like it's a filesystem package directory tree inside of the scl? Anything else? * From the SCLinFedora page, why the differentiation between regular package reviews and scl package reviews?
-Toshio
On 09/16/2013 08:25 PM, Toshio Kuratomi wrote:
Note: This started as notes and questions from FPC members at the meeting, progressed to more notes and questions that FPC members Remi|Fedora and abadger1999 (me) put together after the meeting, and ended with a thorough rewrite by langdon (one of the SCL Guideline drafters) after listening to abadger1999 and Remi|Fedora's explanations of their notes and questions. The final rewrite of these comments mostly takes the form of things that Should be changed in the draft but it must be noted that the final form wasn't created by the FPC so it is quite possible that the FPC will disagree or want to seek other options about some of the suggestions made here. Sorry for the confusing parentage.
Comments on the SCL Draft: https://fedoraproject.org/wiki/User:Bkabrda/SCLGuidelinesDraft
- Can we assume that the implementation can be changed to address
problems, add features, or follow different conventions? Can be done if backwards compat is possible? Or is implementation set in stone so we'd have to kludge around any shortcomings?
Yes, we can assume changes. If you have ideas for improvement I'd like to hear them. Current guidelines are based on usage by wide variety of developers and users, so there shouldn't be needed many changes.
- One of the goals of SCLs is isolation from the rest of the system,
correct? It seems, though, that once you run the enable script, that shell would have a different PATH with the possibility of a different language interpreter. As long as scripts can use /usr/bin/env as a shebang or scripts are using PATH-based commands these will end up using programs from the SCL. Should FPC revisit banning these? Is it okay to run with SCLs even without banning? Just something to document?
There are mainly two possibilities how to run apps in collections. It depends what you wish to do with your application. Please do not ban anything, people have to decide for themselves what is their use-case.
1/ #!/usr/bin/env something is good for developers who want to switch between versions. I guess on Fedora will be some developers. 2/ #!/opt/vendor/something/root/usr/bin/something it's used as wrapper, good for deployed apps, because those who deploy app can define in wrapper, which collections have to be enabled by default
We do not provide wrappers inside collections, projects usually write them for themselves. I would be fix it only in documentation.
== Criteria: When is a Collection the Right Choice? ==
- What is the logical limit of the "size" of that collection?
- (toshio) Criteria should state whether collections are to be
"thin" or "thick". Thin collections would consist of only a specific target (for instance, ruby1.9.3) and the packages needed to implement it. Thick collections would include the collection, the packages to implement it, and packages implemented on top of it.
- langdon has the concept of "Ease of use" packages that are often
needed by people developing in a certain environment (for instance, adding rails to the ruby1.9.3 collection). Including ease of use packages would be a restricted variety of thick collections.
- langdon and I also discussed thick collections allowing as many
packages being added to it as maintainers wanted. To take a python example, if people wanted to support django and flask and pyramid and TurboGears1 and TurboGears2 on python2.6 SCL, they could have packages for all of those in the python2.6 Collection. langdon thought this would be preferable to having framework-level SCLs but I have concerns about how this plays out with a backwards compatibility policy.
- thin collections would depend on inter-scl dependencies to provide
people with things they want for "ease of use". For instance, there could be a rails3 scl that depends on scl-ruby1.9.3 that provides rails3 for ruby1.9.3. * Naming of scls with inter-scl dependencies needs to be worked out. You might have people who want to package: rails3 + ruby1.9.3, rails4 + ruby1.9.3, rails3 + ruby2.0, rails4 + ruby2.0 and all of those need unique names.
We are afraid that giving so much space for combination might lead to chaos. I do not want to see many combinations of ruby and rails of different versions with different versions of bundler etc. I proposed having SCL SIG which would say something like: "We already have Ruby193 and Rails2.x. If you want to just update rubygem-XY it's fine with us." or "We have it, updating rubygem-XY will broke it for sure for those users, create your own collection."
I agree having smaller collections than we have now would be improvement, but I'd like to give boundaries to all those possible combinations. In Fedora must be shipped only those collections, which are known as stable, well tested, with packages working together.
I'm not sure about this whole idea. The freedom might be a good thing, but we would have to set boundaries for what will go into Fedora. I do not have any idea how to do it, then give power to those who work on SCL or testers in bodhi, but than we would need really huge number of +1 or someone with veto.
- What may the purpose of the Collection be?
- Criteria should indicate that a "language" may be included but
not a framework (Rails) or a complete platform (LAMP). However, optional, "ease of use" packages may be included but not installed by the main meta-package. Otherwise, the scls will need to change/fork too fast * (toshio): langdon is convinced this is the right path but I'm not sure. There's a lot of problems with how this interacts with backwards compat requirements. I think a whole package has to be worked out with how those shape up.
Collection in Fedora should give developers, who are not ready to move to latest version, time to develop on older version.
Typically, Ruby and Rails, Python and mod_wsgi, but I do not suggest to add fast moving frameworks. We have cpan/pip for such packages.
* Databases and other "servers" may be included but should never"replace" the native versions
I agree with that. We need some system database (or daemon with some specific functionality).
- Collections and backwards compat:
- A requirement that scls should fit the same size as reasonable
backwards compat defintions from upstream. For instance, if foo-X.Y.Z and every foo-X release is backwards compatible with other foo-X releases, the scl should be for fooX and not fooXY.
Fine.
* The "API" of an scl should never "shrink" only "grow." Forexample, if Rails3 is included in the Ruby1.9 SCL, and Rails4 wants to be included it does not replace Rails3. If, hypothetically, Rails4 cannot be installed "next to" Rails3, a new SCL would need to be created to support Rails4, which requires a "real" change.
Agreed.
* (toshio) What would the new SCL be named? * (toshio) This is problematic without also specifying a lifetimefor SCLs and replacement of existing SCLs. Some of the knock-on questions around this: * What are we going to do about security issues in SCLs? Does our policy say that backwards compat is more important than security issues?
No. We do not want to support not supportable (terribly broken) versions.
* Maybe Fedora simply isn't about backwards compat to thisextent and so we have to allow API breaks within a release.
No, it would loose the purpose of scl for projects building above it.
* Maybe the policy for SCLs in The Outer Rings should have astrict backwards compat section but SCLs in Fedora Commons are more relaxed (that would make the model: Fedora Commons SCLs are primarily for enabling different versions of dependencies for software that we want to ship in Fedora Commons; Outer Ring SCLs are primarily for enabling developers to target an SCL rather than Fedora as their platform.)
Rings are still unclear.
* Maybe our policy should only allow "thin" SCLs. ie: aruby1.9 SCL would just have ruby1.9 and anything it needed in order to run. If someone wanted to ship rails3 and rails4 targeted for ruby1.9, those would be two separate SCLs that depended on the ruby1.9 SCL. This allows individual SCL pieces to be deprecated without interfering with their parents or neighbors. * Only partially addressing the situation, could figure out some way to mark some pieces of an SCL as having guaranteed backwards compat while others are only implementation detail. If you had an SCL that shipped ruby1.9 but it needed a newer version of OpenSSL you might ship that inside the SCL at first. But you wouldn't want people to rely on that particular version being available in the SCL in the future (once the base OS was updated to include that version.)
I might misunderstand. Do you suggest collection should "bundle" such package as openssl? That's security hole, not good for Fedora in circle 1, 2 or 3.
* No matter where it is, need to have some criteria aroud howto obsolete an SCL, how to remove an SCL altogether, and whether two separate SCLs can ship largely the same thing.
Collection can be obsoleted same way as other packages, but let's say that maintainer should say, he do not want to support it for F-22 anymore (before its release). Those still interested could carry on.
* (toshio) Can we make a recommendation on how or when to usescl-dependencies. For example, in the Rails4 example above, there may be scenarios that we want to allow this in Fedora. Although, the recommendation could be that frameworks and the like should be provided outside of Fedora Commons.
Do you mean dependencies among collections?
- Should a collection be used only for things that cannot be parallel
packaged (e.g. python 3.3 can be packaged natively so it's not a candidate for an SCL but ruby1.9 cannot and thus it is)
Depends... Some projects would like to work with collections only, but currently I'm not convinced if it should be a rule or not.
== Naming ==
- problem with the examples (ruby193 and examples from:
http://fedorahosted.org/SoftwareCollections ) For instance, python3.3 would conflict with a python3.3 parallel stack that did not use scls.
Not true. We used python33 as a metapackage without dot.
- Heard about namespace prefixes and that there were thoughts to name
them after vendors (rht-ruby1.9). If we use /opt/$vendor that might make sense.
- Could still have clashes with things like redhat-release,
redhat-rpm-config, etc... make sure the vendors we use doesn't have any pre-existing conflicts of this sort.
- scl makes sense as well. mmaslano had said that end-users wouldn't
know what an scl was but remi says you'd need to use a tool called "scl" to manage which scls are active anyway. So that may not be a problem?
That's true. We were also thinking about users who are interested in app and know nothing about scl, so they just install app, run it and are not aware of scl at all.
- If we have multi-vendor for scl (fedora, rpmfusion, local admin
case again), then vendor may make more sense again.
- Change example naming: ruby193 => ruby1.9.3 Dots are clearer and
thus preferred
Dots and such stuff are evil. I guess dots were omitted on purpose.
- Do we really want minor versions in there? or should this be
ruby1.9 packages? Need to make clear that the version should be about backwards compatibility. So if ruby doesn't ensure that minor versions have compatibility, scl-ruby1.9.3 would be appropriate. OTOH, scl-python2.7 would be appropriate for an upstream that does have minor version backwards compat.
It depends on software. For example many people believes that Ruby do bigger changes in minor releases, on the other hand Python is not known for adding features into minor releases, so it's just python33 and not python335.
== File Location ==
- If we're going to use /opt/vendor:
- scls should then obey the rest of the standard: /var/opt and
/etc/opt are for host specific state and config. Is that the case? can it be fixed if not?
Sometimes were used locations in /var/log or /etc because collections wouldn't work in case of mariadb and other daemons. We had to put init file somewhere and also logfile. It's documented and we didn't find better solution yet.
- We'd need to get a fedora vendor string from LANANA
- Might not want it to be "fedora" that might be more likely to be
taken than something else. "fedoraproject"? "fdr"?
- Other locations:
- /usr/scl
- (toshio): "Large software packages must not use a direct
subdirectory under the /usr hierarchy." which could be interpreted as forbiding this location. (Although others may be more strict about defining "large sofware packages").
- %{_libdir}/scl
- (toshio): I don't see an FHS problem with this although the same
issue arises about putting the host-specific config in /etc and the variable state data in /var. I'm also coming around to /opt/$registered_vendor being the least-wrong place for this.
I support installation into /opt because it will be confusing to have in some downstreams /opt and in other /usr. Also it's possible to do symlinks to workaround some problems and these might broke.
- What does the filesystem layout look like inside of this prefix?
/opt/$vendor/scl/rht-ruby1.9.3/{normal filesystem}?
yes, normal filesystem.
- Does this allow local sites and addon repos to make their own
vendor? How? (Thinking of rhel/centos, epel, and local admin here. Or fedora/rpmfusion, and local admin)
Modify vendor in macros of scl.
- Are scls allowed to install files outside of the scl tree? If so,
what types of files (criteria) and the naming of these? (systemd unit files, rpm macro files, launchers for applications/language interpreters in the scl? Should we go with the FHS recommendation about /opt/vendor containing everything that is necessary for the package and then making copies(symlinks maybe?) to other needed locations?
I'll add those exceptions.
- (langdon) personally, i dont think it should touch anything
besides opt and, potentially, symlinks * (toshio) Well, how do things like the postgres package that start a daemon work then? * (toshio) I don't think we can have symlinks into /opt work as expected but symlinks out of /opt should work. /etc/opt/scl/[...]/config and /var/opt/scl/[...]/state are both host-specific while /opt may be a shared network filesystem.
== Spec Files ==
- Why don't we follow the mingw model and have separate spec files
(mingw is both separate spec and separate package) for scl? Advantages: * Many fewer conditionals in spec files (greater readability/maintainabilty) * Primary maintainer doesn't need to know or care about SCLs if other people do * SCL could be the only spec that is necessary or desired for a particular component
- Can a SCL spec be "only" an SCL spec (will not build a non-scl
rpm)? If so, we need some more content describing this scenario.
- Propose moving the "style guide" to before the example (e.g.
Section 3 before Section 2), so that some of the following questions don't come up for the reader
- Why are there so many macros that mean the same thing? -- %scl,
%pkg_name, %scl_name, %name
I believe we should have branch for scl and other branches might voluntarily add macros, but it shouldn't be mandatory. For example perl or httpd don't have nice specfile and adding more condition wouldn't improve readability. Having scl macros might improve speed of patching some packages, but not all.
=== Specific comments on the draft ===
- Move template to the bottom of the draft. Had many questions after
reading the template that were eventually answered in other parts of the draft.
- Style guide is not to have a period at the end of Summary
- Description in template should probably be more clear that
packagers should fill in a proper description
- Template skips %prep and %build. Are these empty? If so, just
include the empty %prep and %build headers.
- "The -build subpackage should include Requires: scl-utils-build."
Why a should and not a must?
- Content about the enable scriptlet implies that the usual path
macros (_bindir etc) are changed in the scl package. Need to make it clear what standard macros are being overridden to what. Perhaps consider "tables" like http://fedoraproject.org/wiki/Packaging:MinGW#Filesystem_location_macros but with slightly different columns, e,g, macro | override? | Normally Modified? | example | description
- "The macros.%{scl}-config in %{scl}-build package contains the %scl
macro definition (name of the SCL). If you need to use some more macros for this SCL, add them into this file." <= but there's nothing in the spec file that obviously creates this. Need instructions on how to create this file for manual use and then how to include the file in the spec file and not have whatever automated macro normally creates it not overwrite it.
- "essential for its runtime". "Essential" needs to be explained and
hopefully it is just a link to the earlier part of the doc.
- -runtime package must be arch specific => it contains
%{scl_prefix}%{_libdir} so it's arch dependent. (and this means the main package must be arch specific too)
- Creating the enable file in %install -- would rather see macros
like: - %scl_add_path %scl_add_ld_library_path %scl_add_manpath than the HERE-doc style thing that will cut and paste to most specs.
- "set of macros that contain the values of the original RPM macros"
should be expanded on. naming of these could be better => establish a common prefix for anything that is being overridden instead of %pkg_name vs %_root_. also pkg_name probably conflicts with some existing packages. Maybe something like %_prescl_name, %_prescl_libdir, etc. [can change the recommendation from: packagers should define %pkg_name to packagers should define a macro that is the value of pkg_name => %{?scl: %global srcname %pkg_name and %{?!scl: %global srcname %name}
- Little more guidance needed for Requires. Maybe making the examples
more real-life would clarify things. ("Since ifconfig is a binary we're going to use the system version of, do not use scl_prefix with it. rubygem-foo, otoh, has to be in the path of the ruby interpreter which this scl is to be a part of. Therefore it needs to use scl_prefix"). Also, an example of a "Versioned Requires" would be helpful.
- Need more wording on Requires: of non-scl packages as they can act
as conflicts. Something like: Requires: foo = 1.2.3 … where the scl package is the only thing keeping foo back at that version. So you can't update "the OS" without updating the scl. wording might be something like "yes, you can require non-scl stuff … but you have to be very general in your requirements, and if you need something that might be updated you have to either scl bundle it or be willing to rebuild the scl when it updates".
- Should we list somewhere the scl-modified deps scripts? Otherwise
how will someone creating a new ocaml-scl know whether an ocaml-scl dep script has already been written?
- Please elaborate on the function of %scl_package_override()
- In the dealing with macro files section: I think that the "Now
consider a situation[...]" starts a whole new scenario? Correct? If so, needs to be better phrased right now it's associating with the previous examples but it doesn't seem to make sense in that context.
- So If I've understood everything correctly, building in a standard
buildroot will get you a non-scl'd package. Building with the $scl-build package installed will build the scl version?
- Should say that in the first line of Building Packages
- In Building Packages: Don't need to include scl-utils-build
explicitly in the config_opts line (brought in via dependencies), correct?
== Design ==
- What directories is an SCL allowed to modify?
- Naming of packages is a problem as currently designed. Liklihood of
conflict with native packages is very high. Propose to include "scl-" at the begining of names.
Why don't we already prefixed scl before all packages? Because it would be scl-ruby193-rubygems-blabla-1.1.f21. The prefix must not conflict with existing packages. It doesn't have to be ruby193, it might be name which specify what is inside.
- Include "dots" in names, e.g. ruby1.9 rather than ruby19
I prefer without dots, punctuation does everything harder.
== Conversion to SCLs ==
- Parts of this section are more "tips & tricks"? Should it be on a
non-guideline page? * Probably best to vette the advice anyway and then move what makes sense to a separate page afterwards.
- Add information about why /usr/bin/env is not a generally good
thing in SCLs. If this is more general than SCLs, does FPC need to revisit a ban on /usr/bin/env in shebangs?
- sed line should limit to the first line of the file as well as the
shebang pattern.
- how does scl deal with compiled-in paths inside of binaries? Does
it depend on %configure to do the right thing with %{_libdir} etc? If so we should point out to people that old-school builds which aren't configure driven may need patches (actually, this applies to install location as well as compiled-in paths and to scripts that have file paths embedded as well as binaries)
- How do auto provides and requires for elf libraries work?
== Inter-SCL Dependencies ==
- Example of using this with versioned Requires?
- The %scl_require example just doesn't seem right somehow.... The
ruby193 scl brings in rails-3.2.3? That seems like poor naming (if the metapackage brings in rails, then it should be named rails. OTOH, if the package purpose is to provide ruby193 then rails should be optional not essential.) or poor dependency management (Which would be part of the higher level questions of what can be an SCL and what can be inside an SCL).
- Are non-scl packages allowed to dep on scl packages? If so, how do
they do so?
- The filters don't work in EL6 thing.... people seem more excited
about SCLs in EPEL than in fedora so we probably need to document how to filter the provides and requires for EL6. However, we can point to the EPEL6 Differences page from here and document the behaviour there.
== Build & Distribution ==
- What are the impacts on the build system?
Until now we used internal instance of koji. I can sum up what we did there.
* How are we preventing build problems from the mainline packagefrom interfering with the scl package and vice versa?
There are no build interference problems. Package has specific name.
* How are we allowing for a package to be built in more than one scl?
Didn't solve yet. We related build targets (collection) with branch. How to build for more collections is a good question for our releng. But do we need it?
* How are we allowing for differences between scls? (ie:ruby1.9.3 scl with rubygem-foo1.0; ruby2.0 scl with rubygem-foo2.0) * How are we preventing build problems from one scl from interfering with a different scl? * Remi mentions that he thinks it would be done in a separate branch. Would we need a branch per scl that the package is built in?
Can't happen. Every branch is set to build in some tag. Packages tagged in one tag are shipped as one collection. You are able to inherit from one tag to another. As I said I prefer branch per collection. How else would be tags defined?
- How do SCLs get distributed?
- Is this a "separate repo" or is it just in the normal repo(s)?
Currently separate repo.
* (toshio) Both. Some SCLs can go into Fedora Commons whichwould continue to have one repo. SCLs in the outer rings would have some separate structure.
Agreed.
- Build section would benefit from minor sections, perhaps:
"Overview," "Testing" and "Example"? Sections need a bit more color to be understandable.
I'll add it.
== General Comments ==
- mingw is a similar set of guidelines (but in separate spec files
rather than the same one.) So issues like what their naming solves and tracking of fedora packages could be looked at. http://fedoraproject.org/wiki/Packaging:MinGW. The mingw packaging guidelines should also be reviewed for "missing content" vs this proposal.
- No definition of a SCL appears in the document
- Example text: Software Collections (SCLs) give rpm packagers a
means of packaging multiple versions of software for a single distribution. The version from the software collection does not interact with the system version. They can provide backwards and forwards compat versions of a package compared to what's on the system.
- Include "definitions" at the beginning of the document, some
example things that need definition (and suggestions) * SCL -> whole software collection? <- originally the term was "dynamic software collection" for the "whole thing" but it has fallen in to disuse... we could bring it back * SCL defining package -> the srpm-level package that defines the scl. * SCL metapackage -> built from the SCL defining package. defines the essential packages you get when you install the scl. * SCL runtime package -> built from the SCL defining package. It's the setup of the system for this SCL including registering the SCL with the system and setting up the filesystem. * SCL build package -> built from the SCL defining package. It contains macro definitions needed to build an SCL. * SCL package -> any package which can be built for an SCL.
- Elaborate on what "belongs" in the "-runtime" package
- Extracting one of these from the sample repository seems like
it's a filesystem package directory tree inside of the scl? Anything else?
- From the SCLinFedora page, why the differentiation between regular
package reviews and scl package reviews?
-Toshio
packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
Thanks for your input. I was waiting for something like that so I could add what is missing or unclear to people who never build a collection. I'm looking forward to reaction on updated drafts. We will do it with Slavek tomorrow.
Marcela
On Tue, Sep 17, 2013 at 8:33 AM, Marcela Mašláňová mmaslano@redhat.comwrote:
On 09/16/2013 08:25 PM, Toshio Kuratomi wrote:
== Build & Distribution ==
- What are the impacts on the build system?
Until now we used internal instance of koji. I can sum up what we did there.
* How are we preventing build problems from the mainline packagefrom interfering with the scl package and vice versa?
There are no build interference problems. Package has specific name.
* How are we allowing for a package to be built in more than one scl?Didn't solve yet. We related build targets (collection) with branch. How to build for more collections is a good question for our releng. But do we need it?
* How are we allowing for differences between scls? (ie:ruby1.9.3 scl with rubygem-foo1.0; ruby2.0 scl with rubygem-foo2.0) * How are we preventing build problems from one scl from interfering with a different scl? * Remi mentions that he thinks it would be done in a separate branch. Would we need a branch per scl that the package is built in?
Can't happen. Every branch is set to build in some tag. Packages tagged in one tag are shipped as one collection. You are able to inherit from one tag to another. As I said I prefer branch per collection. How else would be tags defined?
- How do SCLs get distributed?
- Is this a "separate repo" or is it just in the normal repo(s)?
Currently separate repo.
* (toshio) Both. Some SCLs can go into Fedora Commons whichwould continue to have one repo. SCLs in the outer rings would have some separate structure.
Agreed.
- Build section would benefit from minor sections, perhaps:
"Overview," "Testing" and "Example"? Sections need a bit more color to be understandable.
I'll add it.
I would like to request that one more question/issue be addressed in this section.
Red Hat has made a GREAT tool available in the devtoolset ( https://access.redhat.com/site/documentation/Red_Hat_Developer_Toolset/ ), but currently there's no way to take advantage of it in the EPEL. Can a plan to enable the use of the devtoolset for Koji builds for the EPEL for EL 5/6 be discussed as well?
The hope is that then packages that make use of the devtoolset could be created and then people who have subscriptions to the devtoolset could make use of them.
For example, I am working on packaging odb ( https://bugzilla.redhat.com/show_bug.cgi?id=975310 ), but it requires gcc 4.5 or greater for plugin support and that is only available on EL 5/6 through the devtoolset. So if it was possible to build on Koji using the devtoolset, then it would enable odb to be packaged for EL 5/6 and then those with access to the devtoolset could make use of it.
Thanks, Dave
On 09/18/2013 09:33 PM, Dave Johansen wrote:
For example, I am working on packaging odb ( https://bugzilla.redhat.com/show_bug.cgi?id=975310 ), but it requires gcc 4.5 or greater for plugin support and that is only available on EL 5/6 through the devtoolset.
What are the causes? To me, this sounds like a non-portable package.
So if it was possible to build on Koji using the devtoolset, then it would enable odb to be packaged for EL 5/6 and then those with access to the devtoolset could make use of it.
To me this boils down to the question: "Are people allowed to use RH's devtoolset in EPEL?".
Correct me if I am wrong, but to my knowledge, RH's devtoolset is only available to RH customers, which would mean EPEL can not use it.
Ralf
On Thu, Sep 19, 2013 at 10:47 AM, Ralf Corsepius rc040203@freenet.dewrote:
On 09/18/2013 09:33 PM, Dave Johansen wrote:
For example, I am working on packaging odb (
https://bugzilla.redhat.com/**show_bug.cgi?id=975310https://bugzilla.redhat.com/show_bug.cgi?id=975310), but it requires gcc 4.5 or greater for plugin support and that is only available on EL 5/6 through the devtoolset.
What are the causes? To me, this sounds like a non-portable package.
There are two parts of ODB: 1) the compiler and 2) the runtime. The compiler takes an input and generates C++ code that can be built against and used with the runtime. The generated code and the runtime can be used with gcc 4.2 or greater.
The issue is that ODB requires gcc's plugin support ( http://gcc.gnu.org/wiki/plugins ) to parse the input (which was introduced in gcc 4.5). Only gcc 4.4 is available on RHEL 5/6, but the devtoolset makes a newer version available, which enables the use of the compiler and not just the runtime.
So if it was possible to build on Koji using
the devtoolset, then it would enable odb to be packaged for EL 5/6 and then those with access to the devtoolset could make use of it.
To me this boils down to the question: "Are people allowed to use RH's devtoolset in EPEL?".
Correct me if I am wrong, but to my knowledge, RH's devtoolset is only available to RH customers, which would mean EPEL can not use it.
Yes, the devtoolset requires the appropriate subscription from RH, but that is the point of my request. I would like to request that devtoolset be made available on Koji so it can be used to build certain packages. The idea is that then it will build the rpm on the server and it will be available in the EPEL (but not the devtoolset itself). This means that anyone who would want to use this package would need to have the devtoolset installed, so it doesn't violate any of the RH/EPEL policies, but would enable the use of the devtoolset with the EPEL.
On 09/19/2013 03:18 PM, Dave Johansen wrote:
On Thu, Sep 19, 2013 at 10:47 AM, Ralf Corsepius <rc040203@freenet.de mailto:rc040203@freenet.de> wrote:
On 09/18/2013 09:33 PM, Dave Johansen wrote: For example, I am working on packaging odb ( https://bugzilla.redhat.com/__show_bug.cgi?id=975310 <https://bugzilla.redhat.com/show_bug.cgi?id=975310> ), but it requires gcc 4.5 or greater for plugin support and that is only available on EL 5/6 through the devtoolset. What are the causes? To me, this sounds like a non-portable package.There are two parts of ODB: 1) the compiler and 2) the runtime. The compiler takes an input and generates C++ code that can be built against and used with the runtime. The generated code and the runtime can be used with gcc 4.2 or greater.
The issue is that ODB requires gcc's plugin support ( http://gcc.gnu.org/wiki/plugins ) to parse the input (which was introduced in gcc 4.5). Only gcc 4.4 is available on RHEL 5/6, but the devtoolset makes a newer version available, which enables the use of the compiler and not just the runtime.
So if it was possible to build on Koji using the devtoolset, then it would enable odb to be packaged for EL 5/6 and then those with access to the devtoolset could make use of it. To me this boils down to the question: "Are people allowed to use RH's devtoolset in EPEL?". Correct me if I am wrong, but to my knowledge, RH's devtoolset is only available to RH customers, which would mean EPEL can not use it.Yes, the devtoolset requires the appropriate subscription from RH, but that is the point of my request. I would like to request that devtoolset be made available on Koji so it can be used to build certain packages. The idea is that then it will build the rpm on the server and it will be available in the EPEL (but not the devtoolset itself). This means that anyone who would want to use this package would need to have the devtoolset installed, so it doesn't violate any of the RH/EPEL policies, but would enable the use of the devtoolset with the EPEL.
I am not an expert, but I believe the point of the devtoolset is that you do *not* need the devtoolset upon deployment. As a result, you would not be required to have the devtoolset to use the rpm (unless you were rebuilding it or something).
-- packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
On Thu, Sep 19, 2013 at 1:28 PM, Langdon White langdon@fedoraproject.orgwrote:
On 09/19/2013 03:18 PM, Dave Johansen wrote:
On Thu, Sep 19, 2013 at 10:47 AM, Ralf Corsepius <rc040203@freenet.de mailto:rc040203@freenet.de> wrote:
On 09/18/2013 09:33 PM, Dave Johansen wrote: For example, I am working on packaging odb ( https://bugzilla.redhat.com/__**show_bug.cgi?id=975310<https://bugzilla.redhat.com/__show_bug.cgi?id=975310> <https://bugzilla.redhat.com/**show_bug.cgi?id=975310<https://bugzilla.redhat.com/show_bug.cgi?id=975310>>), but it requires gcc 4.5 or greater for plugin support and that is only available on EL 5/6 through the devtoolset.
What are the causes? To me, this sounds like a non-portable package.There are two parts of ODB: 1) the compiler and 2) the runtime. The compiler takes an input and generates C++ code that can be built against and used with the runtime. The generated code and the runtime can be used with gcc 4.2 or greater.
The issue is that ODB requires gcc's plugin support ( http://gcc.gnu.org/wiki/**plugins http://gcc.gnu.org/wiki/plugins ) to parse the input (which was introduced in gcc 4.5). Only gcc 4.4 is available on RHEL 5/6, but the devtoolset makes a newer version available, which enables the use of the compiler and not just the runtime.
So if it was possible to build on Koji using the devtoolset, then it would enable odb to be packaged for EL 5/6 and then those with access to the devtoolset could make use of it. To me this boils down to the question: "Are people allowed to use RH's devtoolset in EPEL?". Correct me if I am wrong, but to my knowledge, RH's devtoolset is only available to RH customers, which would mean EPEL can not use it.Yes, the devtoolset requires the appropriate subscription from RH, but that is the point of my request. I would like to request that devtoolset be made available on Koji so it can be used to build certain packages. The idea is that then it will build the rpm on the server and it will be available in the EPEL (but not the devtoolset itself). This means that anyone who would want to use this package would need to have the devtoolset installed, so it doesn't violate any of the RH/EPEL policies, but would enable the use of the devtoolset with the EPEL.
I am not an expert, but I believe the point of the devtoolset is that you do *not* need the devtoolset upon deployment. As a result, you would not be required to have the devtoolset to use the rpm (unless you were rebuilding it or something).
In general that is true, because the devtoolset generates binaries that can run on a vanilla RHEL system (i.e. without the devtoolset installed), but the issue is that the ODB compiler is a gcc plugin that uses gcc itself to parse C++ code (its input). So it's actually making use of the gcc compiler itself when it runs and isn't just code that is built into a stand alone executable like is the normal case for most projects. Since plugin support only exists in gcc 4.5 or greater, that's the reason for needing the devtoolset.
Also worth noting is that this is only true for the ODB compiler because the ODB runtime works with gcc 4.2 or greater and doesn't require the devtoolset or anything special.
On Thu, Sep 19, 2013 at 12:18 PM, Dave Johansen davejohansen@gmail.com wrote:
On Thu, Sep 19, 2013 at 10:47 AM, Ralf Corsepius rc040203@freenet.de wrote:
Correct me if I am wrong, but to my knowledge, RH's devtoolset is only available to RH customers, which would mean EPEL can not use it.
Yes, the devtoolset requires the appropriate subscription from RH, but that is the point of my request. I would like to request that devtoolset be made available on Koji so it can be used to build certain packages. The idea is that then it will build the rpm on the server and it will be available in the EPEL (but not the devtoolset itself). This means that anyone who would want to use this package would need to have the devtoolset installed, so it doesn't violate any of the RH/EPEL policies, but would enable the use of the devtoolset with the EPEL.
This is a bit of a tangent from the comments on the packaging guideline draft but in addition to Fedora passing draft guidelines, needing a separate subscription from RH may run afoul of a different EPEL Guideline -- the packages that EPEL bases against are not all of RHEL but only ones that come/came with the base subscription. This is a little ambiguous because this was figured out in the RHEL5 times but RHEL6 has slightly different subscriptions/channels/what have you. I'm not sure if the devtoolset you talk about will be a part of that or not. If it isn't you (or someone) would need to package those up for EPEL before they could be used to build other packages.
Do note that pulling the devtoolset into EPEL's buildroot directly from RedHat isn't something that we'd decide upon in the Fedora Packaging Committee. EPEL people sent you here because what we (FPC) decide about SCLs in Fedora will inform their views on whether and how EPEL should implement SCLs. So they're waiting on us but their decisions will be separate.
-Toshio
On Thu, Sep 19, 2013 at 4:59 PM, Toshio Kuratomi a.badger@gmail.com wrote:
On Thu, Sep 19, 2013 at 12:18 PM, Dave Johansen davejohansen@gmail.com wrote:
On Thu, Sep 19, 2013 at 10:47 AM, Ralf Corsepius rc040203@freenet.de wrote:
Correct me if I am wrong, but to my knowledge, RH's devtoolset is only available to RH customers, which would mean EPEL can not use it.
Yes, the devtoolset requires the appropriate subscription from RH, but
that
is the point of my request. I would like to request that devtoolset be
made
available on Koji so it can be used to build certain packages. The idea
is
that then it will build the rpm on the server and it will be available in the EPEL (but not the devtoolset itself). This means that anyone who
would
want to use this package would need to have the devtoolset installed, so
it
doesn't violate any of the RH/EPEL policies, but would enable the use of
the
devtoolset with the EPEL.
This is a bit of a tangent from the comments on the packaging guideline draft but in addition to Fedora passing draft guidelines, needing a separate subscription from RH may run afoul of a different EPEL Guideline -- the packages that EPEL bases against are not all of RHEL but only ones that come/came with the base subscription. This is a little ambiguous because this was figured out in the RHEL5 times but RHEL6 has slightly different subscriptions/channels/what have you. I'm not sure if the devtoolset you talk about will be a part of that or not. If it isn't you (or someone) would need to package those up for EPEL before they could be used to build other packages.
As I was reading through the original email, I realized that this was probably not directly to the issue being discussed here (i.e. the potential use of an SCL in the EPEL rather than guidelines for packaging of SCLs in Fedora/EPEL). But this is where I was asked to bring up the issue at, so I brought it up here in hopes that it would be able to become part of the discussion.
The devtoolset requires a "developer subscription" so it's not part of the base subscription. The devtoolset seems to be a new sort of animal when it comes to RHEL subscriptions, so maybe this isn't true, but wouldn't re-packaging it violate the rule about replacing packages ( http://fedoraproject.org/wiki/EPEL/FAQ#Does_EPEL_replace_packages_provided_w...
Do note that pulling the devtoolset into EPEL's buildroot directly from RedHat isn't something that we'd decide upon in the Fedora Packaging Committee. EPEL people sent you here because what we (FPC) decide about SCLs in Fedora will inform their views on whether and how EPEL should implement SCLs. So they're waiting on us but their decisions will be separate.
Like I said before, at least in regards to this issue, I'm more interested in enabling "the use of an SCL" rather than enabling "the packaging of one". So what can I do to help make sure that the final guidelines allow for that sort of use case? Because it seems that without the guidelines allowing for that, the EPEL guys won't even be able to make a decision at all.
Thanks, Dave
On Sep 19, 2013 9:12 PM, "Dave Johansen" davejohansen@gmail.com wrote:
On Thu, Sep 19, 2013 at 4:59 PM, Toshio Kuratomi a.badger@gmail.com
wrote:
On Thu, Sep 19, 2013 at 12:18 PM, Dave Johansen davejohansen@gmail.com
wrote:
On Thu, Sep 19, 2013 at 10:47 AM, Ralf Corsepius rc040203@freenet.de wrote:
Correct me if I am wrong, but to my knowledge, RH's devtoolset is only available to RH customers, which would mean EPEL can not use it.
Yes, the devtoolset requires the appropriate subscription from RH, but
that
is the point of my request. I would like to request that devtoolset be
made
available on Koji so it can be used to build certain packages. The
idea is
that then it will build the rpm on the server and it will be available
in
the EPEL (but not the devtoolset itself). This means that anyone who
would
want to use this package would need to have the devtoolset installed,
so it
doesn't violate any of the RH/EPEL policies, but would enable the use
of the
devtoolset with the EPEL.
This is a bit of a tangent from the comments on the packaging guideline draft but in addition to Fedora passing draft guidelines, needing a separate subscription from RH may run afoul of a different EPEL Guideline -- the packages that EPEL bases against are not all of RHEL but only ones that come/came with the base subscription. This is a little ambiguous because this was figured out in the RHEL5 times but RHEL6 has slightly different subscriptions/channels/what have you. I'm not sure if the devtoolset you talk about will be a part of that or not. If it isn't you (or someone) would need to package those up for EPEL before they could be used to build other packages.
As I was reading through the original email, I realized that this was
probably not directly to the issue being discussed here (i.e. the potential use of an SCL in the EPEL rather than guidelines for packaging of SCLs in Fedora/EPEL). But this is where I was asked to bring up the issue at, so I brought it up here in hopes that it would be able to become part of the discussion.
The devtoolset requires a "developer subscription" so it's not part of
the base subscription. The devtoolset seems to be a new sort of animal when it comes to RHEL subscriptions, so maybe this isn't true, but wouldn't re-packaging it violate the rule about replacing packages ( http://fedoraproject.org/wiki/EPEL/FAQ#Does_EPEL_replace_packages_provided_w...
Interesting. I was sure we didn't cover layered products because if we did there would be certain classes of packages that are unbuildable. Better ask nirik if that documentation is out dated.
Do note that pulling the devtoolset into EPEL's buildroot directly from RedHat isn't something that we'd decide upon in the Fedora Packaging Committee. EPEL people sent you here because what we (FPC) decide about SCLs in Fedora will inform their views on whether and how EPEL should implement SCLs. So they're waiting on us but their decisions will be separate.
Like I said before, at least in regards to this issue, I'm more
interested in enabling "the use of an SCL" rather than enabling "the packaging of one". So what can I do to help make sure that the final guidelines allow for that sort of use case? Because it seems that without the guidelines allowing for that, the EPEL guys won't even be able to make a decision at all.
If it's purely about using an scl then there's nothing that this guideline discussion will do to aid your purpose. This discussion is just about packaging scls.
You may need to go back to epel and make that clear. At which time they *may* say that devtoolset needs to be packaged in epel. And then (if I understand correctly) you may need to come back to this discussion to make sure you'll be able to package the scl that you need for epel. And finally after all that's done you'd be able to build the package you really want.
Sorry for the somewhat circuitous nature of this path... scls are a bit new and we (fedora packagers, fpc, epel) are all learning about all aspects of them.
(P.s. thanks for participating in this discussion. Your question sparks one that I need for the guidelines: there seems to be a slight preference for non-scl packages to not be able to require scl packages. But your situation brings up the question of whether it would be okay for them to buildrequire an scl package.)
-Toshio
On Thu, Sep 19, 2013 at 10:02 PM, Toshio Kuratomi a.badger@gmail.comwrote:
On Sep 19, 2013 9:12 PM, "Dave Johansen" davejohansen@gmail.com wrote:
On Thu, Sep 19, 2013 at 4:59 PM, Toshio Kuratomi a.badger@gmail.com
wrote:
On Thu, Sep 19, 2013 at 12:18 PM, Dave Johansen davejohansen@gmail.com
wrote:
On Thu, Sep 19, 2013 at 10:47 AM, Ralf Corsepius <rc040203@freenet.de
wrote:
Correct me if I am wrong, but to my knowledge, RH's devtoolset is
only
available to RH customers, which would mean EPEL can not use it.
Yes, the devtoolset requires the appropriate subscription from RH,
but that
is the point of my request. I would like to request that devtoolset
be made
available on Koji so it can be used to build certain packages. The
idea is
that then it will build the rpm on the server and it will be
available in
the EPEL (but not the devtoolset itself). This means that anyone who
would
want to use this package would need to have the devtoolset installed,
so it
doesn't violate any of the RH/EPEL policies, but would enable the use
of the
devtoolset with the EPEL.
This is a bit of a tangent from the comments on the packaging guideline draft but in addition to Fedora passing draft guidelines, needing a separate subscription from RH may run afoul of a different EPEL Guideline -- the packages that EPEL bases against are not all of RHEL but only ones that come/came with the base subscription. This is a little ambiguous because this was figured out in the RHEL5 times but RHEL6 has slightly different subscriptions/channels/what have you. I'm not sure if the devtoolset you talk about will be a part of that or not. If it isn't you (or someone) would need to package those up for EPEL before they could be used to build other packages.
As I was reading through the original email, I realized that this was
probably not directly to the issue being discussed here (i.e. the potential use of an SCL in the EPEL rather than guidelines for packaging of SCLs in Fedora/EPEL). But this is where I was asked to bring up the issue at, so I brought it up here in hopes that it would be able to become part of the discussion.
The devtoolset requires a "developer subscription" so it's not part of
the base subscription. The devtoolset seems to be a new sort of animal when it comes to RHEL subscriptions, so maybe this isn't true, but wouldn't re-packaging it violate the rule about replacing packages ( http://fedoraproject.org/wiki/EPEL/FAQ#Does_EPEL_replace_packages_provided_w...
Interesting. I was sure we didn't cover layered products because if we did there would be certain classes of packages that are unbuildable. Better ask nirik if that documentation is out dated.
Do note that pulling the devtoolset into EPEL's buildroot directly from RedHat isn't something that we'd decide upon in the Fedora Packaging Committee. EPEL people sent you here because what we (FPC) decide about SCLs in Fedora will inform their views on whether and how EPEL should implement SCLs. So they're waiting on us but their decisions will be separate.
Like I said before, at least in regards to this issue, I'm more
interested in enabling "the use of an SCL" rather than enabling "the packaging of one". So what can I do to help make sure that the final guidelines allow for that sort of use case? Because it seems that without the guidelines allowing for that, the EPEL guys won't even be able to make a decision at all.
If it's purely about using an scl then there's nothing that this guideline discussion will do to aid your purpose. This discussion is just about packaging scls.
You may need to go back to epel and make that clear. At which time they *may* say that devtoolset needs to be packaged in epel. And then (if I understand correctly) you may need to come back to this discussion to make sure you'll be able to package the scl that you need for epel. And finally after all that's done you'd be able to build the package you really want.
Sorry for the somewhat circuitous nature of this path... scls are a bit new and we (fedora packagers, fpc, epel) are all learning about all aspects of them.
No problem. I know that SCls are still a new thing and that working out all of the kinks and issues with new stuff takes some back and forth. I'll take the conversation back to the EPEL and see if I can do a better job of explaining the situation and get some traction there.
(P.s. thanks for participating in this discussion. Your question sparks
one that I need for the guidelines: there seems to be a slight preference for non-scl packages to not be able to require scl packages. But your situation brings up the question of whether it would be okay for them to buildrequire an scl package.)
I'm still somewhat new to the whole packaging arena, but I would think that not directly requiring an SCL would be a good rule. However, allowing things like a BuildRequires with a requirement on a minimum version number that can be satisfied by a SCL would be the right solution.
Right now, with the devtoolset that's not the case. It doesn't have a Provides statement that indicates that it makes gcc 4.7 available, but if it did and that was properly supported and used by the packaging, then it would make its use a LOT easier. I realize that the devtoolset is probably a little different than most SCLs because it's a toolchain/compiler/etc, but it would be nice if it "played nicer" with the whole packaging process.
Note, I'm posting the complete list of questions and answers onto a wiki page. But please, do discussion here and I'll update the wiki page as discussion progresses (wiki will allow to easily see what questions are answered. Mailing list will allow for easier discussion.)
On Tue, Sep 17, 2013 at 05:33:03PM +0200, Marcela Mašláňová wrote:
- One of the goals of SCLs is isolation from the rest of the system,
correct? It seems, though, that once you run the enable script, that shell would have a different PATH with the possibility of a different language interpreter. As long as scripts can use /usr/bin/env as a shebang or scripts are using PATH-based commands these will end up using programs from the SCL. Should FPC revisit banning these? Is it okay to run with SCLs even without banning? Just something to document?
There are mainly two possibilities how to run apps in collections. It depends what you wish to do with your application. Please do not ban anything, people have to decide for themselves what is their use-case.
1/ #!/usr/bin/env something is good for developers who want to switch between versions. I guess on Fedora will be some developers. 2/ #!/opt/vendor/something/root/usr/bin/something it's used as wrapper, good for deployed apps, because those who deploy app can define in wrapper, which collections have to be enabled by default
We do not provide wrappers inside collections, projects usually write them for themselves. I would be fix it only in documentation.
I think you're discussing the use of /usr/bin/env inside of an scl which was a question asked under iConversion to SCLs. This section was actually asking about the following type of scenario:
The Fedora Core provided yum package has /usr/bin/yum. The shebang for that is: #!/usr/bin/env python A sys admin has both that package installed and a python2.4 scl. When the sysadmin enables the python2.4 scl, yum will suddenly start using the scl provided python executable (I assume, only in the shell that the sysadmin enabled the scl in? Correct me if the scl can somehow be enabled for a user's whole session instead of within a single shell session). This is very likely to break yum.
FPC discussed this issue https://fedoraproject.org/wiki/Script_Interpreters_(draft) before: http://meetbot.fedoraproject.org/fedora-meeting/2009-08-19/fedora-meeting.20...
and at the time decided not to implement a ban on /usr/bin/env in shebangs.
I'm wondering if SCLs change the need for that as before the scenario was that the end user had to install something from source in order to break scripts which use /usr/bin/env. Now we'll provide packages that break those scripts. However, the end user still has to manually enable the scl so perhaps it's still something that we should just stick in documentation.
== Criteria: When is a Collection the Right Choice? ==
* Naming of scls with inter-scl dependencies needs to be workedout. You might have people who want to package: rails3 + ruby1.9.3, rails4 + ruby1.9.3, rails3 + ruby2.0, rails4 + ruby2.0 and all of those need unique names.
We are afraid that giving so much space for combination might lead to chaos. I do not want to see many combinations of ruby and rails of different versions with different versions of bundler etc. I proposed having SCL SIG which would say something like: "We already have Ruby193 and Rails2.x. If you want to just update rubygem-XY it's fine with us." or "We have it, updating rubygem-XY will broke it for sure for those users, create your own collection."
(Quick note -- if we allow these at all then we need to figure out naming that allows for it. This bullet was about naming but that doesn't seem to be addressed at all in the anwer :-).
I agree having smaller collections than we have now would be improvement, but I'd like to give boundaries to all those possible combinations. In Fedora must be shipped only those collections, which are known as stable, well tested, with packages working together.
I'm not sure about this whole idea. The freedom might be a good thing, but we would have to set boundaries for what will go into Fedora. I do not have any idea how to do it, then give power to those who work on SCL or testers in bodhi, but than we would need really huge number of +1 or someone with veto.
I expressed some of this in the ticket but let me go a little more in depth here:
I agree that it would be good for maintenance to limit the combinations.
However, the method of limiting that I'd propose is to have written criteria that prevents too many combinations. If having a group decide on allowing exceptions is in lieu of well defined criteria then I think that's a bad idea as it leaves everyone wondering where the line should be drawn and leaves people feeling slighted when their SCL is rejected but another SCL that they consider similar is accepted. We have experience with something like that in the bundled library exception process. Many people find it easier to ask for a bundling exceptions than to analyze the situation and try to correct the code first and its a constant struggle to be sure that we're fairly applying the guidelines.
Now as for what group should apply the written criteria to the proposed SCLs.... I could see this as something that packagers take on similar to package reviews. IMHO, this should be the case if we can come up with fairly complete criteria. If we think that our criteria leave some gaping holes we may need to do something more like the bundled library exception process where new SCLs are approved by a group that follows the criteria. From past experience, the group should not be subject to high turnover, should not be elected, and should have a focus on documenting new criteria when an SCL is accepted that doesn't seem to fit under the existing criteria. These are because the group needs to be consistent in how it applies the criteria, should be very willing to say no, and needs to record their decisions in a manner that allows people to have their SCLs revisited for inclusion as the criteria changes. SIGs are probably much too ad hoc to do this. Perhaps the Stacks and Environments Working Group would be okay but they might have other things that they'll need to work on (and we don't know how it will compose itself moving into the future either... it might be higher turnover than is good for a group making these decisions).
- What may the purpose of the Collection be?
- Criteria should indicate that a "language" may be included but
not a framework (Rails) or a complete platform (LAMP). However, optional, "ease of use" packages may be included but not installed by the main meta-package. Otherwise, the scls will need to change/fork too fast * (toshio): langdon is convinced this is the right path but I'm not sure. There's a lot of problems with how this interacts with backwards compat requirements. I think a whole package has to be worked out with how those shape up.
Collection in Fedora should give developers, who are not ready to move to latest version, time to develop on older version.
Typically, Ruby and Rails, Python and mod_wsgi, but I do not suggest to add fast moving frameworks. We have cpan/pip for such packages.
If we want to enable ruby on rails then we really are looking at enabling an scl up to the framework level. How we implement that is separate from that -- the framework could be a part of the language scl or it could be its own scl with a language dependency or the language could be a part of the framework scl.
- The "API" of an scl should never "shrink" only "grow." For
example, if Rails3 is included in the Ruby1.9 SCL, and Rails4 wants to be included it does not replace Rails3. If, hypothetically, Rails4 cannot be installed "next to" Rails3, a new SCL would need to be created to support Rails4, which requires a "real" change.
Agreed.
* (toshio) What would the new SCL be named? * (toshio) This is problematic without also specifying a lifetimefor SCLs and replacement of existing SCLs. Some of the knock-on questions around this: * What are we going to do about security issues in SCLs? Does our policy say that backwards compat is more important than security issues?
No. We do not want to support not supportable (terribly broken) versions.
So... what do we do with them? Do we break backwards compatibility in those cases? Do we remove the SCL? what happens if the security issue is just in a piece of the SCL? For instance, if we have a thick SCL of ruby1.9 + rails3 and rails3 has an unresovled security issue, do we have an option to only drop the rails portion?
* Maybe Fedora simply isn't about backwards compat to thisextent and so we have to allow API breaks within a release.
No, it would loose the purpose of scl for projects building above it.
It would lose one part of the purpose but there are other reasons for scls to exist. ** Note: We passed the question of backwards compat to fesco at today's FPC meeting. So we should have this question answered externally at the next meeting. https://fedorahosted.org/fesco/ticket/1175
* Maybe the policy for SCLs in The Outer Rings should have astrict backwards compat section but SCLs in Fedora Commons are more relaxed (that would make the model: Fedora Commons SCLs are primarily for enabling different versions of dependencies for software that we want to ship in Fedora Commons; Outer Ring SCLs are primarily for enabling developers to target an SCL rather than Fedora as their platform.)
Rings are still unclear.
So are the guidelines for SCLs :-) But we have to anticipate that they're coming and figure out how the pieces all fit together.
* Only partially addressing the situation, could figure outsome way to mark some pieces of an SCL as having guaranteed backwards compat while others are only implementation detail. If you had an SCL that shipped ruby1.9 but it needed a newer version of OpenSSL you might ship that inside the SCL at first. But you wouldn't want people to rely on that particular version being available in the SCL in the future (once the base OS was updated to include that version.)
I might misunderstand. Do you suggest collection should "bundle" such package as openssl? That's security hole, not good for Fedora in circle 1, 2 or 3.
I don't suggest it "should". I suggest it is a requirement of enabling scls (at least as it was explained to me). An SCL provides a layer that can be used as compatibility between OS releases and even between OS's. The further the bases you are targetting get from each other, the more packages that you may need to include in order for your SCL to function on all of those platforms. So at some point we aren't going to be able to ship the same backwards compatible SCL on a newer version of Fedora without also shipping something that it depends on. My example above was openssl but the dependent package could be either better or worse.
If we don't want SCLs to include thngs they depend on we need to put in criteria to that effect. Possibilities: * SCLs must be built on other packages from the system. they cannot include packages that they depend on. * SCLs must not include any of the following packages: * a libc implementation * crypto implementations (bindings to system crypto libraries are okay but be careful -- if the API of the library changes in the future and those details leak through to the bindings you are giving to users of your SCL the SCL may need to be retired because it would be a backwards incompatible change.
* No matter where it is, need to have some criteria aroud howto obsolete an SCL, how to remove an SCL altogether, and whether two separate SCLs can ship largely the same thing.
Collection can be obsoleted same way as other packages, but let's say that maintainer should say, he do not want to support it for F-22 anymore (before its release). Those still interested could carry on.
It's more difficult than this though... Let's say we have a ruby1.9 + rails3 SCL. No one wants to maintain this any longer. Someone else would like to maintain a ruby1.9 without rails3 SCL. Are they allowed to do so? Can it take the old SCL's name? Does it have a new name? If so, what does that new name need to be? If a new maintainer could do this, is there anything stopping the old maintainer from doing the same thing in order to make their backwards incompatible change? removing an SCL: Does an SCL get removed from all releases of Fedora or only from the rawhide branch? Is the name available for reuse? Immediately? After the last Fedora Release it was present in has gone EOL?
- (toshio) Can we make a recommendation on how or when to use
scl-dependencies. For example, in the Rails4 example above, there may be scenarios that we want to allow this in Fedora. Although, the recommendation could be that frameworks and the like should be provided outside of Fedora Commons.
Do you mean dependencies among collections?
Yep.
== Naming ==
- problem with the examples (ruby193 and examples from:
http://fedorahosted.org/SoftwareCollections ) For instance, python3.3 would conflict with a python3.3 parallel stack that did not use scls.
Not true. We used python33 as a metapackage without dot.
Uh huh. There's two reasons I'm going to say this argument doesn't hold water: * The naming guidelines allow versions in package names to either use the dot or drop it (mostly historical... dot used to be prohibited but whenwe were presented with the fact that versions were being put into names of compat packages, we voted to allow dots if those were part of versions. But the current guidelines don't prohibit not using a dot in versions either: "Note that we do not use delimiters in the name in this situation. We attach the version number to the name, optionally removing the dot from the version."). * Officially promote package names that differ only in whether they have a dot or not is a road to madness: You'll have people installing the wrong package, misfiling bugs, blogging instructions with the wrong name, looking up ownership information of the wrong component, etc etc etc. It's bad enough when we have upstreams that choose similar names (ddrescue vs dd_rescue) that we're not going to make Fedora Policy that canonicalizes doing the same thing.
- Heard about namespace prefixes and that there were thoughts to name
them after vendors (rht-ruby1.9). If we use /opt/$vendor that might make sense.
- Could still have clashes with things like redhat-release,
redhat-rpm-config, etc... make sure the vendors we use doesn't have any pre-existing conflicts of this sort.
- scl makes sense as well. mmaslano had said that end-users wouldn't
know what an scl was but remi says you'd need to use a tool called "scl" to manage which scls are active anyway. So that may not be a problem?
That's true. We were also thinking about users who are interested in app and know nothing about scl, so they just install app, run it and are not aware of scl at all.
Ah, yeah -- so that would tie into whether we want to allow packages which are not scls to depend on scls. If we don't allow that, then an application that depended on the version of a stack in an scl (puppet depending on ruby1.9.3, for instance).would have to be in its own scl. And then the package name would be scl-puppet.... OTOH, there's also the desire by langdon that the SCLs store as few resources as possible outside of /opt/vendor/scl. If we follow that, then the user would still have to scl enable the puppet scl to get it working. (OTOH, we could decide that is a non-goal and ship a wrapper).
In my vision of the non-scl-depending-on-scl case a wrapper would be used so invoking puppet will scl enable the ruby1.9 scl before running puppet.
- If we have multi-vendor for scl (fedora, rpmfusion, local admin
case again), then vendor may make more sense again.
- Change example naming: ruby193 => ruby1.9.3 Dots are clearer and
thus preferred
Dots and such stuff are evil. I guess dots were omitted on purpose.
Dots in version are less confusing. Is ruby111 Ruby-11.1, Ruby-1.1.1 or Ruby-1.11? The guidelines allow non-dots in versions of mainstream packages because dots were on the prohibited list before and it proved confusing. We shouldn't make the same mistake with a brand new area.
== File Location ==
- If we're going to use /opt/vendor:
- scls should then obey the rest of the standard: /var/opt and
/etc/opt are for host specific state and config. Is that the case? can it be fixed if not?
Sometimes were used locations in /var/log or /etc because collections wouldn't work in case of mariadb and other daemons. We had to put init file somewhere and also logfile. It's documented and we didn't find better solution yet.
<nod> I think we can't avoid storing some files in /var and /etc. We have to assume that /opt follows the same pattern (shared between hosts; read-only) as /usr which makes it unsuitable for some files. As part of this question I was wondering if we're following the FHS specification to use /var/opt/ and /etc/opt/ instead of things in either the "normal" /etc/ and /var/ or in the /opt/vendor/scl hierarchy and if we aren't whether it would be possible to do so.
- What does the filesystem layout look like inside of this prefix?
/opt/$vendor/scl/rht-ruby1.9.3/{normal filesystem}?
yes, normal filesystem.
and it's /opt/$vendor/scl ?
I'm checking that scls are less likely to conflict with other vendor packages.
- Does this allow local sites and addon repos to make their own
vendor? How? (Thinking of rhel/centos, epel, and local admin here. Or fedora/rpmfusion, and local admin)
Modify vendor in macros of scl.
Is this the macros inside the SCL spec file? Inside the scl-utils package? Or elsewhere? Is it one macro to change and it affects everything else?
== Spec Files ==
- Why don't we follow the mingw model and have separate spec files
(mingw is both separate spec and separate package) for scl? Advantages:
- Many fewer conditionals in spec files (greater readability/maintainabilty)
- Primary maintainer doesn't need to know or care about SCLs if
other people do
- SCL could be the only spec that is necessary or desired for a
particular component
- Can a SCL spec be "only" an SCL spec (will not build a non-scl
rpm)? If so, we need some more content describing this scenario.
- Propose moving the "style guide" to before the example (e.g.
Section 3 before Section 2), so that some of the following questions don't come up for the reader
- Why are there so many macros that mean the same thing? -- %scl,
%pkg_name, %scl_name, %name
I believe we should have branch for scl and other branches might voluntarily add macros, but it shouldn't be mandatory. For example perl or httpd don't have nice specfile and adding more condition wouldn't improve readability. Having scl macros might improve speed of patching some packages, but not all.
With this and the other information about how you're currently building I'm starting to think that following the mingw model of separate packages is a better choice than doing these in the same spec file in the same git tree. But I will admit that I don't understand all of what you are currently doing so I'm not 100% settled on this opinion yet.
== Design ==
- What directories is an SCL allowed to modify?
- Naming of packages is a problem as currently designed. Liklihood of
conflict with native packages is very high. Propose to include "scl-" at the begining of names.
Why don't we already prefixed scl before all packages? Because it would be scl-ruby193-rubygems-blabla-1.1.f21. The prefix must not conflict with existing packages. It doesn't have to be ruby193, it might be name which specify what is inside.
I'm not sure what you're getting at here? Could you give some full examples to explain?
- Include "dots" in names, e.g. ruby1.9 rather than ruby19
I prefer without dots, punctuation does everything harder.
I explained the rationale for dots earlier. Could you write some justification for how leaving the separator in the version is "evil" or "harder".
== Build & Distribution ==
- What are the impacts on the build system?
Until now we used internal instance of koji. I can sum up what we did there.
Please do.
- How are we preventing build problems from the mainline package
from interfering with the scl package and vice versa?
There are no build interference problems. Package has specific name.
Not just name but also preventing changes to the spec file. From earlier answers, my undertanding is that you have a separate branch for each scl needing to be built for now.
- How are we allowing for a package to be built in more than one scl?
Didn't solve yet. We related build targets (collection) with branch. How to build for more collections is a good question for our releng. But do we need it?
Yeah, almost definitely. The idea I've heard is that dependent packages that are no longer provided by the system would need to be packaged inside of an scl. If they aren't in a separate scl then they would need to be packaged inside of each scl individually.
You could also have two different scls that require different versions of a third package.
Oh -- one note about terminology; we should avoid "collections" as a synonymous term with scls because collections and branches are already synonymous terms.
* How are we allowing for differences between scls? (ie:ruby1.9.3 scl with rubygem-foo1.0; ruby2.0 scl with rubygem-foo2.0)
- How are we preventing build problems from one scl from
interfering with a different scl?
- Remi mentions that he thinks it would be done in a separate
branch. Would we need a branch per scl that the package is built in?
Can't happen. Every branch is set to build in some tag. Packages tagged in one tag are shipped as one collection. You are able to inherit from one tag to another. As I said I prefer branch per collection. How else would be tags defined?
I'm afraid I'm not understanding the big picture here. Or perhaps the details. Anyhow -- I'm not understanding what you're doing and what you think is impossible and whether that's what Remi had in mind or not.
- How do SCLs get distributed?
- Is this a "separate repo" or is it just in the normal repo(s)?
Currently separate repo.
* (toshio) Both. Some SCLs can go into Fedora Commons whichwould continue to have one repo. SCLs in the outer rings would have some separate structure.
Agreed.
- Build section would benefit from minor sections, perhaps:
"Overview," "Testing" and "Example"? Sections need a bit more color to be understandable.
I'll add it.
First of all, thanks for doing so thorough review/comment summary. I really appreciate the amount of work you've put into this. In this mail, I'm going to respond just to the parts about the draft itself. I think Marcela covered the other parts pretty well and there is nothing I want to add there. I've done some changes in the draft according to your comments... please see the wiki page history to see the specific adjustments.
Comments on the SCL Draft: https://fedoraproject.org/wiki/User:Bkabrda/SCLGuidelinesDraft
- Can we assume that the implementation can be changed to address
problems, add features, or follow different conventions? Can be done if backwards compat is possible? Or is implementation set in stone so we'd have to kludge around any shortcomings?
So, this is a very general question, so my answer is: Depends. If we come up with a sane proposal, I think the scl-utils upstream will accept, but I can't really say that in general.
- One of the goals of SCLs is isolation from the rest of the system,
correct? It seems, though, that once you run the enable script, that shell would have a different PATH with the possibility of a different language interpreter. As long as scripts can use /usr/bin/env as a shebang or scripts are using PATH-based commands these will end up using programs from the SCL. Should FPC revisit banning these? Is it okay to run with SCLs even without banning? Just something to document?
So, I think we've already mentioned this during our Python-3-shift-discussion. My opinion on this is, that if you're packaging a script for system, it should run with system interpreter, regardless of current $PATH.
=== Specific comments on the draft ===
- Move template to the bottom of the draft. Had many questions after
reading the template that were eventually answered in other parts of the draft.
Do you mean the metapackage template?
- Style guide is not to have a period at the end of Summary
Fixed, sorry :)
- Description in template should probably be more clear that
packagers should fill in a proper description
I added a line with "Provide some useful info about this SCL." into the description.
- Template skips %prep and %build. Are these empty? If so, just
include the empty %prep and %build headers.
Yikes! The %prep section actually has to be there, because macros.scl use %{buildsubdir}, which is defined during %prep... fixed.
- "The -build subpackage should include Requires: scl-utils-build."
Why a should and not a must?
I guess I can take this one out, assuming that scl-utils-build package is always tagged in buildroot. The reason why I included the line about that Requires is, that each <scl>-build subpackage needs scl-utils-build to work (in the sense that having only <scl>-build without scl-utils-build will fail). So... some explicitness, maybe unnecessary.
- Content about the enable scriptlet implies that the usual path
macros (_bindir etc) are changed in the scl package. Need to make it clear what standard macros are being overridden to what. Perhaps consider "tables" like http://fedoraproject.org/wiki/Packaging:MinGW#Filesystem_location_macros but with slightly different columns, e,g, macro | override? | Normally Modified? | example | description
I don't think we'd need a table for this, since all the macros are just prefixed with "/opt/<provider>/<scl>/root/". I mentioned this in the draft.
- "The macros.%{scl}-config in %{scl}-build package contains the %scl
macro definition (name of the SCL). If you need to use some more macros for this SCL, add them into this file." <= but there's nothing in the spec file that obviously creates this. Need instructions on how to create this file for manual use and then how to include the file in the spec file and not have whatever automated macro normally creates it not overwrite it.
This macro file is created by %scl_install and it's location is %{buildroot}%{_root_sysconfdir}/rpm/macros.%{scl}-config. I mentioned that in the draft.
- "essential for its runtime". "Essential" needs to be explained and
hopefully it is just a link to the earlier part of the doc.
Rephrased.
- -runtime package must be arch specific => it contains
%{scl_prefix}%{_libdir} so it's arch dependent. (and this means the main package must be arch specific too)
Is this unclear in the draft? It seems pretty clear to me. Do you have a specific wording for the draft in mind?
- Creating the enable file in %install -- would rather see macros
like: - %scl_add_path %scl_add_ld_library_path %scl_add_manpath than the HERE-doc style thing that will cut and paste to most specs.
That's actually an interesting idea and should probably be proposed to upstream. I'm afraid that I can't do anything about this until upstream includes such macros.
- "set of macros that contain the values of the original RPM macros"
should be expanded on. naming of these could be better => establish a common prefix for anything that is being overridden instead of %pkg_name vs %_root_. also pkg_name probably conflicts with some existing packages. Maybe something like %_prescl_name, %_prescl_libdir, etc. [can change the recommendation from: packagers should define %pkg_name to packagers should define a macro that is the value of pkg_name => %{?scl: %global srcname %pkg_name and %{?!scl: %global srcname %name}
- I included an example of the original macro and redefined _root_* macro. The problem with the %pkg_name part is, that it has to be %pkg_name (unless upstream changes that), so that you're able to use one macro throughout your specfile. The name of the original macros are upstream thing. I myself am ok with them, but feel free to propose your idea to scl-utils upstream.
- Little more guidance needed for Requires. Maybe making the examples
more real-life would clarify things. ("Since ifconfig is a binary we're going to use the system version of, do not use scl_prefix with it. rubygem-foo, otoh, has to be in the path of the ruby interpreter which this scl is to be a part of. Therefore it needs to use scl_prefix"). Also, an example of a "Versioned Requires" would be helpful.
Done.
- Need more wording on Requires: of non-scl packages as they can act
as conflicts. Something like: Requires: foo = 1.2.3 … where the scl package is the only thing keeping foo back at that version. So you can't update "the OS" without updating the scl. wording might be something like "yes, you can require non-scl stuff … but you have to be very general in your requirements, and if you need something that might be updated you have to either scl bundle it or be willing to rebuild the scl when it updates".
I added a note about this.
- Should we list somewhere the scl-modified deps scripts? Otherwise
how will someone creating a new ocaml-scl know whether an ocaml-scl dep script has already been written?
Yep, we can probably add a section like this when we actually gather scripts like these.
- Please elaborate on the function of %scl_package_override()
I put a detailed explanation in admon/tip frame.
- In the dealing with macro files section: I think that the "Now
consider a situation[...]" starts a whole new scenario? Correct? If so, needs to be better phrased right now it's associating with the previous examples but it doesn't seem to make sense in that context.
Correct. I placed a subsubsubsection header before that - it should clearly separate the two scenarios.
- So If I've understood everything correctly, building in a standard
buildroot will get you a non-scl'd package. Building with the $scl-build package installed will build the scl version?
- Should say that in the first line of Building Packages
Yes, that is assuming you do everything right :) Done.
- In Building Packages: Don't need to include scl-utils-build
explicitly in the config_opts line (brought in via dependencies), correct?
Not really. Without scl-utils-build, you won't be able to build metapackage in Koji, as it will fail on composing SRPM, because it won't be able to parse specfile (Unknown tag: %scl_package).
== Conversion to SCLs ==
- Parts of this section are more "tips & tricks"? Should it be on a
non-guideline page?
Yes, it can be, I don't really mind. I just find it handy to be at the same place.
- Probably best to vette the advice anyway and then move what makes
sense to a separate page afterwards.
- Add information about why /usr/bin/env is not a generally good
thing in SCLs. If this is more general than SCLs, does FPC need to revisit a ban on /usr/bin/env in shebangs?
As mentioned above, I think this should be more general.
- sed line should limit to the first line of the file as well as the
shebang pattern.
Limited.
- how does scl deal with compiled-in paths inside of binaries? Does
it depend on %configure to do the right thing with %{_libdir} etc? If so we should point out to people that old-school builds which aren't configure driven may need patches (actually, this applies to install location as well as compiled-in paths and to scripts that have file paths embedded as well as binaries)
SCL builds depend on not-hardcoded paths, so some amount of patching may be needed. I added an admon/caution about this to guidelines.
- How do auto provides and requires for elf libraries work?
Problematically :) Filters for these (prepending "%{scl}-") have to be provided by collection packagers). I know that there was a scl-utils bug for this, but I can't find it right now. I'll put this on my TODO list and find it out.
== Inter-SCL Dependencies ==
- Example of using this with versioned Requires?
Added.
- The %scl_require example just doesn't seem right somehow.... The
ruby193 scl brings in rails-3.2.3? That seems like poor naming (if the metapackage brings in rails, then it should be named rails. OTOH, if the package purpose is to provide ruby193 then rails should be optional not essential.) or poor dependency management (Which would be part of the higher level questions of what can be an SCL and what can be inside an SCL).
Poor naming. I changed the scl name in the example to rails323.
- Are non-scl packages allowed to dep on scl packages? If so, how do
they do so?
Generally, we should keep dependencies of non-scl packages on scl packages at minimum, although it does make sense in some cases - for example, user just wants to install OpenStack and he doesn't care if it depends on a collection or not, so it makes perfect sense to have "openstack" package depend on SCL. On the other hand, this can quickly become a terrible mess, if not done cautiously. I'll try to think about some wording when this is allowed and when not.
- The filters don't work in EL6 thing.... people seem more excited
about SCLs in EPEL than in fedora so we probably need to document how to filter the provides and requires for EL6. However, we can point to the EPEL6 Differences page from here and document the behaviour there.
Yes, that is a good idea. I see that Marcela has already added a reference to a bug that explains that filters don't work in epel. I added link to the wiki page that describes filtering on epel.
== General Comments ==
- No definition of a SCL appears in the document
- Example text: Software Collections (SCLs) give rpm packagers a
means of packaging multiple versions of software for a single distribution. The version from the software collection does not interact with the system version. They can provide backwards and forwards compat versions of a package compared to what's on the system.
- Include "definitions" at the beginning of the document, some
example things that need definition (and suggestions)
- SCL -> whole software collection? <- originally the term was
"dynamic software collection" for the "whole thing" but it has fallen in to disuse... we could bring it back
- SCL defining package -> the srpm-level package that defines the scl.
- SCL metapackage -> built from the SCL defining package. defines
the essential packages you get when you install the scl.
- SCL runtime package -> built from the SCL defining package. It's
the setup of the system for this SCL including registering the SCL with the system and setting up the filesystem.
- SCL build package -> built from the SCL defining package. It
contains macro definitions needed to build an SCL.
- SCL package -> any package which can be built for an SCL.
- Elaborate on what "belongs" in the "-runtime" package
- Extracting one of these from the sample repository seems like
it's a filesystem package directory tree inside of the scl? Anything else?
I added a general section that contains some definitions and explanations.
-Toshio
Thanks, Slavek.
Le 19/09/2013 13:28, Bohuslav Kabrda a écrit :
https://fedoraproject.org/wiki/User:Bkabrda/SCLGuidelinesDraft
If you use any of the above, that contains %{_libdir}, the metapackage must be arch specific, otherwise it may be noarch.
I think the main scl package MUST always be arched.
With scl-utils-20121110-1, trying to build a package noarch SCL package on a x86_64 system result with an error:
Installed (but unpackaged) files found: /opt/rh/foo/root/lib64
If this is a bug in scl-utils, we should add
"If all the packages in the SCL are noarch ..." (else /usr/lib64 will be owned).
Remi.
----- Original Message -----
Le 19/09/2013 13:28, Bohuslav Kabrda a écrit :
https://fedoraproject.org/wiki/User:Bkabrda/SCLGuidelinesDraft
If you use any of the above, that contains %{_libdir}, the metapackage must be arch specific, otherwise it may be noarch.
I think the main scl package MUST always be arched.
With scl-utils-20121110-1, trying to build a package noarch SCL package on a x86_64 system result with an error:
Installed (but unpackaged) files found: /opt/rh/foo/root/lib64
If this is a bug in scl-utils, we should add
"If all the packages in the SCL are noarch ..." (else /usr/lib64 will be owned).
Remi.
Ok, I just tested and it seems that this is an actual bug. I reported it here: https://bugzilla.redhat.com/show_bug.cgi?id=1009921
Converting Packages for SCL The resulting package should be buildable even without SCL
I Agree, only "should". Especially as some packager could prefer to have a SCL specific spec (there is no rule which imply to use the same spec in "all" branches. Generic spec could be useful for some simple package, but sometime, having too much conditional is really bad for readbility).
So, in
Every SCL specfile must have %scl_package macro specified
Agree
(like any other macro, it must be conditionalized). this:
Disagree: I prefer "should" (if I don't care of a generic spec)
Remi.
----- Original Message -----
SCL Metapackage %_scl_prefix /opt/myorganization
%_scl_prefix is a distro choice.
I don't think packager should have to define it. (Guildelines are for Fedora specific Guildelines)
Agreed. I'm not sure if there is an agreement on what "myorganization" should be (I guess we'll use "fedora", right?). If/when there is an agreement, I'll open bug for scl-utils maintainer to patch this downstream, so that it's defined for all Fedora SCLs. Does that sound good?
Le 19/09/2013 15:09, Bohuslav Kabrda a écrit :
----- Original Message -----
SCL Metapackage %_scl_prefix /opt/myorganization
%_scl_prefix is a distro choice.
I don't think packager should have to define it. (Guildelines are for Fedora specific Guildelines)
Agreed. I'm not sure if there is an agreement on what "myorganization" should be (I guess we'll use "fedora", right?).
This is another discussion ;)
If/when there is an agreement, I'll open bug for scl-utils maintainer to patch this downstream, so that it's defined for all Fedora SCLs. Does that sound good?
Yes.
Dealing With Macro Files
The macro files must be renamed by appending .%{scl} to their name...
Agree
But in the example
This is fine: %__python2 %{_bindir}/python
This is obviously not fine, it doesn't respect previous rule ;)
The following examples (with %__python26_python2) are better.
Remi.
----- Original Message -----
Dealing With Macro Files
The macro files must be renamed by appending .%{scl} to their name...
Agree
But in the example
This is fine: %__python2 %{_bindir}/python
This is obviously not fine, it doesn't respect previous rule ;)
You're right. Fixed - is this [1] better?
The following examples (with %__python26_python2) are better.
Remi.
Slavek.
[1] https://fedoraproject.org/w/index.php?title=User%3ABkabrda%2FSCLGuidelinesDr...
Dealing With Automatic Provides/Requires and Filtering
%{?scl:%filter_from_requires s|pkgconfig|%{?scl_prefix}pkgconfig|g}
This is only an example, but probably we need to notice user that filering must be done carrefully, not using a too large criteria: you can requires a pkgconfig(foo) from system and a pkgconfig(bar) from the scl.
Need also some proposal about the libraries, when the provided soname exists in the system which can create awful dependencies issues.
Remi.
----- Original Message -----
Dealing With Automatic Provides/Requires and Filtering
%{?scl:%filter_from_requires s|pkgconfig|%{?scl_prefix}pkgconfig|g}
This is only an example, but probably we need to notice user that filering must be done carrefully, not using a too large criteria: you can requires a pkgconfig(foo) from system and a pkgconfig(bar) from the scl.
I added a warning about this [1]
Need also some proposal about the libraries, when the provided soname exists in the system which can create awful dependencies issues.
Yeah, I'll try to come up with something.
Remi.
Slavek.
[1] https://fedoraproject.org/w/index.php?title=User%3ABkabrda%2FSCLGuidelinesDr...
packaging@lists.fedoraproject.org
-
Bohuslav Kabrda -
Dave Johansen -
Marcela Mašláňová -
Ralf Corsepius -
Remi Collet -
Toshio Kuratomi -
White, Langdon