Hi,
Can somebody enlighten me, what is the purpose of ruby(abi) (replace by python(abi) if you wish) virtual provide? Especially, why Ruby packaging guidelines mandate "Requires: ruby(abi) = 1.9.1", i.e. versioned require? And why in Python packages, python(abi) is automatically generated?
If the package is noarch, there is very high chance, that it would work with Ruby 1.8 as good as with Ruby 1.9 or even Ruby 2.0. If the package is arch dependent, it has automatically generated dependency on libruby.so.1.9. So there is no chance to install it or run it with other version of Ruby.
Now why I am asking? We would like to have in F19 Ruby 2.0 accompanied by JRuby 1.7 and in the future, we would love to see these interpreters interchangeable, i.e. it doesn't matter which interpreter is on your system, since it will be able to run the gem/application shipped in Fedora. However, while Ruby 2.0 should provide "ruby(abi) = 2.0", JRuby are not yet fully 2.0 ready, so they should not provide "ruby(abi) = 2.0". Moreover, the term ABI with JRuby is a bit misleading.
So how we can make every noarch ruby package compatible with both interpreters in terms of requires? One possibility would be to drop the abi version unless it is explicitly needed. However, it renders ruby(abi) as not the best virtual provide name for this goal.
Any ideas?
Vít
Vít Ondruch (vondruch@redhat.com) said:
Hi,
Can somebody enlighten me, what is the purpose of ruby(abi) (replace by python(abi) if you wish) virtual provide? Especially, why Ruby packaging guidelines mandate "Requires: ruby(abi) = 1.9.1", i.e. versioned require? And why in Python packages, python(abi) is automatically generated?
In the python case, it's because that python extension modules install in a version-specific directory ($libdir/python2.7, for example.) This makes them explicitly tied to that version of python.
Bill
Dne 19.12.2012 14:12, Bill Nottingham napsal(a):
Vít Ondruch (vondruch@redhat.com) said:
Hi,
Can somebody enlighten me, what is the purpose of ruby(abi) (replace by python(abi) if you wish) virtual provide? Especially, why Ruby packaging guidelines mandate "Requires: ruby(abi) = 1.9.1", i.e. versioned require? And why in Python packages, python(abi) is automatically generated?
In the python case, it's because that python extension modules install in a version-specific directory ($libdir/python2.7, for example.) This makes them explicitly tied to that version of python.
Well, it should be better to use "python(filesystem) = 2.7-1" or something like that IMO. It turns out that for integration of JRuby, even if we would stay with Ruby 1.9.3, we would need to change filesystem layout and ruby(abi) cannot reflect it. I feel that the above mentioned usecase is abuse of python(abi).
Vít
Bill
packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
On Wed, 2012-12-19 at 14:20 +0100, Vít Ondruch wrote:
Dne 19.12.2012 14:12, Bill Nottingham napsal(a):
Vít Ondruch (vondruch@redhat.com) said:
Hi,
Can somebody enlighten me, what is the purpose of ruby(abi) (replace by python(abi) if you wish) virtual provide? Especially, why Ruby packaging guidelines mandate "Requires: ruby(abi) = 1.9.1", i.e. versioned require? And why in Python packages, python(abi) is automatically generated?
It seems to me that caution is required when comparing Ruby and Python in this context. Python offers very strong ABI guarantees within a minor release, whereas iitc Ruby appears to offer ABI guarantees only within a micro release (forgive me if I'm wrong here, I'm much more familiar with Python than Ruby).
What's good for Ruby may not necessarily be good for Python, and vice versa.
In the python case, it's because that python extension modules install in a version-specific directory ($libdir/python2.7, for example.) This makes them explicitly tied to that version of python.
Well, it should be better to use "python(filesystem) = 2.7-1" or something like that IMO. It turns out that for integration of JRuby, even if we would stay with Ruby 1.9.3, we would need to change filesystem layout and ruby(abi) cannot reflect it. I feel that the above mentioned usecase is abuse of python(abi).
Using "2.7-1" would be wrong.
Python has two different binary formats with guarantees: bytecode (pyc files) and the ABI of .so files. The latter is covered by autogenerated rpm deps extracted from DSO dependencies, so e.g.:
$ rpm -qR gdb | grep python libpython2.7.so.1.0()(64bit)
but the .pyc one isn't, hence the need for the "python(abi) = " metadata.
Both of these binary formats are guaranteed to be consistent within a minor release (such as "2.7" or "3.3"). They don't change with the microrelease (2.7.3) or with the rpm version ("2.7-1").
We don't need to rebuild every python package when python itself is rebuilt; we only need to do this when updating to a new minor release (2.6 -> 2.7, or 3.2 -> 3.3).
Hope this is helpful Dave
Dne 21.12.2012 17:34, David Malcolm napsal(a):
On Wed, 2012-12-19 at 14:20 +0100, Vít Ondruch wrote:
Dne 19.12.2012 14:12, Bill Nottingham napsal(a):
Vít Ondruch (vondruch@redhat.com) said:
Hi,
Can somebody enlighten me, what is the purpose of ruby(abi) (replace by python(abi) if you wish) virtual provide? Especially, why Ruby packaging guidelines mandate "Requires: ruby(abi) = 1.9.1", i.e. versioned require? And why in Python packages, python(abi) is automatically generated?
It seems to me that caution is required when comparing Ruby and Python in this context. Python offers very strong ABI guarantees within a minor release, whereas iitc Ruby appears to offer ABI guarantees only within a micro release (forgive me if I'm wrong here, I'm much more familiar with Python than Ruby).
What's good for Ruby may not necessarily be good for Python, and vice versa.
In the python case, it's because that python extension modules install in a version-specific directory ($libdir/python2.7, for example.) This makes them explicitly tied to that version of python.
Well, it should be better to use "python(filesystem) = 2.7-1" or something like that IMO. It turns out that for integration of JRuby, even if we would stay with Ruby 1.9.3, we would need to change filesystem layout and ruby(abi) cannot reflect it. I feel that the above mentioned usecase is abuse of python(abi).
Using "2.7-1" would be wrong.
Python has two different binary formats with guarantees: bytecode (pyc files) and the ABI of .so files. The latter is covered by autogenerated rpm deps extracted from DSO dependencies, so e.g.:
$ rpm -qR gdb | grep python libpython2.7.so.1.0()(64bit)
but the .pyc one isn't, hence the need for the "python(abi) = " metadata.
Both of these binary formats are guaranteed to be consistent within a minor release (such as "2.7" or "3.3"). They don't change with the microrelease (2.7.3) or with the rpm version ("2.7-1").
We don't need to rebuild every python package when python itself is rebuilt; we only need to do this when updating to a new minor release (2.6 -> 2.7, or 3.2 -> 3.3).
Hope this is helpful Dave
Thank you Dave,
You confirmed my conclusion from my other email [1]. I guess you will get there later ;)
Vít
[1] http://lists.fedoraproject.org/pipermail/packaging/2012-December/008816.html
On Fri, Dec 21, 2012 at 11:34:52AM -0500, David Malcolm wrote:
Both of these binary formats are guaranteed to be consistent within a minor release (such as "2.7" or "3.3"). They don't change with the microrelease (2.7.3) or with the rpm version ("2.7-1").
Does python make any promises about other changes within a minor release? For instance, changes to the language itself or changes to the stdlib API? I know that I've seen changes rejected for a minor release series upstream because they introduced those types of changes but I'm not sure how strict that is and what it covers?
-Toshio
On 2012-12-19 5:12, Bill Nottingham wrote:
Vít Ondruch (vondruch@redhat.com) said:
Can somebody enlighten me, what is the purpose of ruby(abi) (replace by python(abi) if you wish) virtual provide? Especially, why Ruby packaging guidelines mandate "Requires: ruby(abi) = 1.9.1", i.e. versioned require? And why in Python packages, python(abi) is automatically generated?
In the python case, it's because that python extension modules install in a version-specific directory ($libdir/python2.7, for example.) This makes them explicitly tied to that version of python.
There's also the fact that the ABI for the bytecode that gets generated at build time is specific to each x.y series of python releases.
-- Garrett Holmstrom
Dne 20.12.2012 01:43, Garrett Holmstrom napsal(a):
On 2012-12-19 5:12, Bill Nottingham wrote:
Vít Ondruch (vondruch@redhat.com) said:
Can somebody enlighten me, what is the purpose of ruby(abi) (replace by python(abi) if you wish) virtual provide? Especially, why Ruby packaging guidelines mandate "Requires: ruby(abi) = 1.9.1", i.e. versioned require? And why in Python packages, python(abi) is automatically generated?
In the python case, it's because that python extension modules install in a version-specific directory ($libdir/python2.7, for example.) This makes them explicitly tied to that version of python.
There's also the fact that the ABI for the bytecode that gets generated at build time is specific to each x.y series of python releases.
For that, you could have "Require: python-libs = 2.7" instead.
Vít
-- Garrett Holmstrom -- packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
On 12/20/2012 02:40 AM, Vít Ondruch wrote:
Dne 20.12.2012 01:43, Garrett Holmstrom napsal(a):
On 2012-12-19 5:12, Bill Nottingham wrote:
Vít Ondruch (vondruch@redhat.com) said:
Can somebody enlighten me, what is the purpose of ruby(abi) (replace by python(abi) if you wish) virtual provide? Especially, why Ruby packaging guidelines mandate "Requires: ruby(abi) = 1.9.1", i.e. versioned require? And why in Python packages, python(abi) is automatically generated?
In the python case, it's because that python extension modules install in a version-specific directory ($libdir/python2.7, for example.) This makes them explicitly tied to that version of python.
There's also the fact that the ABI for the bytecode that gets generated at build time is specific to each x.y series of python releases.
For that, you could have "Require: python-libs = 2.7" instead.
What's the practical difference?
-- rex
Dne 20.12.2012 15:46, Rex Dieter napsal(a):
On 12/20/2012 02:40 AM, Vít Ondruch wrote:
Dne 20.12.2012 01:43, Garrett Holmstrom napsal(a):
On 2012-12-19 5:12, Bill Nottingham wrote:
Vít Ondruch (vondruch@redhat.com) said:
Can somebody enlighten me, what is the purpose of ruby(abi) (replace by python(abi) if you wish) virtual provide? Especially, why Ruby packaging guidelines mandate "Requires: ruby(abi) = 1.9.1", i.e. versioned require? And why in Python packages, python(abi) is automatically generated?
In the python case, it's because that python extension modules install in a version-specific directory ($libdir/python2.7, for example.) This makes them explicitly tied to that version of python.
There's also the fact that the ABI for the bytecode that gets generated at build time is specific to each x.y series of python releases.
For that, you could have "Require: python-libs = 2.7" instead.
What's the practical difference?
You follow general practices? You don't have to think if {ruby,python}(abi) makes any sense for {JRuby,Jython} and what version it should provide in comparison to {ruby,python}. You don't force people to ask why Ruby 1.9.3 has ruby(abi) = 1.9.1. You don't have to answer such question as what is {ruby,python}(abi) good for?
Vít
-- rex
-- packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
On 12/20/2012 05:40 PM, Vít Ondruch wrote:
Dne 20.12.2012 15:46, Rex Dieter napsal(a):
On 12/20/2012 02:40 AM, Vít Ondruch wrote:
Dne 20.12.2012 01:43, Garrett Holmstrom napsal(a):
On 2012-12-19 5:12, Bill Nottingham wrote:
Vít Ondruch (vondruch@redhat.com) said:
Can somebody enlighten me, what is the purpose of ruby(abi) (replace by python(abi) if you wish) virtual provide? Especially, why Ruby packaging guidelines mandate "Requires: ruby(abi) = 1.9.1", i.e. versioned require? And why in Python packages, python(abi) is automatically generated?
In the python case, it's because that python extension modules install in a version-specific directory ($libdir/python2.7, for example.) This makes them explicitly tied to that version of python.
There's also the fact that the ABI for the bytecode that gets generated at build time is specific to each x.y series of python releases.
For that, you could have "Require: python-libs = 2.7" instead.
What's the practical difference?
You follow general practices?
Well, general practise happens to be virtual dependencies, not package names.
You don't have to think if {ruby,python}(abi) makes any sense for {JRuby,Jython} and what version it should provide in comparison to {ruby,python}. You don't force people to ask why Ruby 1.9.3 has ruby(abi) = 1.9.1. You don't have to answer such question as what is {ruby,python}(abi) good for?
The point of virtual provides is that they are not dependent of actual package names and so are not subject to issues like distro naming policies (and changes in them), package splits etc. This happens to be pretty much a must for automatically generated dependencies.
For example python-libs got split out of python not that many releases ago, but python(abi) dependencies avoided the need to modify and rebuild every bleeping python package because of it. On top of the other already explained rationale for python(abi) existence, that is.
As for ruby(abi), I dont know the exact rationale. I'd be mildly surprised if it existed "just because" though.
- Panu -
Dne 20.12.2012 19:10, Panu Matilainen napsal(a):
On 12/20/2012 05:40 PM, Vít Ondruch wrote:
Dne 20.12.2012 15:46, Rex Dieter napsal(a):
On 12/20/2012 02:40 AM, Vít Ondruch wrote:
Dne 20.12.2012 01:43, Garrett Holmstrom napsal(a):
On 2012-12-19 5:12, Bill Nottingham wrote:
Vít Ondruch (vondruch@redhat.com) said: > Can somebody enlighten me, what is the purpose of ruby(abi) > (replace > by python(abi) if you wish) virtual provide? Especially, why Ruby > packaging guidelines mandate "Requires: ruby(abi) = 1.9.1", i.e. > versioned require? And why in Python packages, python(abi) is > automatically generated?
In the python case, it's because that python extension modules install in a version-specific directory ($libdir/python2.7, for example.) This makes them explicitly tied to that version of python.
There's also the fact that the ABI for the bytecode that gets generated at build time is specific to each x.y series of python releases.
For that, you could have "Require: python-libs = 2.7" instead.
What's the practical difference?
You follow general practices?
Well, general practise happens to be virtual dependencies, not package names.
You don't have to think if {ruby,python}(abi) makes any sense for {JRuby,Jython} and what version it should provide in comparison to {ruby,python}. You don't force people to ask why Ruby 1.9.3 has ruby(abi) = 1.9.1. You don't have to answer such question as what is {ruby,python}(abi) good for?
The point of virtual provides is that they are not dependent of actual package names and so are not subject to issues like distro naming policies (and changes in them), package splits etc. This happens to be pretty much a must for automatically generated dependencies.
For example python-libs got split out of python not that many releases ago, but python(abi) dependencies avoided the need to modify and rebuild every bleeping python package because of it. On top of the other already explained rationale for python(abi) existence, that is.
As for ruby(abi), I dont know the exact rationale. I'd be mildly surprised if it existed "just because" though.
I appreciate virtual provides (although FPC dropped rubygem(foo) virtual provides from Ruby packaging guidelines :/ ). The thing is that no virtual provide can save you in case you discover, that its name does no more describe it purpose. So for Ruby MRI and JRuby, we should probably introduce ruby(interpreter) or ruby(compatibility) virtual provide instead of ruby(abi).
Vít
- Panu --- packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
On Thu, Dec 20, 2012 at 04:40:17PM +0100, Vít Ondruch wrote:
Dne 20.12.2012 15:46, Rex Dieter napsal(a):
On 12/20/2012 02:40 AM, Vít Ondruch wrote:
Dne 20.12.2012 01:43, Garrett Holmstrom napsal(a):
On 2012-12-19 5:12, Bill Nottingham wrote:
Vít Ondruch (vondruch@redhat.com) said:
Can somebody enlighten me, what is the purpose of ruby(abi) (replace by python(abi) if you wish) virtual provide? Especially, why Ruby packaging guidelines mandate "Requires: ruby(abi) = 1.9.1", i.e. versioned require? And why in Python packages, python(abi) is automatically generated?
In the python case, it's because that python extension modules install in a version-specific directory ($libdir/python2.7, for example.) This makes them explicitly tied to that version of python.
There's also the fact that the ABI for the bytecode that gets generated at build time is specific to each x.y series of python releases.
For that, you could have "Require: python-libs = 2.7" instead.
What's the practical difference?
You follow general practices? You don't have to think if {ruby,python}(abi) makes any sense for {JRuby,Jython} and what version it should provide in comparison to {ruby,python}. You don't force people to ask why Ruby 1.9.3 has ruby(abi) = 1.9.1. You don't have to answer such question as what is {ruby,python}(abi) good for?
I don't think I follow your reasoning here:
1) If there is no python(abi) and no automatic dependency checking, then the packager assumes the burden of adding Requires: python-libs = VERSION in all of their packages. They also assume the responsibility of figuring out what VERSION needs to be. This is creating more work that should be done centrally.
2) If there is no python(abi) but the automatic dep checking creates the Provides and Requires on python-libs = VERSION, then there doesn't seem to be any difference in terms of the work done. Someone who maintains the central dep checking still has to figure out what VERSION should be in either case. (And, assuming that your question about Ruby 1.9.3 having ruby(abi) = 1.9.1 was more than hypothetical, it sounds like there might just be a lack of someone performing this duty for Ruby).
-Toshio
----- Original Message -----
On Thu, Dec 20, 2012 at 04:40:17PM +0100, Vít Ondruch wrote:
Dne 20.12.2012 15:46, Rex Dieter napsal(a):
On 12/20/2012 02:40 AM, Vít Ondruch wrote:
Dne 20.12.2012 01:43, Garrett Holmstrom napsal(a):
On 2012-12-19 5:12, Bill Nottingham wrote:
Vít Ondruch (vondruch@redhat.com) said: >Can somebody enlighten me, what is the purpose of ruby(abi) >(replace >by python(abi) if you wish) virtual provide? Especially, why >Ruby >packaging guidelines mandate "Requires: ruby(abi) = 1.9.1", >i.e. >versioned require? And why in Python packages, python(abi) is >automatically generated?
In the python case, it's because that python extension modules install in a version-specific directory ($libdir/python2.7, for example.) This makes them explicitly tied to that version of python.
There's also the fact that the ABI for the bytecode that gets generated at build time is specific to each x.y series of python releases.
For that, you could have "Require: python-libs = 2.7" instead.
What's the practical difference?
You follow general practices? You don't have to think if {ruby,python}(abi) makes any sense for {JRuby,Jython} and what version it should provide in comparison to {ruby,python}. You don't force people to ask why Ruby 1.9.3 has ruby(abi) = 1.9.1. You don't have to answer such question as what is {ruby,python}(abi) good for?
I don't think I follow your reasoning here:
- If there is no python(abi) and no automatic dependency checking,
then the packager assumes the burden of adding Requires: python-libs = VERSION in all of their packages. They also assume the responsibility of figuring out what VERSION needs to be. This is creating more work that should be done centrally.
- If there is no python(abi) but the automatic dep checking creates
the Provides and Requires on python-libs = VERSION, then there doesn't seem to be any difference in terms of the work done. Someone who maintains the central dep checking still has to figure out what VERSION should be in either case. (And, assuming that your question about Ruby 1.9.3 having ruby(abi) = 1.9.1 was more than hypothetical, it sounds like there might just be a lack of someone performing this duty for Ruby).
-Toshio
I think that one of the questions here is "why 'abi' and not something else?" We currently enforce every Ruby package to have "Requires: ruby(abi)", but JRuby doesn't in fact provide any ABI (I don't think the Java API can be called ABI). Therefore it doesn't make sense for the pure-Ruby packages to require ruby(abi). It'd be better to have something that would explain that all they need is _anything_ that can interpret Ruby, e.g. "ruby(interpreter)". Packages requiring ruby(abi) should IMHO be only these that have C extension or are otherwise bound to cRuby. Similarly, JRuby-only packages should depend on jruby (or its virtual provide, if we choose to create one).
On Dec 20, 2012 11:12 PM, "Bohuslav Kabrda" bkabrda@redhat.com wrote:
I think that one of the questions here is "why 'abi' and not something
else?"
We currently enforce every Ruby package to have "Requires: ruby(abi)",
but JRuby doesn't in fact provide any ABI (I don't think the Java API can be called ABI). Therefore it doesn't make sense for the pure-Ruby packages to require ruby(abi). It'd be better to have something that would explain that all they need is _anything_ that can interpret Ruby, e.g. "ruby(interpreter)". Packages requiring ruby(abi) should IMHO be only these that have C extension or are otherwise bound to cRuby. Similarly, JRuby-only packages should depend on jruby (or its virtual provide, if we choose to create one).
This sounds like a productive piece of discussion. So is sounds like you desire to create as few packages as possible to create a full stack for c-ruby, jruby, and other implementations of the ruby language.
What rpm dependencies are used for are to guarantee that when a user installs a package, the things that package needs to run are also installed.
( Note: If I use "when" below, i'm asking what conventions exist to map those types of changes to bumps in the version number of upstream releases.)
It's when we put these two things together that things start to get tricky. When does the ruby language itself change? (For instance, if blocks were only specified with curly braces in one version but could be specified with either curly braces or parentheses in a later version.) (Another for instance, if a new keyword is added to the language). Is the ruby stdlib shipped separately from the interpreter to users? Are types part of the stdlib or the interpreter? When can those be added or changed? These are examples of the types of incompatibilities that we use python(abi) for (rather than a true c-library usage of the term).
You say that jruby doesn't provide an abi. Are you meaning for extensions or the types of changes mentioned above? Does jruby target certain revisions of the above that certain versions of cruby have implemented at a certain version or is it a mixture of some features from one release and some features from a different revision?
Depending on where we can get compatibility, it may make sense to have one virtual provide or one for the interpreter and one for the stdlib or one for the interpreter and several for the stdlib. We might also end up deciding to implement something similar to Debian/Ubuntu's python-commons setup for ruby or deciding that building a full stack from one version of the packageset is too hard to validate so having separate packages is better in the long run.
-Toshio
----- Original Message -----
On Dec 20, 2012 11:12 PM, "Bohuslav Kabrda" < bkabrda@redhat.com > wrote:
I think that one of the questions here is "why 'abi' and not something else?" We currently enforce every Ruby package to have "Requires: ruby(abi)", but JRuby doesn't in fact provide any ABI (I don't think the Java API can be called ABI). Therefore it doesn't make sense for the pure-Ruby packages to require ruby(abi). It'd be better to have something that would explain that all they need is _anything_ that can interpret Ruby, e.g. "ruby(interpreter)". Packages requiring ruby(abi) should IMHO be only these that have C extension or are otherwise bound to cRuby. Similarly, JRuby-only packages should depend on jruby (or its virtual provide, if we choose to create one).
This sounds like a productive piece of discussion. So is sounds like you desire to create as few packages as possible to create a full stack for c-ruby, jruby, and other implementations of the ruby language. What rpm dependencies are used for are to guarantee that when a user installs a package, the things that package needs to run are also installed.
( Note: If I use "when" below, i'm asking what conventions exist to map those types of changes to bumps in the version number of upstream releases.) It's when we put these two things together that things start to get tricky. When does the ruby language itself change? (For instance, if blocks were only specified with curly braces in one version but could be specified with either curly braces or parentheses in a later version.) (Another for instance, if a new keyword is added to the language).
The standard for Ruby language is the cRuby. E.g. Matz decides what appears and what does not appear in next versions. Tiny versions keep the ABI and API mostly the same (e.g. 1.8.3 can run anything that 1.8.2 could, but 1.8.3 may have some minor feauture enahancements, AFAIK), whereas minor and major versions may change both ABI and API (for example 1.8.x to 1.9.x or 1.9.x to 2.x.x).
<blockquote>
Is the ruby stdlib shipped separately from the interpreter to users? Are types part of the stdlib or the interpreter? When can those be added or changed? These are examples of the types of incompatibilities that we use python(abi) for (rather than a true c-library usage of the term). </blockquote> Ruby stdlib is shipped with the interpreter, types are part of the interpreter - they can be changed as explained above.
<blockquote>
You say that jruby doesn't provide an abi. Are you meaning for extensions or the types of changes mentioned above? Does jruby target certain revisions of the above that certain versions of cruby have implemented at a certain version or is it a mixture of some features from one release and some features from a different revision? </blockquote> I mean "abi" as in "libruby.so.x.x", so for extensions. JRuby always supports certain version(s) - compat modes - of Ruby. Right now, it supports 1.9 and 1.8 (and new features from next version of cRuby continuously land in the latest compat mode, so e.g. the 1.9 compat mode of JRuby may be in fact a bit different from cRuby 1.9). <blockquote>
Depending on where we can get compatibility, it may make sense to have one virtual provide or one for the interpreter and one for the stdlib or one for the interpreter and several for the stdlib. We might also end up deciding to implement something similar to Debian/Ubuntu's python-commons setup for ruby or deciding that building a full stack from one version of the packageset is too hard to validate so having separate packages is better in the long run. </blockquote> As mentioned, I would prefer having something like "ruby(interpreter)" provided by both cRuby and JRuby. This would be required by all Ruby packages. Packages with implementation specific extension would then require "ruby(abi)" or "jruby(?)" (maybe just jruby).
On Dec 21, 2012 2:25 AM, "Bohuslav Kabrda" bkabrda@redhat.com wrote
This sounds like a productive piece of discussion. So is sounds like
you desire to create as few packages as possible to create a full stack for c-ruby, jruby, and other implementations of the ruby language.
What rpm dependencies are used for are to guarantee that when a user
installs a package, the things that package needs to run are also installed.
( Note: If I use "when" below, i'm asking what conventions exist to map
those types of changes to bumps in the version number of upstream releases.)
It's when we put these two things together that things start to get
tricky. When does the ruby language itself change? (For instance, if blocks were only specified with curly braces in one version but could be specified with either curly braces or parentheses in a later version.) (Another for instance, if a new keyword is added to the language).
The standard for Ruby language is the cRuby. E.g. Matz decides what
appears and what does not appear in next versions. Tiny versions keep the ABI and API mostly the same (e.g. 1.8.3 can run anything that 1.8.2 could, but 1.8.3 may have some minor feauture enahancements, AFAIK), whereas minor and major versions may change both ABI and API (for example 1.8.x to 1.9.x or 1.9.x to 2.x.x).
After reading your whole post the first thing I need to make sure of is that when you talk about abi and api here you are talking about the c library correct? You are not talking about the other things that the python(abi) virtual provide covers or the things that the ruby(abi) as currently written is intended to cover?
Second, when you talk about minor feature enhancements, that includes the things that I mentioned? (Changes to the ruby language, api additions, deletions, and changes to the ruby stdlib, changes to the filesystem layout that have visibility to other ruby packages or ruby users). Is it policy that those types of changes are allowed or do they just slip in once in a while because no one realized that a particular change would have that sort of effect?
Is the ruby stdlib shipped separately from the interpreter to users?
Are types part of the stdlib or the interpreter? When can those be added or changed? These are examples of the types of incompatibilities that we use python(abi) for (rather than a true c-library usage of the term).
Ruby stdlib is shipped with the interpreter, types are part of the
interpreter - they can be changed as explained above.
Thanks further thoughts/questions as a result of this included above.
You say that jruby doesn't provide an abi. Are you meaning for
extensions or the types of changes mentioned above? Does jruby target certain revisions of the above that certain versions of cruby have implemented at a certain version or is it a mixture of some features from one release and some features from a different revision?
I mean "abi" as in "libruby.so.x.x", so for extensions.
Okay. That's not how fedora traditionally used these virtual provides as that would be redundant with the requires on the shared library. The virtual provides were used on the things that I mentioned instead (it sounds like those things might be called "compat modes" by ruby people. Is that correct?)
JRuby always supports certain version(s) - compat modes - of Ruby. Right
now, it supports 1.9 and 1.8 (and new features from next version of cRuby continuously land in the latest compat mode, so e.g. the 1.9 compat mode of JRuby may be in fact a bit different from cRuby 1.9).
This sounds like a version of a compat mode isn't finalized until the last cruby release of that version is released? Does compat mode cover both the stdlib and the language or just the language? How long does jruby lag behind cruby in terms of implementing these features? Is compat mode canonized in the cruby and jruby code or is it just a convenient concept?
Depending on where we can get compatibility, it may make sense to have
one virtual provide or one for the interpreter and one for the stdlib or one for the interpreter and several for the stdlib. We might also end up deciding to implement something similar to Debian/Ubuntu's python-commons setup for ruby or deciding that building a full stack from one version of the packageset is too hard to validate so having separate packages is better in the long run.
As mentioned, I would prefer having something like "ruby(interpreter)"
provided by both cRuby and JRuby. This would be required by all Ruby packages.
Does that also cover changes to the ruby stdlib, filesystem layout, etc? If so, it should simply be a rename of ruby(abi) for cruby and packages that depend on it. Adding jruby to the mix, i'm not yet certain of how much compatibility there is here. I'm also not sure how much incompatibility we're willing to push onto users.
Packages with implementation specific extension would then require "ruby(abi)" or "jruby(?)" (maybe just jruby).
If I understand what you're saying here, there's no point to ruby(abi) in those circumstances. Extensions would depend on certain versions of the shared library (which is what you seem to have thought ruby(abi) represented) and they probably also need to depend on ruby(interpreter) or whatever virtual provide we use for the combination of language features, stdlib apis, filesystem layout, and other things that determine if a package will install and run with an installed interpreter.
-Toshio
-- Regards, Bohuslav "Slavek" Kabrda.
-- packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
----- Original Message -----
On Dec 21, 2012 2:25 AM, "Bohuslav Kabrda" < bkabrda@redhat.com > wrote
This sounds like a productive piece of discussion. So is sounds like you desire to create as few packages as possible to create a full stack for c-ruby, jruby, and other implementations of the ruby language.
What rpm dependencies are used for are to guarantee that when a user installs a package, the things that package needs to run are also installed.
( Note: If I use "when" below, i'm asking what conventions exist to map those types of changes to bumps in the version number of upstream releases.)
It's when we put these two things together that things start to get tricky. When does the ruby language itself change? (For instance, if blocks were only specified with curly braces in one version but could be specified with either curly braces or parentheses in a later version.) (Another for instance, if a new keyword is added to the language).
The standard for Ruby language is the cRuby. E.g. Matz decides what appears and what does not appear in next versions. Tiny versions keep the ABI and API mostly the same (e.g. 1.8.3 can run anything that 1.8.2 could, but 1.8.3 may have some minor feauture enahancements, AFAIK), whereas minor and major versions may change both ABI and API (for example 1.8.x to 1.9.x or 1.9.x to 2.x.x).
After reading your whole post the first thing I need to make sure of is that when you talk about abi and api here you are talking about the c library correct? You are not talking about the other things that the python(abi) virtual provide covers or the things that the ruby(abi) as currently written is intended to cover? Second, when you talk about minor feature enhancements, that includes the things that I mentioned? (Changes to the ruby language, api additions, deletions, and changes to the ruby stdlib, changes to the filesystem layout that have visibility to other ruby packages or ruby users). Is it policy that those types of changes are allowed or do they just slip in once in a while because no one realized that a particular change would have that sort of effect?
Well, when I say "abi", I mean the C library, but when I say "api", I mean the pure Ruby language, which can be interpreted by any Ruby interpreter. I think we need to distinguish these two - ruby(api) for pure Ruby libraries and ruby(abi) or jruby(???) for those that contain an exten sion. As for the second point, AFAIK they "slip once in a while ".
Is the ruby stdlib shipped separately from the interpreter to users? Are types part of the stdlib or the interpreter? When can those be added or changed? These are examples of the types of incompatibilities that we use python(abi) for (rather than a true c-library usage of the term).
Ruby stdlib is shipped with the interpreter, types are part of the interpreter - they can be changed as explained above.
Thanks further thoughts/questions as a result of this included above.
You say that jruby doesn't provide an abi. Are you meaning for extensions or the types of changes mentioned above? Does jruby target certain revisions of the above that certain versions of cruby have implemented at a certain version or is it a mixture of some features from one release and some features from a different revision?
I mean "abi" as in "libruby.so.x.x", so for extensions.
Okay. That's not how fedora traditionally used these virtual provides as that would be redundant with the requires on the shared library. The virtual provides were used on the things that I mentioned instead (it sounds like those things might be called "compat modes" by ruby people. Is that correct?)
Yep, I think so.
JRuby always supports certain version(s) - compat modes - of Ruby. Right now, it supports 1.9 and 1.8 (and new features from next version of cRuby continuously land in the latest compat mode, so e.g. the 1.9 compat mode of JRuby may be in fact a bit different from cRuby 1.9).
This sounds like a version of a compat mode isn't finalized until the last cruby release of that version is released?
What do you mean when saying "finalized"? If you mean that e.g. features from 1.8.x to 1.8.y land continuously in the JRuby's 1.8 compat mode, then the answer to your question is yes.
Does compat mode cover both the stdlib and the language or just the language?
Both.
How long does jruby lag behind cruby in terms of implementing these features?
Now that's a tricky question :) It depends on the feature complexity, I guess, but AFAIK the JRuby upstream is trying to keep as close as possible and doesn't delay anything on purpose.
Is compat mode canonized in the cruby and jruby code or is it just a convenient concept?
More of a convenient concept , to my knowledge.
Depending on where we can get compatibility, it may make sense to have one virtual provide or one for the interpreter and one for the stdlib or one for the interpreter and several for the stdlib. We might also end up deciding to implement something similar to Debian/Ubuntu's python-commons setup for ruby or deciding that building a full stack from one version of the packageset is too hard to validate so having separate packages is better in the long run.
As mentioned, I would prefer having something like "ruby(interpreter)" provided by both cRuby and JRuby. This would be required by all Ruby packages.
Does that also cover changes to the ruby stdlib, filesystem layout, etc? If so, it should simply be a rename of ruby(abi) for cruby and packages that depend on it. Adding jruby to the mix, i'm not yet certain of how much compatibility there is here. I'm also not sure how much incompatibility we're willing to push onto users. Packages with implementation specific extension would then require "ruby(abi)" or "jruby(?)" (maybe just jruby).
If I understand what you're saying here, there's no point to ruby(abi) in those circumstances. Extensions would depend on certain versions of the shared library (which is what you seem to have thought ruby(abi) represented) and they probably also need to depend on ruby(interpreter) or whatever virtual provide we use for the combination of language features, stdlib apis, filesystem layout, and other things that determine if a package will install and run with an installed interpreter.
Yes, exactly. If we say that ruby(abi) is for depending on a certain version of shared library, we don't need it, because RPM will take care of that. The only reason that would make sense for ruby(abi) to exist is, that JRuby extension packages will have Requires: jruby, so it may be nice to have a cRuby counterpart of that, just for convenience. And yes, ruby(interpreter) or ruby(compat) (have a better name?) is what all the ruby packages (both pure and with extensions) should contain.
-Toshio
On Mon, Jan 07, 2013 at 03:33:58AM -0500, Bohuslav Kabrda wrote:
Second, when you talk about minor feature enhancements, that includes the things that I mentioned? (Changes to the ruby language, api additions, deletions, and changes to the ruby stdlib, changes to the filesystem layout that have visibility to other ruby packages or ruby users). Is it policy that those types of changes are allowed or do they just slip in once in a while because no one realized that a particular change would have that sort of effect?Well, when I say "abi", I mean the C library, but when I say "api", I mean the pure Ruby language, which can be interpreted by any Ruby interpreter. I think we need to distinguish these two - ruby(api) for pure Ruby libraries and ruby (abi) or jruby(???) for those that contain an extension. As for the second point, AFAIK they "slip once in a while".
[snip]
Okay. That's not how fedora traditionally used these virtual provides as that would be redundant with the requires on the shared library. The virtual provides were used on the things that I mentioned instead (it sounds like those things might be called "compat modes" by ruby people. Is that correct?)Yep, I think so.
So problem 1) The Ruby SIG finds ruby(abi) hard to understand. We can simply rename it to ruby(compat-mode) to resolve this. There's no need to make a new ruby(abi) virtual provide for the C extensions because the automatic library dep finding should take care of that. Requires a guideline update (discussing here) and a mass rebuild of the packages that provide and require ruby(abi).
The mass rebuild should be doable at this point in the cycle. If we run into issues deciding on the rest of the proposal we should decide on whether to make this rename of virtual provides separately (and first) so that it can go into a mass rebuild. (mass rebuilds are usually done several weeks (up to a month) before alpha).
Now on to the harder parts of the proposal...
> JRuby always supports certain version(s) - compat modes - of Ruby. Right now, it supports 1.9 and 1.8 (and new features from next version of cRuby continuously land in the latest compat mode, so e.g. the 1.9 compat mode of JRuby may be in fact a bit different from cRuby 1.9). This sounds like a version of a compat mode isn't finalized until the last cruby release of that version is released?What do you mean when saying "finalized"? If you mean that e.g. features from 1.8.x to 1.8.y land continuously in the JRuby's 1.8 compat mode, then the answer to your question is yes.
With "finalized" I was talking about cRuby but I may have misunderstood what you wrote earlier (with "new features from the next version of cRuby"). I was trying to find out if cRuby continued to add things to the MAJOR.MINOR compat mode after the release of the first MAJOR.MINOR:
For cRuby MAJOR.MINOR (example: 1.8), is it cRuby policy that 1.8.1, 1.8.2, 1.8.3, etc do not have changes that affect compat mode? No new features or changes to existing ones that mean that code written on 1.8.0 will/will not run on 1.8.3? What about the opposite: is there policy about adding and changing features so that code written on 1.8.3 will run on 1.8.0? Is this policy frequently broken by accidents or rarely?
If I re-read what you wrote to apply to jRuby instead of cRuby, then I have another question: how do compat modes line up with jRuby releases? Does jRuby only ship releases when a compat mode is 100%? (It sounds like the answer is no since you mention that no release of jRuby support fork(), for instance). Is there some compat-mode-completion criteria for shipping a release of jRuby?
If we don't have 100% compatibility the present structure of virtual proivides is not well adapted to a shared compat-mode virtual provide. (Similar dangers exist in python where, for instance pypy is close to compatible with python2 but has a few corner cases: http://pypy.readthedocs.org/en/latest/cpython_differences.html ). I don't believe we have precedent as to "how compatible" something would need to be to share this type of virtual provide but in general, the right thing to do is to err on the side of caution in order to protect the end user.
Debian has a method that they use for their python stack that is pretty complex and a little hacky that might be usable. It requires packagers to manually mark their package as being compatible with certain versions of python and then the needed module files are symlinked into the library paths for the compatible python versions when the package is installed. Something along these lines is a lot more work to setup but may be another possibility.
-Toshio
Dne 7.1.2013 19:58, Toshio Kuratomi napsal(a):
On Mon, Jan 07, 2013 at 03:33:58AM -0500, Bohuslav Kabrda wrote:
Second, when you talk about minor feature enhancements, that includes the things that I mentioned? (Changes to the ruby language, api additions, deletions, and changes to the ruby stdlib, changes to the filesystem layout that have visibility to other ruby packages or ruby users). Is it policy that those types of changes are allowed or do they just slip in once in a while because no one realized that a particular change would have that sort of effect?Well, when I say "abi", I mean the C library, but when I say "api", I mean the pure Ruby language, which can be interpreted by any Ruby interpreter. I think we need to distinguish these two - ruby(api) for pure Ruby libraries and ruby (abi) or jruby(???) for those that contain an extension. As for the second point, AFAIK they "slip once in a while".
[snip]
Okay. That's not how fedora traditionally used these virtual provides as that would be redundant with the requires on the shared library. The virtual provides were used on the things that I mentioned instead (it sounds like those things might be called "compat modes" by ruby people. Is that correct?)Yep, I think so.
So problem 1) The Ruby SIG finds ruby(abi) hard to understand. We can simply rename it to ruby(compat-mode) to resolve this.
We are thinking about following virtual provide names:
ruby(release) ruby(version) ruby(compat) ruby(compat-mode)
Currently, we prefer to call this virtual provide ruby(release), which fits best IMO. I wouldn't go with any "compat" in virtual provide name, since it might be confused with -compat packages.
There's no need to make a new ruby(abi) virtual provide for the C extensions because the automatic library dep finding should take care of that.
Exactly.
Requires a guideline update (discussing here) and a mass rebuild of the packages that provide and require ruby(abi).
Yes, we will need mass rebuild anyway, even if we just change ruby(abi) version. Moreover, we are going to do minor adjustment of RubyGems filesystem layout for further compatibility with alternative Ruby implementations (mainly targeted at Rubinius ATM)
The mass rebuild should be doable at this point in the cycle. If we run into issues deciding on the rest of the proposal we should decide on whether to make this rename of virtual provides separately (and first) so that it can go into a mass rebuild. (mass rebuilds are usually done several weeks (up to a month) before alpha).
We are planing to ask rel-eng for separate buildroot tag. This would ease the revert in case things go awry
Now on to the harder parts of the proposal...
> JRuby always supports certain version(s) - compat modes - of Ruby. Right now, it supports 1.9 and 1.8 (and new features from next version of cRuby continuously land in the latest compat mode, so e.g. the 1.9 compat mode of JRuby may be in fact a bit different from cRuby 1.9). This sounds like a version of a compat mode isn't finalized until the last cruby release of that version is released?What do you mean when saying "finalized"? If you mean that e.g. features from 1.8.x to 1.8.y land continuously in the JRuby's 1.8 compat mode, then the answer to your question is yes.
With "finalized" I was talking about cRuby but I may have misunderstood what you wrote earlier (with "new features from the next version of cRuby"). I was trying to find out if cRuby continued to add things to the MAJOR.MINOR compat mode after the release of the first MAJOR.MINOR:
For cRuby MAJOR.MINOR (example: 1.8), is it cRuby policy that 1.8.1, 1.8.2, 1.8.3, etc do not have changes that affect compat mode? No new features or changes to existing ones that mean that code written on 1.8.0 will/will not run on 1.8.3? What about the opposite: is there policy about adding and changing features so that code written on 1.8.3 will run on 1.8.0? Is this policy frequently broken by accidents or rarely?
Let me explain a bit.
1.8.7, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 2.0.0, all these are major versions. These major versions are typically released every 18 months. They may or may not break things. However, you cannot know that from version. What is clear is that patch releases should not break anything (although for example fixing CVE-2011-4815 [1] broke some compatibility, due to change in natural ordering of hashes, but that was always wrong assumption). They are typically safe to update.
For example, as you probably know, the ABI was kept compatible in between 1.9.1..1.9.3 versions where there will be ABI breakage for Ruby 2.0.0. However, in between 1.9.1 and 1.9.2, there was polished Unicode support, which of course might break something. It depends how lucky you are. For example, between 1.9.3 and 2.0.0, there is going to be change in default encoding of Ruby scripts from US-ASCII to UTF-8. I am not sure if you can foresee every possible issue with this change, I cannot.
To conclude, there is not yet know what will be next version and what it will break. This will be known later, when its development begins. And it will be clearly declared and communicated [2].
I can assure you that there is no unnoticed feature creep.
If I re-read what you wrote to apply to jRuby instead of cRuby, then I have another question: how do compat modes line up with jRuby releases? Does jRuby only ship releases when a compat mode is 100%? (It sounds like the answer is no since you mention that no release of jRuby support fork(), for instance). Is there some compat-mode-completion criteria for shipping a release of jRuby?
There does not exist nothing like 100% compatibility, and that is not just about fork, but about C Ruby extensions and so on. The JRuby compatibility is based on best effort. I.e. it should work in most of the cases and if it does not work in your case, JRuby team will try hard to solve the issues.
The compatibility of Ruby implementations is assured/measured by RubySpec project [3]. The JRuby is trying hard to use original unmodified cRuby standard library, this can give you overview how compatible these implementations are.
There is also expected that for example you will start to develop you Ruby on Rails application using cRuby but you will run it by JRuby in production.
If we don't have 100% compatibility the present structure of virtual proivides is not well adapted to a shared compat-mode virtual provide. (Similar dangers exist in python where, for instance pypy is close to compatible with python2 but has a few corner cases: http://pypy.readthedocs.org/en/latest/cpython_differences.html ). I don't believe we have precedent as to "how compatible" something would need to be to share this type of virtual provide but in general, the right thing to do is to err on the side of caution in order to protect the end user.
Obviously, there is no universal answer to compatibility nor what interpreter is the best. Therefore we want to allow to use every Ruby implementation in the world as easy as possible, providing sensible defaults to those who don't care.
Debian has a method that they use for their python stack that is pretty complex and a little hacky that might be usable. It requires packagers to manually mark their package as being compatible with certain versions of python and then the needed module files are symlinked into the library paths for the compatible python versions when the package is installed. Something along these lines is a lot more work to setup but may be another possibility.
The problem with this approach is that even if the code is compatible with certain version/implementation and maintainer does not provide the symlinks, it will not work. We would like to go the opposite way, be proactive. I.e. we expect that everything works and if not, either fix it to really work or disable it to not be broken.
Vít
-Toshio
[1] https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2011-4815 [2] http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/50443 [3] http://rubyspec.org/
On Tue, Jan 08, 2013 at 10:17:31AM +0100, Vít Ondruch wrote:
Dne 7.1.2013 19:58, Toshio Kuratomi napsal(a):
I was trying to find out if cRuby continued to add things to the MAJOR.MINOR compat mode after the release of the first MAJOR.MINOR:
For cRuby MAJOR.MINOR (example: 1.8), is it cRuby policy that 1.8.1, 1.8.2, 1.8.3, etc do not have changes that affect compat mode? No new features or changes to existing ones that mean that code written on 1.8.0 will/will not run on 1.8.3? What about the opposite: is there policy about adding and changing features so that code written on 1.8.3 will run on 1.8.0? Is this policy frequently broken by accidents or rarely?
Let me explain a bit.
1.8.7, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 2.0.0, all these are major versions. These major versions are typically released every 18 months. They may or may not break things. However, you cannot know that from version. What is clear is that patch releases should not break anything (although for example fixing CVE-2011-4815 [1] broke some compatibility, due to change in natural ordering of hashes, but that was always wrong assumption). They are typically safe to update.
Okay. So this is why the rub(abi) virtual provide has a version of MAJOR.MINOR.MICRO. Thre's no knowing by the version number whether a MICRO version changed things relevant to the compat-mode or not so the virtual provide has to go to that level of detail.
For example, as you probably know, the ABI was kept compatible in between 1.9.1..1.9.3 versions where there will be ABI breakage for Ruby 2.0.0.
To remain clear here, when you say ABI, are you talking about the elf library abi or the compat-mode criteria (ruby's stdlib, keywords, and other features discussed earlier)? I'm reading it as elf library abi but I want to be sure since we aren't talking about elf library abi in the rest of this discussion so I'm not sure if I'm misinterpreting or there's a reason for bringing it up that I'm not seeing.
However, in between 1.9.1 and 1.9.2, there was polished Unicode support, which of course might break something. It depends how lucky you are. For example, between 1.9.3 and 2.0.0, there is going to be change in default encoding of Ruby scripts from US-ASCII to UTF-8. I am not sure if you can foresee every possible issue with this change, I cannot.
To conclude, there is not yet know what will be next version and what it will break. This will be known later, when its development begins. And it will be clearly declared and communicated [2].
I can assure you that there is no unnoticed feature creep.
I'm only incidentally looking for unnoticed compat-mode relevant changes Although, some of my questions were to know if you thought that upstream was on top of that, I'd certainly hope that you were on top of it even if they weren't. The real question, though is where the compat-mode relevant changes are entering in and how that affects the version. From your message, it sounds like compat-mode relevant changes can enter in at any time (in relation to the version numbers that we're given).
If I re-read what you wrote to apply to jRuby instead of cRuby, then I have another question: how do compat modes line up with jRuby releases? Does jRuby only ship releases when a compat mode is 100%? (It sounds like the answer is no since you mention that no release of jRuby support fork(), for instance). Is there some compat-mode-completion criteria for shipping a release of jRuby?
There does not exist nothing like 100% compatibility, and that is not just about fork, but about C Ruby extensions and so on. The JRuby compatibility is based on best effort. I.e. it should work in most of the cases and if it does not work in your case, JRuby team will try hard to solve the issues.
The compatibility of Ruby implementations is assured/measured by RubySpec project [3]. The JRuby is trying hard to use original unmodified cRuby standard library, this can give you overview how compatible these implementations are.
There is also expected that for example you will start to develop you Ruby on Rails application using cRuby but you will run it by JRuby in production.
If we don't have 100% compatibility the present structure of virtual proivides is not well adapted to a shared compat-mode virtual provide. (Similar dangers exist in python where, for instance pypy is close to compatible with python2 but has a few corner cases: http://pypy.readthedocs.org/en/latest/cpython_differences.html ). I don't believe we have precedent as to "how compatible" something would need to be to share this type of virtual provide but in general, the right thing to do is to err on the side of caution in order to protect the end user.
Obviously, there is no universal answer to compatibility nor what interpreter is the best. Therefore we want to allow to use every Ruby implementation in the world as easy as possible, providing sensible defaults to those who don't care.
Debian has a method that they use for their python stack that is pretty complex and a little hacky that might be usable. It requires packagers to manually mark their package as being compatible with certain versions of python and then the needed module files are symlinked into the library paths for the compatible python versions when the package is installed. Something along these lines is a lot more work to setup but may be another possibility.
The problem with this approach is that even if the code is compatible with certain version/implementation and maintainer does not provide the symlinks, it will not work. We would like to go the opposite way, be proactive. I.e. we expect that everything works and if not, either fix it to really work or disable it to not be broken.
From what I'm reading so far, I don't think that your approach is going to be able to pass as a guideline update. You essentially are asking to remove the version from the virtual provide and instead let all packages containing ruby code to merely specify that they need "Ruby"... not ruby-1.8.1 vs ruby-2.0.0, etc. This is too loose a coupling for proper package management as it allows users to install incompatible combinations of packages because the dependencies aren't specified precisely enough.
OTOH, although all the guidelines I looked at do tell you how to require the langugage itself, not all of them make it mandatory:
Language Virtual Provide Required/Optional java java >= specific_version Required lisp None but see note (1) mono None but mono is compiled and automatic deps are extracted that require mono ocaml ocaml(runtime) = MAJOR.MINOR.MICRO Required for libraries perl perl(:MODULE_COMPAT_MAJOR.MINOR.MICRO) Required python python(abi) = MAJOR.MINOR Required (auto-filled) php php(language) >= VERSION Optional (only if it needs a specific version) R R-Core Unknown (2) ruby ruby(abi) = MAJOR.MINOR.MICRO Required tcl Requires: tcl(abi) = MAJOR.MINOR Required
(1) lisp doesn't have a Virtual Provide at all but the lisp language isn't making rapid changes so the assumption seems to be that all lisp source is compatible. The compiled files, which are incompatible between lisp interpreters and perhaps versions of those interpreters is generated after the package is installed which is similar in that respect to what Debian is doing for python.
(2) R-Core is in the spec templates. It is unversioned there. Nothing says whether a versioned virtual provide should be used or not and some R packages use a versioned Requirement on R-Core while others use an unversioned one.
Looking at this list, it seems that the major languages do have required versioned Requires. It may be that the ones where it's optional (php, lisp) or don't specify a version (R) should be updated to make that a mandatory requirement.
-Toshio
Dne 9.1.2013 01:05, Toshio Kuratomi napsal(a):
On Tue, Jan 08, 2013 at 10:17:31AM +0100, Vít Ondruch wrote:
Dne 7.1.2013 19:58, Toshio Kuratomi napsal(a):
I was trying to find out if cRuby continued to add things to the MAJOR.MINOR compat mode after the release of the first MAJOR.MINOR:
For cRuby MAJOR.MINOR (example: 1.8), is it cRuby policy that 1.8.1, 1.8.2, 1.8.3, etc do not have changes that affect compat mode? No new features or changes to existing ones that mean that code written on 1.8.0 will/will not run on 1.8.3? What about the opposite: is there policy about adding and changing features so that code written on 1.8.3 will run on 1.8.0? Is this policy frequently broken by accidents or rarely?
Let me explain a bit.
1.8.7, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 2.0.0, all these are major versions. These major versions are typically released every 18 months. They may or may not break things. However, you cannot know that from version. What is clear is that patch releases should not break anything (although for example fixing CVE-2011-4815 [1] broke some compatibility, due to change in natural ordering of hashes, but that was always wrong assumption). They are typically safe to update.
Okay. So this is why the rub(abi) virtual provide has a version of MAJOR.MINOR.MICRO. Thre's no knowing by the version number whether a MICRO version changed things relevant to the compat-mode or not so the virtual provide has to go to that level of detail.
For example, as you probably know, the ABI was kept compatible in between 1.9.1..1.9.3 versions where there will be ABI breakage for Ruby 2.0.0.
To remain clear here, when you say ABI, are you talking about the elf library abi or the compat-mode criteria (ruby's stdlib, keywords, and other features discussed earlier)? I'm reading it as elf library abi but I want to be sure since we aren't talking about elf library abi in the rest of this discussion so I'm not sure if I'm misinterpreting or there's a reason for bringing it up that I'm not seeing.
I am always talking about elf library ABI.
However, in between 1.9.1 and 1.9.2, there was polished Unicode support, which of course might break something. It depends how lucky you are. For example, between 1.9.3 and 2.0.0, there is going to be change in default encoding of Ruby scripts from US-ASCII to UTF-8. I am not sure if you can foresee every possible issue with this change, I cannot.
To conclude, there is not yet know what will be next version and what it will break. This will be known later, when its development begins. And it will be clearly declared and communicated [2].
I can assure you that there is no unnoticed feature creep.
I'm only incidentally looking for unnoticed compat-mode relevant changes Although, some of my questions were to know if you thought that upstream was on top of that, I'd certainly hope that you were on top of it even if they weren't. The real question, though is where the compat-mode relevant changes are entering in and how that affects the version. From your message, it sounds like compat-mode relevant changes can enter in at any time (in relation to the version numbers that we're given).
once in 18 months is not exactly anytime ;)
If I re-read what you wrote to apply to jRuby instead of cRuby, then I have another question: how do compat modes line up with jRuby releases? Does jRuby only ship releases when a compat mode is 100%? (It sounds like the answer is no since you mention that no release of jRuby support fork(), for instance). Is there some compat-mode-completion criteria for shipping a release of jRuby?
There does not exist nothing like 100% compatibility, and that is not just about fork, but about C Ruby extensions and so on. The JRuby compatibility is based on best effort. I.e. it should work in most of the cases and if it does not work in your case, JRuby team will try hard to solve the issues.
The compatibility of Ruby implementations is assured/measured by RubySpec project [3]. The JRuby is trying hard to use original unmodified cRuby standard library, this can give you overview how compatible these implementations are.
There is also expected that for example you will start to develop you Ruby on Rails application using cRuby but you will run it by JRuby in production.
If we don't have 100% compatibility the present structure of virtual proivides is not well adapted to a shared compat-mode virtual provide. (Similar dangers exist in python where, for instance pypy is close to compatible with python2 but has a few corner cases: http://pypy.readthedocs.org/en/latest/cpython_differences.html ). I don't believe we have precedent as to "how compatible" something would need to be to share this type of virtual provide but in general, the right thing to do is to err on the side of caution in order to protect the end user.
Obviously, there is no universal answer to compatibility nor what interpreter is the best. Therefore we want to allow to use every Ruby implementation in the world as easy as possible, providing sensible defaults to those who don't care.
Debian has a method that they use for their python stack that is pretty complex and a little hacky that might be usable. It requires packagers to manually mark their package as being compatible with certain versions of python and then the needed module files are symlinked into the library paths for the compatible python versions when the package is installed. Something along these lines is a lot more work to setup but may be another possibility.
The problem with this approach is that even if the code is compatible with certain version/implementation and maintainer does not provide the symlinks, it will not work. We would like to go the opposite way, be proactive. I.e. we expect that everything works and if not, either fix it to really work or disable it to not be broken.
From what I'm reading so far, I don't think that your approach is going to be able to pass as a guideline update. You essentially are asking to remove the version from the virtual provide and instead let all packages containing ruby code to merely specify that they need "Ruby"... not ruby-1.8.1 vs ruby-2.0.0, etc. This is too loose a coupling for proper package management as it allows users to install incompatible combinations of packages because the dependencies aren't specified precisely enough.
Not exactly. The virtual provide should be versioned. So the ruby-libs should provide:
Provides: ruby(release) = 2.0.0
But the require side would do nonversioned Requires: ruby(release). If there would be some reason, e.g. library which is working just with some specific version of Ruby, it could use Requires: ruby(release) >= 2.0.
JRuby 1.7 could then provide:
Provides: ruby(release) = 1.9 Provides: ruby(release) = 1.8
To claim compatibility with older releases.
BTW, it is interesting that rpmlint complains about multiple virtual provides with different versions:
ruby-libs.x86_64: E: useless-provides ruby(release) This package provides 2 times the same capacity. It should only provide it once.
while it works as expected in reality.
OTOH, although all the guidelines I looked at do tell you how to require the langugage itself, not all of them make it mandatory:
Language Virtual Provide Required/Optional java java >= specific_version Required lisp None but see note (1) mono None but mono is compiled and automatic deps are extracted that require mono ocaml ocaml(runtime) = MAJOR.MINOR.MICRO Required for libraries perl perl(:MODULE_COMPAT_MAJOR.MINOR.MICRO) Required python python(abi) = MAJOR.MINOR Required (auto-filled) php php(language) >= VERSION Optional (only if it needs a specific version) R R-Core Unknown (2) ruby ruby(abi) = MAJOR.MINOR.MICRO Required tcl Requires: tcl(abi) = MAJOR.MINOR Required
(1) lisp doesn't have a Virtual Provide at all but the lisp language isn't making rapid changes so the assumption seems to be that all lisp source is compatible. The compiled files, which are incompatible between lisp interpreters and perhaps versions of those interpreters is generated after the package is installed which is similar in that respect to what Debian is doing for python.
(2) R-Core is in the spec templates. It is unversioned there. Nothing says whether a versioned virtual provide should be used or not and some R packages use a versioned Requirement on R-Core while others use an unversioned one.
Looking at this list, it seems that the major languages do have required versioned Requires. It may be that the ones where it's optional (php, lisp) or don't specify a version (R) should be updated to make that a mandatory requirement.
From that list I would say that Perl is the closest one, although I do not understand why they don't version the virtual provide, as I am proposing above. But this was not working in previous versions of RPM I guess.
Vít
Dne 20.12.2012 21:42, Toshio Kuratomi napsal(a):
On Thu, Dec 20, 2012 at 04:40:17PM +0100, Vít Ondruch wrote:
Dne 20.12.2012 15:46, Rex Dieter napsal(a):
On 12/20/2012 02:40 AM, Vít Ondruch wrote:
Dne 20.12.2012 01:43, Garrett Holmstrom napsal(a):
On 2012-12-19 5:12, Bill Nottingham wrote:
Vít Ondruch (vondruch@redhat.com) said: > Can somebody enlighten me, what is the purpose of ruby(abi) (replace > by python(abi) if you wish) virtual provide? Especially, why Ruby > packaging guidelines mandate "Requires: ruby(abi) = 1.9.1", i.e. > versioned require? And why in Python packages, python(abi) is > automatically generated? In the python case, it's because that python extension modules install in a version-specific directory ($libdir/python2.7, for example.) This makes them explicitly tied to that version of python.
There's also the fact that the ABI for the bytecode that gets generated at build time is specific to each x.y series of python releases.
For that, you could have "Require: python-libs = 2.7" instead.
What's the practical difference?
You follow general practices? You don't have to think if {ruby,python}(abi) makes any sense for {JRuby,Jython} and what version it should provide in comparison to {ruby,python}. You don't force people to ask why Ruby 1.9.3 has ruby(abi) = 1.9.1. You don't have to answer such question as what is {ruby,python}(abi) good for?
I don't think I follow your reasoning here:
- If there is no python(abi) and no automatic dependency checking, then the
packager assumes the burden of adding Requires: python-libs = VERSION in all of their packages. They also assume the responsibility of figuring out what VERSION needs to be. This is creating more work that should be done centrally.
- If there is no python(abi) but the automatic dep checking creates the
Provides and Requires on python-libs = VERSION, then there doesn't seem to be any difference in terms of the work done. Someone who maintains the central dep checking still has to figure out what VERSION should be in either case. (And, assuming that your question about Ruby 1.9.3 having ruby(abi) = 1.9.1
Please do not judge so quickly. Ruby keeps ABI compatibility between Ruby 1.9.1 till Ruby 1.9.3 (check the .so name as well), hence the ruby(abi) = 1.9.1. If the virtual provide would not be called ruby(abi), it could have probably different version.
was more than hypothetical, it sounds like there might just be a lack of someone performing this duty for Ruby).
-Toshio
There are two types of packages for Python as well as for Ruby:
1) Packages with binary extension - these packages already have automatically generated dependency on python-libs/ruby-libs, as if it would be with C libraries for example. You don't have to explicitly specify anything. No python(abi) is required to mandate the correct Python version.
2) Plain Python/Ruby packages - here you have to somehow specify that the package depends on python/ruby. And here is the biggest difference between Python and Ruby packages. For the most of the Ruby packages, it does not matter with what interpreter version or implementation they run (yes, there are exceptions I neglect at this point). But the biggest difference between Ruby and Python is the Python's bytecode, which version specific, therefore you have to exactly specify against which version of python the plain Python libraries were build. You choose for this purpose the python(abi) virtual provide, which is probably fine, but I would call it python(bytecode) for example.
So my conclusion is:
For Python: * python(abi) is unfortunate name and should be dropped. * python(abi) should be split into virtual provides which would better describe their purpose
I would suggest python(bytecode) - describes the version of Python bytecode and is probably the same as version of python. python(filesystem) - this could describe the layout of Python's filesystem for extensions etc. I personally don't see need for this virtual and checking by Fedora's version should be OK IMO. I'm just mentioning it here, since this point appeared earlier in this discussion. python(interpreter) - if there might be Python packages without bytecode (not sure if that is allowed or if it make sense), they should require generic python interpreter.
For Ruby: * ruby(abi) is completely useless and should be dropped. * there should be introduced replacement, such as ruby(interpreter), ruby(version) or something which would describe that package needs Ruby interpreter and it should conform to MRI Ruby of some version. Ideas how to call this virtual provide are welcome. * ruby(filesystem) - see above the python(filesystem). I prefer checks by Fedora version instead, so no need for this virtual provide.
Vít
BTW Here are a relevant changelog entries, which introduces the python(abi):
* Mon Dec 6 2004 Jeff Johnson jbj@jbj.org 2.4-2 - add Provide: python(abi) = 2.4
* Mon Nov 24 2003 Mihai Ibanescu misa@redhat.com 2.3.2-6 - added a Provides: python-abi
and ruby(abi):
* Wed May 17 2006 Akira TAGOH tagoh@redhat.com - 1.8.4-6 - added a Provides: ruby(abi) to ruby-libs.
So Ruby packaging probably imitates Python, but hard to say what else was behind introduction ruby(abi).
On 12/20/2012 01:43 AM, Garrett Holmstrom wrote:
There's also the fact that the ABI for the bytecode that gets generated at build time is specific to each x.y series of python releases.
Well it is even specific for each architecture and we do not mind to flag such packages as noarch. I mean that .pyc generated on i686 is different from x86_64 version. However I'm not sure how much those two are compatible.
On Dec 21, 2012 2:53 AM, "Miroslav Suchý" msuchy@redhat.com wrote:
On 12/20/2012 01:43 AM, Garrett Holmstrom wrote:
There's also the fact that the ABI for the bytecode that gets generated at build time is specific to each x.y series of python releases.
Well it is even specific for each architecture and we do not mind to flag
such packages as noarch. I mean that .pyc generated on i686 is different from x86_64 version.
However I'm not sure how much those two are compatible.
This is incorrect. Python bytecode is arch independent. The .pyc files from one compilation are different from a second compilation out its true. But this is because a timestamp is embedded in the bytecode cache files (.pyc) so that python can tell whether a .pyc or a .py should be used when it finds both on disk.
-Toshio
-- Miroslav Suchy Red Hat Systems Management Engineering
-- packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
packaging@lists.fedoraproject.org
-
Bill Nottingham -
Bohuslav Kabrda -
David Malcolm -
Garrett Holmstrom -
Miroslav Suchý -
Panu Matilainen -
Rex Dieter -
Toshio Kuratomi -
Vít Ondruch