Hi guys,
after FUDCon, and then some more thoughts in private email and between my ears, I think I might have the bare bones for a plan to attack the Ruby challenge;
First of all, let me emphasize that the complete plan is based on the fact we want different Ruby stacks to be available, namely those Ruby stacks that someone might target (in development). That said, I think the list of Ruby stacks one can target right now are:
- ruby-1.8.5 as shipped in EL-5 - ruby-1.8.6 as shipped in Fedora currently, and most likely also EL-6 - ruby-1.9.1 as targeted for Fedora 13
The goal of this thread is to figure out what we should do in terms of packaging, but having given it a little thought here's how I think I would prefer to attack the problem. Please bear with me when I start of rambling 'cause I don't know how to put this in an overview that everyone understands (but we'll get there);
- Ruby packages themselves are to be named ruby-<major>.<minor>.<teeny>, as to not have to rename packages to compat-ruby every time a newer M.M.T is released.
- A base 'ruby' (e.g., the package's %{name} == "ruby") package will pull in the default stack (targeted to be ruby-1.9.1 in Fedora 13).
- The ruby-<m>.<m>.<t> packages are going to provide:
1) ruby(ABI) = <major>.<minor> 2) ruby(ABI) = <major>.<minor>.<teeny> 3) ruby(API) = <major>.<minor> 4) ruby(API) = <major>.<minor>.<teeny>
The Ruby ABI is pretty stable in between teeny versions, but much less so for the Ruby API.
- The base directory for Ruby modules is going to be %{ruby_vendorlib} and %{ruby_vendorarch}, provided through /etc/rpm/macros.ruby, resulting in /usr/share/ruby/ and %{_libdir}/ruby/.
This is where packaged gems and library extensions to Ruby should end up initially. If gems turn out to be compatible with one specific Ruby version only, one can use %{ruby_vendorlib_186} (again provided through /etc/rpm/macros.ruby), or if a gem is compatible with 1.8.x, %{ruby_vendorlib_18} should be used.
gems and library extensions installed manually should end up in the appropriate /usr/local/ directories, which will also appear first in each of ruby's search paths.
- Search paths are going to look similar to:
/usr/local/lib{,64}/ruby/<major>.<minor>.<teeny>/ /usr/local/lib{,64}/ruby/<major>.<minor>/ /usr/local/lib{,64}/ruby/ /usr/local/share/ruby/<major>.<minor>.<teeny>/ /usr/local/share/ruby/<major>.<minor>/ /usr/local/share/ruby/ /usr/lib{,64}/ruby/<major>.<minor>.<teeny>/ /usr/lib{,64}/ruby/<major>.<minor>/ /usr/lib{,64}/ruby/ /usr/share/ruby/<major>.<minor>.<teeny>/ /usr/share/ruby/<major>.<minor>/ /usr/share/ruby/
as to make sure that rubygem-foo can be packaged for all ruby versions easily, while making the exception (ruby version specific patch?) is also perfectly feasible (by putting another copy in the more specific search path).
To illustrate my packaging points, I would appreciate if you look at the following git repository, where I keep a couple of .spec files as well as the macros.ruby file I intend to use:
git://fedorapeople.org/home/fedora/kanarip/public_git/ruby-specs.git
Along with that comes a git repository for Ruby itself, to be found at:
git://fedorapeople.org/home/fedora/kanarip/public_git/ruby.git
Preliminary packages can be found on:
For Fedora 12:
- http://www.kanarip.com/custom/f12-ruby/ and - http://mirror.nl.kanarip.com/custom/f12-ruby/
For Fedora Rawhide:
- http://www.kanarip.com/custom/rawhide-ruby/ and - http://mirror.nl.kanarip.com/custom/rawhide-ruby/
That said, these packages could ruin your current Ruby stack (different search paths and all that).
Please be so kind as to test these versions and provide me with the necessary feedback, so we can move things along.
Thanks in advance,
Kind regards,
Jeroen van Meeuwen -kanarip
On 12/22/2009 12:32 AM, Jeroen van Meeuwen wrote:
Hi guys,
after FUDCon, and then some more thoughts in private email and between my ears, I think I might have the bare bones for a plan to attack the Ruby challenge;
Continuing on this topic, primarily for ruby-1.9.1, I'm at a stage where:
# gem install facter
installs facter to /usr/local/share/ruby/gems/facter-<version>, along with the facter.gemspec in /usr/local/share/ruby/specifications/ and so forth, but facter itself won't work.
Attached is an strace where you can recognize the gross inefficiency of Ruby searching through all kinds of paths looking for facter.
I'm interested to know how you guys think we might solve this problem.
-- Jeroen
== # /usr/local/share/ruby/gems/facter-1.5.7/bin/facter /usr/local/share/ruby/gems/facter-1.5.7/bin/facter:50:in `require': no such file to load -- facter (LoadError) from /usr/local/share/ruby/gems/facter-1.5.7/bin/facter:50:in `<main>'
On 12/24/2009 01:32 PM, Jeroen van Meeuwen wrote:
On 12/22/2009 12:32 AM, Jeroen van Meeuwen wrote:
Hi guys,
after FUDCon, and then some more thoughts in private email and between my ears, I think I might have the bare bones for a plan to attack the Ruby challenge;
Continuing on this topic, primarily for ruby-1.9.1, I'm at a stage where:
# gem install facter
installs facter to /usr/local/share/ruby/gems/facter-<version>, along with the facter.gemspec in /usr/local/share/ruby/specifications/ and so forth, but facter itself won't work.
Attached is an strace where you can recognize the gross inefficiency of Ruby searching through all kinds of paths looking for facter.
Did I forget to attach the strace?
-- Jeroen
Reviving an old discussion.
On Tue, 2009-12-22 at 00:32 +0100, Jeroen van Meeuwen wrote:
- ruby-1.8.5 as shipped in EL-5
- ruby-1.8.6 as shipped in Fedora currently, and most likely also EL-6
- ruby-1.9.1 as targeted for Fedora 13
I don't take it as a given that we need to support two parallel stacks, since that leads to major gyrations. Do we have any idea how much breakage there is on a 1.9.x stack ?
There's only ~ 250 ruby and rubygem packages in Fedora - how about some sort of mass rebuild of these against a 1.9.x to see how badly things break ?
The ruby-<m>.<m>.<t> packages are going to provide:
- ruby(ABI) = <major>.<minor>
- ruby(ABI) = <major>.<minor>.<teeny>
- ruby(API) = <major>.<minor>
- ruby(API) = <major>.<minor>.<teeny>
The Ruby ABI is pretty stable in between teeny versions, but much less so for the Ruby API.
I assume that the ABI/API distinction is there so that pure-ruby packages can depend on ruby(api) and only binary packages require ruby(abi).
Is that really necessary ? And how would packagers use that in a meaningful way ? Do we have examples from 1.8.x where that distinction would have been useful ? I think the main reason for these fine-grained distinctions is the desire for parallel Ruby stacks.
- Search paths are going to look similar to:
/usr/local/lib{,64}/ruby/<major>.<minor>.<teeny>/ /usr/local/lib{,64}/ruby/<major>.<minor>/ /usr/local/lib{,64}/ruby/ /usr/local/share/ruby/<major>.<minor>.<teeny>/ /usr/local/share/ruby/<major>.<minor>/ /usr/local/share/ruby/ /usr/lib{,64}/ruby/<major>.<minor>.<teeny>/ /usr/lib{,64}/ruby/<major>.<minor>/ /usr/lib{,64}/ruby/ /usr/share/ruby/<major>.<minor>.<teeny>/ /usr/share/ruby/<major>.<minor>/ /usr/share/ruby/
Besides the multi-version issues (which we'd probably keep even if we only support 1.9.x), there are two issues with these directories:
* Ruby and Fedora have differing conventions for the location of binary packages: /usr/lib{,64} for Fedora, and $arch subdirs for Ruby * Ruby and Fedora have differing conventions for the location of local files: /usr/local for Fedora, $sitedir for Ruby
We should cater to the combination of these conventions, not their union.
Finally, I don't think it's all that useful to add /usr/local variants to the search path, just in case somebody wants to use them. 90% of 'local' installs will be gem installs into random locations, in which people will have to set GEM_PATH/HOME - the other cases can be covered with a RUBYLIB setting.
With all that, I'd suggest a loadpath of
/usr/local/lib{,64}/ruby/<major>.<minor>/site_ruby/$arch /usr/local/share/ruby/<major>.<minor>/site_ruby /usr/lib{,64}/ruby/<major>.<minor>/vendor/$arch /usr/lib{,64}/ruby/<major>.<minor>/$arch /usr/share/ruby/<major>.<minor>/vendor/ /usr/share/ruby/<major>.<minor>/
Preliminary packages can be found on:
For Fedora 12:
For Fedora Rawhide:
That said, these packages could ruin your current Ruby stack (different search paths and all that).
Please be so kind as to test these versions and provide me with the necessary feedback, so we can move things along.
Did you say 'get yourself a rawhide VM' ?
David
On Wed, Feb 3, 2010 at 12:39 PM, David Lutterkort lutter@redhat.com wrote:
Reviving an old discussion.
On Tue, 2009-12-22 at 00:32 +0100, Jeroen van Meeuwen wrote:
- ruby-1.8.5 as shipped in EL-5
- ruby-1.8.6 as shipped in Fedora currently, and most likely also EL-6
- ruby-1.9.1 as targeted for Fedora 13
I don't take it as a given that we need to support two parallel stacks, since that leads to major gyrations. Do we have any idea how much breakage there is on a 1.9.x stack ?
I still think puppet has issues with 1.9. As far as I am concerned, that's a show stopper for moving ruby to 1.9 and not having 1.8. I tried puppet on 1.9 last week with some issues :(
stahnma
On Wed, 2010-02-03 at 19:58 -0600, Michael Stahnke wrote:
On Wed, Feb 3, 2010 at 12:39 PM, David Lutterkort lutter@redhat.com wrote:
Reviving an old discussion.
On Tue, 2009-12-22 at 00:32 +0100, Jeroen van Meeuwen wrote:
- ruby-1.8.5 as shipped in EL-5
- ruby-1.8.6 as shipped in Fedora currently, and most likely also EL-6
- ruby-1.9.1 as targeted for Fedora 13
I don't take it as a given that we need to support two parallel stacks, since that leads to major gyrations. Do we have any idea how much breakage there is on a 1.9.x stack ?
I still think puppet has issues with 1.9. As far as I am concerned, that's a show stopper for moving ruby to 1.9 and not having 1.8.
Yes, definitely.
I tried puppet on 1.9 last week with some issues :(
Did you file bugs upstream for that ?
The way I see it, the migration pain is inevitable; we can either go through it now, lend upstream a hand, and save ourselves a compat stack, or do it piecemeal with a mess of packages that work on 1.9 and on 1.8 selectively.
David
On 02/04/2010 02:32 AM, David Lutterkort wrote:
I tried puppet on 1.9 last week with some issues :(
Did you file bugs upstream for that ?
There are a few bugs upstream about puppet and 1.9. Some of these are fairly long standing ( http://projects.reductivelabs.com/issues/2191 ) and I seem to recall a conversation recently that went along the lines of '1.9 isnt on the horizon at the moment, maybe for puppet-1;.
On 02/03/2010 07:39 PM, David Lutterkort wrote:
Reviving an old discussion.
On Tue, 2009-12-22 at 00:32 +0100, Jeroen van Meeuwen wrote:
- ruby-1.8.5 as shipped in EL-5
- ruby-1.8.6 as shipped in Fedora currently, and most likely also EL-6
- ruby-1.9.1 as targeted for Fedora 13
I don't take it as a given that we need to support two parallel stacks, since that leads to major gyrations. Do we have any idea how much breakage there is on a 1.9.x stack ?
Actually, I don't. Most of all upstreams will have had to comply with the 1.9.x series at some point, and to some extent, 'cause the ruby developers started using it but I'm completely unfamiliar with how far they actually are.
There's only ~ 250 ruby and rubygem packages in Fedora - how about some sort of mass rebuild of these against a 1.9.x to see how badly things break ?
For one though, ruby-shadow is one of those libs that apparently noone ever uses or actively works on. We'd be looking at a potential small heap of these kinds of problems.
The ruby-<m>.<m>.<t> packages are going to provide:
- ruby(ABI) = <major>.<minor>
- ruby(ABI) = <major>.<minor>.<teeny>
- ruby(API) = <major>.<minor>
- ruby(API) = <major>.<minor>.<teeny>
The Ruby ABI is pretty stable in between teeny versions, but much less so for the Ruby API.
I assume that the ABI/API distinction is there so that pure-ruby packages can depend on ruby(api) and only binary packages require ruby(abi).
Is that really necessary ? And how would packagers use that in a meaningful way ? Do we have examples from 1.8.x where that distinction would have been useful ? I think the main reason for these fine-grained distinctions is the desire for parallel Ruby stacks.
While the ABI has always been stable pretty much, the 1.8.x series has had some minor yet significant API changes. Long story short, I'm not trusting upstream to handle this very well, even within a stable stream, and so I want to provide a safe-guard for us and our consumers.
One would want to be able to specifically require the Major.Minor.Teeny version of the stack in order to prevent the problem from happening (and being able to act proactively by "missing dependency"). For us, we can still go with Major.Minor requirements for 99% of all cases.
- Search paths are going to look similar to:
/usr/local/lib{,64}/ruby/<major>.<minor>.<teeny>/ /usr/local/lib{,64}/ruby/<major>.<minor>/ /usr/local/lib{,64}/ruby/ /usr/local/share/ruby/<major>.<minor>.<teeny>/ /usr/local/share/ruby/<major>.<minor>/ /usr/local/share/ruby/ /usr/lib{,64}/ruby/<major>.<minor>.<teeny>/ /usr/lib{,64}/ruby/<major>.<minor>/ /usr/lib{,64}/ruby/ /usr/share/ruby/<major>.<minor>.<teeny>/ /usr/share/ruby/<major>.<minor>/ /usr/share/ruby/
Besides the multi-version issues (which we'd probably keep even if we only support 1.9.x), there are two issues with these directories:
* Ruby and Fedora have differing conventions for the location of binary packages: /usr/lib{,64} for Fedora, and $arch subdirs for Ruby * Ruby and Fedora have differing conventions for the location of local files: /usr/local for Fedora, $sitedir for RubyWe should cater to the combination of these conventions, not their union.
What we do in Ruby now (the most recent set of packages in my repositories) is make /usr/share/ be the vendorlib, and /usr/lib64/ be the vendorarch. The sitelib and sitearch end up in /usr/local/.
Finally, I don't think it's all that useful to add /usr/local variants to the search path, just in case somebody wants to use them. 90% of 'local' installs will be gem installs into random locations, in which people will have to set GEM_PATH/HOME - the other cases can be covered with a RUBYLIB setting.
With all that, I'd suggest a loadpath of
/usr/local/lib{,64}/ruby/<major>.<minor>/site_ruby/$arch /usr/local/share/ruby/<major>.<minor>/site_ruby /usr/lib{,64}/ruby/<major>.<minor>/vendor/$arch /usr/lib{,64}/ruby/<major>.<minor>/$arch /usr/share/ruby/<major>.<minor>/vendor/ /usr/share/ruby/<major>.<minor>/
The $arch, site_ruby and vendor are duplication.
The aforementioned modifications listed are actually done quite easily; easier and cleaner then the modifications used to be moving everything to %{_libdir} (but not %{_libdir} for 64-bit).
The question quickly became, how would we want to handle packaging a .gem or .so package, provided we would have had multiple stacks. Now that we're dropping that specific hurdle... It's a question of rebuilding everything.
Preliminary packages can be found on:
For Fedora 12:
For Fedora Rawhide:
That said, these packages could ruin your current Ruby stack (different search paths and all that).
Please be so kind as to test these versions and provide me with the necessary feedback, so we can move things along.
Did you say 'get yourself a rawhide VM' ?
It doesn't have to be rawhide, just something you're not doing any serious Ruby work on; that's why I include Fedora 12 packages ;-)
In order to test I need to be able to yum -y remove *ruby* every once in a while, or even the occasional rpm -e --nodeps / rpm -Uvh --force. I can imagine you would not want to do that on your primary workstation, and I can only imagine what pains you would create running a dedicated Rawhide VM for it ;-)
-- Jeroen
On 02/04/2010 04:18 AM, Jeroen van Meeuwen wrote:
On 02/03/2010 07:39 PM, David Lutterkort wrote:
Reviving an old discussion.
On Tue, 2009-12-22 at 00:32 +0100, Jeroen van Meeuwen wrote:
- ruby-1.8.5 as shipped in EL-5
- ruby-1.8.6 as shipped in Fedora currently, and most likely also EL-6
- ruby-1.9.1 as targeted for Fedora 13
I don't take it as a given that we need to support two parallel stacks, since that leads to major gyrations. Do we have any idea how much breakage there is on a 1.9.x stack ?
Actually, I don't. Most of all upstreams will have had to comply with the 1.9.x series at some point, and to some extent, 'cause the ruby developers started using it but I'm completely unfamiliar with how far they actually are.
Also it's a given that there will always be those projects / developers that haven't / won't migrate. The question is at what percentage (or perceived percentage) do we say that's it we're switching over. Even though it might be tricky to support the parallel stacks I do think 1.8 is still going to be around / used for a while, and it'll probably be a tough sell to push back at all the upstream communities to update for a while (afterall most ruby developers just use gem for all their package management).
Fundamentally a 'ruby version needed' field is needed in the gem spec and/or as a feature on gemcutter, if some way to derive this info doesn't exist already. Obviously all the gems would need to be updated to include this info on a voluntary basis, which still poses a problem, but would at least help push this dependency problem/solution to the package source / upstream.
There's only ~ 250 ruby and rubygem packages in Fedora - how about some sort of mass rebuild of these against a 1.9.x to see how badly things break ?
For one though, ruby-shadow is one of those libs that apparently noone ever uses or actively works on. We'd be looking at a potential small heap of these kinds of problems.
Also we need to consider gems to be packaged, just because a non-1.9-compliant gem isn't in Fedora yet, doesn't mean there won't be need for it sometime in the future.
The ruby-<m>.<m>.<t> packages are going to provide:
- ruby(ABI) =<major>.<minor>
- ruby(ABI) =<major>.<minor>.<teeny>
- ruby(API) =<major>.<minor>
- ruby(API) =<major>.<minor>.<teeny>
The Ruby ABI is pretty stable in between teeny versions, but much less so for the Ruby API.
I assume that the ABI/API distinction is there so that pure-ruby packages can depend on ruby(api) and only binary packages require ruby(abi).
Is that really necessary ? And how would packagers use that in a meaningful way ? Do we have examples from 1.8.x where that distinction would have been useful ? I think the main reason for these fine-grained distinctions is the desire for parallel Ruby stacks.
While the ABI has always been stable pretty much, the 1.8.x series has had some minor yet significant API changes. Long story short, I'm not trusting upstream to handle this very well, even within a stable stream, and so I want to provide a safe-guard for us and our consumers.
Agree on provide a buffer for upstream changes, but also not sure about the API/ABI distinction. Are these maintained / managed separately in the upstream community? If every API release has a corresponding ABI release, even if it doesn't have any changes, it seems making the distinction would just add complexity. Are there other examples of where this is done?
One would want to be able to specifically require the Major.Minor.Teeny version of the stack in order to prevent the problem from happening (and being able to act proactively by "missing dependency"). For us, we can still go with Major.Minor requirements for 99% of all cases.
- Search paths are going to look similar to:
/usr/local/lib{,64}/ruby/<major>.<minor>.<teeny>/ /usr/local/lib{,64}/ruby/<major>.<minor>/ /usr/local/lib{,64}/ruby/ /usr/local/share/ruby/<major>.<minor>.<teeny>/ /usr/local/share/ruby/<major>.<minor>/ /usr/local/share/ruby/ /usr/lib{,64}/ruby/<major>.<minor>.<teeny>/ /usr/lib{,64}/ruby/<major>.<minor>/ /usr/lib{,64}/ruby/ /usr/share/ruby/<major>.<minor>.<teeny>/ /usr/share/ruby/<major>.<minor>/ /usr/share/ruby/
Besides the multi-version issues (which we'd probably keep even if we only support 1.9.x), there are two issues with these directories:
* Ruby and Fedora have differing conventions for the location of binary packages: /usr/lib{,64} for Fedora, and $arch subdirs for Ruby * Ruby and Fedora have differing conventions for the location of local files: /usr/local for Fedora, $sitedir for RubyWe should cater to the combination of these conventions, not their union.
What we do in Ruby now (the most recent set of packages in my repositories) is make /usr/share/ be the vendorlib, and /usr/lib64/ be the vendorarch. The sitelib and sitearch end up in /usr/local/.
This is not the current case w/ ruby / rubygems in the current Fedora repo correct? Regardless if I yum or gem install a ruby gem, it currently ends up in /usr/lib/ruby/gems/.
With these new ruby package guidelines is the idea to put a gem in /usr/ when installing it via yum and to put it in/usr/local when installing it via gem? Getting the major.minor.teeny path down for the rpm/yum scenario shouldn't be to hard so long as the spec specifies the dependencies correctly but as far as doing the same via gem might be trickier as it currently only defaults to the current ruby version's major.minor. Is this something we would have to patch rubygems to do?
Finally, I don't think it's all that useful to add /usr/local variants to the search path, just in case somebody wants to use them. 90% of 'local' installs will be gem installs into random locations, in which people will have to set GEM_PATH/HOME - the other cases can be covered with a RUBYLIB setting.
With all that, I'd suggest a loadpath of
/usr/local/lib{,64}/ruby/<major>.<minor>/site_ruby/$arch /usr/local/share/ruby/<major>.<minor>/site_ruby /usr/lib{,64}/ruby/<major>.<minor>/vendor/$arch /usr/lib{,64}/ruby/<major>.<minor>/$arch /usr/share/ruby/<major>.<minor>/vendor/ /usr/share/ruby/<major>.<minor>/The $arch, site_ruby and vendor are duplication.
The aforementioned modifications listed are actually done quite easily; easier and cleaner then the modifications used to be moving everything to %{_libdir} (but not %{_libdir} for 64-bit).
The question quickly became, how would we want to handle packaging a .gem or .so package, provided we would have had multiple stacks. Now that we're dropping that specific hurdle... It's a question of rebuilding everything.
Would you say the major hurdle right now w/ multiple stacks is testing / making sure everything works side by side? Multiple version support is definitely tricky but also needed.
Preliminary packages can be found on:
For Fedora 12:
For Fedora Rawhide:
That said, these packages could ruin your current Ruby stack (different search paths and all that).
Please be so kind as to test these versions and provide me with the necessary feedback, so we can move things along.
Did you say 'get yourself a rawhide VM' ?
It doesn't have to be rawhide, just something you're not doing any serious Ruby work on; that's why I include Fedora 12 packages ;-)
In order to test I need to be able to yum -y remove *ruby* every once in a while, or even the occasional rpm -e --nodeps / rpm -Uvh --force. I can imagine you would not want to do that on your primary workstation, and I can only imagine what pains you would create running a dedicated Rawhide VM for it ;-)
-- Jeroen
How would you say would be the best way to go about testing the support for the different ruby versions short of finding / picking random packages / features that are supported in different interpreters? I feel a formal testing strategy is needed, which is definitely hard given the scope of what is being done.
I hope to gear my polisher tool to help testing some of this stuff out for us, hopefully setting up some automated testing environments / scenarios which we can use to test features. In that same vein, I briefly looked into rvm, which might help us w/ some of this testing, though probably not for packaging. It brings the Java approach to ruby allowing you to package all your gem dependencies w/ your ruby application. We might be able to leverage the stack isolation for various Fedora testing purposes, but ultimately dependencies all have to be independently packaged.
Any thoughts on other ways to help test this stuff?
-Mo
ruby-sig@lists.fedoraproject.org
-
David Lutterkort -
Jeroen van Meeuwen -
Karanbir Singh -
Michael Stahnke -
Mo Morsi