Hi all,
Ruby upstream is implementing more and more stuff directly in Ruby. We already had issues, that build of Ruby required Ruby when we did some modifications [1]. In subsequent ticket, one of Ruby committers said [2]:
... snip ... BASERUBY is already a build requirement ... snip ... I would like to implement more of Ruby using Ruby, so miniruby may
depend on prelude one day.
With recent changes, such as [3], I am afraid that the day has come.
Previously, if you wanted to patch lets say "gem_prelude.rb", it was enough to patch it. But now you *need* Ruby to process it into miniprelude.c. There are possibly 4 ways out of this.
1) Build Ruby in two stages. a) build (mini)ruby, apply patches b) build Ruby using the previously built (mini)ruby.
2) Use previous version of Ruby available in Fedora to bootstrap Ruby. But this does not work ATM, at least when RubyGems are installed. And upstream is doing what they can to make RubyGems inseparable [4].
3) Prepare patches locally and apply the required changes also to the pregenerated files. But the problem here is, that the patches might unpredictably fail between updates. I don't think that they keep any API/ABI promises for the tools used to generate those files.
4) Don't use the upstream tarball, but generate it from sources with patches integrated.
I think we should probably start to take look at 1), specifically into the *miniruby* variant if that is enough. If that is done, the 2) could optionally blend in. And in the mean time use 3) because otherwise I really don't know how to integrate the ABRT hook support. I don't like 4) at all, unless we have some Fedora standardized way of doing so.
On the positive side, 1(2) would allow us to stay better in line with "Pregenerated code" guidelines [5], because there is already quite a lot of pre-generated code shipped in Ruby release tarball.
Thoughts?
Vít
[1] https://src.fedoraproject.org/rpms/ruby/c/c0513dfb8c81a228619c6142195c5117aa...
[2] https://bugs.ruby-lang.org/issues/15306#note-1
[3] https://github.com/ruby/ruby/pull/2655
[4] https://bugs.ruby-lang.org/issues/16431
[5] https://docs.fedoraproject.org/en-US/packaging-guidelines/what-can-be-packag...
With recent changes, such as [3], I am afraid that the day has come.
It seems that the day came on Ruby 2.7.0, right? Ruby 2.7.0 includes the commit [3].
Thoughts?
On the positive side, 1(2) would allow us to stay better in line with "Pregenerated code" guidelines [5], because there is already quite a lot of pre-generated code shipped in Ruby release tarball.
For my first impression,, I agree with the way 1) and 2).
- Use previous version of Ruby available in Fedora to bootstrap Ruby.
But this does not work ATM, at least when RubyGems are installed. And upstream is doing what they can to make RubyGems inseparable [4].
If we create the SRPM for "Use previous version of Ruby available in Fedora to bootstrap Ruby", which name do you like?
The candidates: * rpms/ruby-bootstrap * rpms/bootstrap-ruby * rpms/previous-ruby
Dne 02. 01. 20 v 12:06 Jun Aruga napsal(a):
With recent changes, such as [3], I am afraid that the day has come.
It seems that the day came on Ruby 2.7.0, right? Ruby 2.7.0 includes the commit [3].
Thoughts? On the positive side, 1(2) would allow us to stay better in line with "Pregenerated code" guidelines [5], because there is already quite a lot of pre-generated code shipped in Ruby release tarball.
For my first impression,, I agree with the way 1) and 2).
- Use previous version of Ruby available in Fedora to bootstrap Ruby.
But this does not work ATM, at least when RubyGems are installed. And upstream is doing what they can to make RubyGems inseparable [4].
If we create the SRPM for "Use previous version of Ruby available in Fedora to bootstrap Ruby", which name do you like?
The candidates:
- rpms/ruby-bootstrap
- rpms/bootstrap-ruby
- rpms/previous-ruby
Not this way. We would just incrementally build Ruby using previous Ruby (IOW there would be `BR: ruby` in .spec file). That would be likely good as long as Fedora does not introduce new architecture. Then we would need manually build some Ruby, probably without patches and then use this Ruby for build of Ruby with patches.
I think if we are able to do this right, then we can choose from option 1 and 2 by setting the right flags. Both options has their pros and cons.
Vít
Not this way. We would just incrementally build Ruby using previous Ruby (IOW there would be `BR: ruby` in .spec file). That would be likely good as long as Fedora does not introduce new architecture. Then we would need manually build some Ruby, probably without patches and then use this Ruby for build of Ruby with patches.
I think if we are able to do this right, then we can choose from option 1 and 2 by setting the right flags. Both options has their pros and cons.
I see. The pattern `BR: ruby` in ruby.spec file is actually like what gcc is doing as `BuildRequires: gcc, gcc-c++` in gcc.spec file. Maybe we are able to do this right.
Wow, it has been year since I started this thread. Time flies.
But since I was fighting again with one revert, which was not possible to do without boostrapping, I went ahead and now I have this PR:
https://src.fedoraproject.org/rpms/ruby/pull-request/72
This is the case (1) I have mentioned bellow. As you can see from the patch, it turns out it is not that easy as I was expecting.
* Building `miniruby` target is not enough.
* Loading of dynamic libraries is not enabled by default in miniruby.
* There are required several extensions (ripper and stringio atm).
* There is required some script which acts as "ruby" to glue all this together.
So based on the experience, I have several discussion topics:
* Is it worth of building miniruby with all this hassle? It could be better to build "bootstrap-ruby" instead, which could be possible better interchangeable with system ruby. IOW that would bring this closer to the variant (2).
* Not sure if the miniruby configure options changes a lot. I have the feeling that the build system tries to pull in even the parts which should be unused :/
* Would it be worth of cleaning up the "miniruby" directory, to have more control about what could be used/included. I have diff of the stuff which is in tarball and what is the minimal set of files required to build miniruby at my hand, if anybody is interested.
So what did I miss? Of course I am interested in any feedback and especially in suggestions for improvement.
Vít
Dne 20. 12. 19 v 17:26 Vít Ondruch napsal(a):
Hi all,
Ruby upstream is implementing more and more stuff directly in Ruby. We already had issues, that build of Ruby required Ruby when we did some modifications [1]. In subsequent ticket, one of Ruby committers said [2]:
... snip ... BASERUBY is already a build requirement ... snip ... I would like to implement more of Ruby using Ruby, so miniruby may
depend on prelude one day.
With recent changes, such as [3], I am afraid that the day has come.
Previously, if you wanted to patch lets say "gem_prelude.rb", it was enough to patch it. But now you *need* Ruby to process it into miniprelude.c. There are possibly 4 ways out of this.
- Build Ruby in two stages. a) build (mini)ruby, apply patches b) build
Ruby using the previously built (mini)ruby.
- Use previous version of Ruby available in Fedora to bootstrap Ruby.
But this does not work ATM, at least when RubyGems are installed. And upstream is doing what they can to make RubyGems inseparable [4].
- Prepare patches locally and apply the required changes also to the
pregenerated files. But the problem here is, that the patches might unpredictably fail between updates. I don't think that they keep any API/ABI promises for the tools used to generate those files.
- Don't use the upstream tarball, but generate it from sources with
patches integrated.
I think we should probably start to take look at 1), specifically into the *miniruby* variant if that is enough. If that is done, the 2) could optionally blend in. And in the mean time use 3) because otherwise I really don't know how to integrate the ABRT hook support. I don't like 4) at all, unless we have some Fedora standardized way of doing so.
On the positive side, 1(2) would allow us to stay better in line with "Pregenerated code" guidelines [5], because there is already quite a lot of pre-generated code shipped in Ruby release tarball.
Thoughts?
Vít
[1] https://src.fedoraproject.org/rpms/ruby/c/c0513dfb8c81a228619c6142195c5117aa...
[2] https://bugs.ruby-lang.org/issues/15306#note-1
[3] https://github.com/ruby/ruby/pull/2655
[4] https://bugs.ruby-lang.org/issues/16431
[5] https://docs.fedoraproject.org/en-US/packaging-guidelines/what-can-be-packag...
ruby-sig mailing list -- ruby-sig@lists.fedoraproject.org To unsubscribe send an email to ruby-sig-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ruby-sig@lists.fedoraproject.o...
----- Original Message -----
From: "Vít Ondruch" vondruch@redhat.com To: ruby-sig@lists.fedoraproject.org Sent: Tuesday, November 10, 2020 7:19:38 PM Subject: Re: Two stage Ruby compilation / Bootstrapping
Wow, it has been year since I started this thread. Time flies.
But since I was fighting again with one revert, which was not possible to do without boostrapping, I went ahead and now I have this PR:
Looks Great!
Although I don't see any comment on 0001-Revert-Add-GC.auto_compact-true-false-and-GC.auto_co.patch
- The patch doesn't seem maintainable. (Am I wrong?) - What it's exactly needed for? (Trim miniruby dependencies?)
This is the case (1) I have mentioned bellow. As you can see from the patch, it turns out it is not that easy as I was expecting.
Building `miniruby` target is not enough.
Loading of dynamic libraries is not enabled by default in miniruby.
There are required several extensions (ripper and stringio atm).
There is required some script which acts as "ruby" to glue all this
together.
So based on the experience, I have several discussion topics:
- Is it worth of building miniruby with all this hassle? It could be
better to build "bootstrap-ruby" instead, which could be possible better interchangeable with system ruby. IOW that would bring this closer to the variant (2).
`ruby` project does not use any bootstrap, right? What specifically do you mean by bootstrap-ruby? Is it RPM ~bootstrap?
Incidentally, ruby has `bootstraptest` folder with tests... I don't think it's used for any regular build, but maybe we could use it for such bootstrapping?
- Not sure if the miniruby configure options changes a lot. I have the
feeling that the build system tries to pull in even the parts which should be unused :/
- Would it be worth of cleaning up the "miniruby" directory, to have
more control about what could be used/included. I have diff of the stuff which is in tarball and what is the minimal set of files required to build miniruby at my hand, if anybody is interested.
Is upstream interested in such trimming? Seems like a worth-while thing to have. Or is it not upstreamable?
So what did I miss? Of course I am interested in any feedback and especially in suggestions for improvement.
Is there a build? simple-koji-ci does not seem to run, same for tmt CI.
Is there any testing we can do to help?
Thanks, Pavel
Vít
Dne 20. 12. 19 v 17:26 Vít Ondruch napsal(a):
Hi all,
Ruby upstream is implementing more and more stuff directly in Ruby. We already had issues, that build of Ruby required Ruby when we did some modifications [1]. In subsequent ticket, one of Ruby committers said [2]:
... snip ... BASERUBY is already a build requirement ... snip ... I would like to implement more of Ruby using Ruby, so miniruby may
depend on prelude one day.
With recent changes, such as [3], I am afraid that the day has come.
Previously, if you wanted to patch lets say "gem_prelude.rb", it was enough to patch it. But now you *need* Ruby to process it into miniprelude.c. There are possibly 4 ways out of this.
- Build Ruby in two stages. a) build (mini)ruby, apply patches b) build
Ruby using the previously built (mini)ruby.
- Use previous version of Ruby available in Fedora to bootstrap Ruby.
But this does not work ATM, at least when RubyGems are installed. And upstream is doing what they can to make RubyGems inseparable [4].
- Prepare patches locally and apply the required changes also to the
pregenerated files. But the problem here is, that the patches might unpredictably fail between updates. I don't think that they keep any API/ABI promises for the tools used to generate those files.
- Don't use the upstream tarball, but generate it from sources with
patches integrated.
I think we should probably start to take look at 1), specifically into the *miniruby* variant if that is enough. If that is done, the 2) could optionally blend in. And in the mean time use 3) because otherwise I really don't know how to integrate the ABRT hook support. I don't like 4) at all, unless we have some Fedora standardized way of doing so.
On the positive side, 1(2) would allow us to stay better in line with "Pregenerated code" guidelines [5], because there is already quite a lot of pre-generated code shipped in Ruby release tarball.
Thoughts?
Vít
[1] https://src.fedoraproject.org/rpms/ruby/c/c0513dfb8c81a228619c6142195c5117aa...
[2] https://bugs.ruby-lang.org/issues/15306#note-1
[3] https://github.com/ruby/ruby/pull/2655
[4] https://bugs.ruby-lang.org/issues/16431
[5] https://docs.fedoraproject.org/en-US/packaging-guidelines/what-can-be-packag...
Dne 11. 11. 20 v 3:19 Pavel Valena napsal(a):
----- Original Message -----
From: "Vít Ondruch" vondruch@redhat.com To: ruby-sig@lists.fedoraproject.org Sent: Tuesday, November 10, 2020 7:19:38 PM Subject: Re: Two stage Ruby compilation / Bootstrapping
Wow, it has been year since I started this thread. Time flies.
But since I was fighting again with one revert, which was not possible to do without boostrapping, I went ahead and now I have this PR:
Looks Great!
Although I don't see any comment on 0001-Revert-Add-GC.auto_compact-true-false-and-GC.auto_co.patch
- The patch doesn't seem maintainable. (Am I wrong?)
- What it's exactly needed for? (Trim miniruby dependencies?)
This is just "random" patch which can't be applied without having some Ruby in place. If you apply it, the build immediately fails:
~~~
+ /usr/bin/make -O -j8 V=1 VERBOSE=1 'COPY=cp -p' Q= BASERUBY = echo executable host ruby is required. use --with-baseruby option.; false CC = gcc LD = ld LDSHARED = gcc -shared CFLAGS = -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -m64 XCFLAGS = -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-strict-overflow -DRUBY_DEVEL=1 -fvisibility=hidden -fexcess-precision=standard -DRUBY_EXPORT -DCANONICALIZATION_FOR_MATHN -I. -I.ext/include/x86_64-linux -I./include -I. -I./enc/unicode/12.1.0 CPPFLAGS = DLDFLAGS = -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-soname,libruby.so.3.0 -fstack-protector-strong -m64 SOLIBS = -lpthread -lrt -lrt -lgmp -ldl -lcrypt -lm LANG = C LC_ALL = LC_CTYPE = MFLAGS = -j8 -Otarget --jobserver-auth=4,5 gcc (GCC) 10.2.1 20201016 (Red Hat 10.2.1-6) Copyright (C) 2020 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
echo executable host ruby is required. use --with-baseruby option.; false ./tool/generic_erb.rb -I. -o miniprelude.c \ ./template/prelude.c.tmpl ./ast.rb ./dir.rb ./gc.rb ./integer.rb ./io.rb ./pack.rb ./trace_point.rb ./warning.rb ./array.rb ./kernel.rb ./ractor.rb ./prelude.rb ./gem_prelude.rb executable host ruby is required. use --with-baseruby option. make: *** [uncommon.mk:1113: miniprelude.c] Error 1 make: *** Waiting for unfinished jobs....
~~~
But if you really want to know what is the patch about, the please look here:
https://bugs.ruby-lang.org/issues/17306
This is the case (1) I have mentioned bellow. As you can see from the patch, it turns out it is not that easy as I was expecting.
Building `miniruby` target is not enough.
Loading of dynamic libraries is not enabled by default in miniruby.
There are required several extensions (ripper and stringio atm).
There is required some script which acts as "ruby" to glue all this
together.
So based on the experience, I have several discussion topics:
- Is it worth of building miniruby with all this hassle? It could be
better to build "bootstrap-ruby" instead, which could be possible better interchangeable with system ruby. IOW that would bring this closer to the variant (2).
`ruby` project does not use any bootstrap, right? What specifically do you mean by bootstrap-ruby? Is it RPM ~bootstrap?
Incidentally, ruby has `bootstraptest` folder with tests... I don't think it's used for any regular build, but maybe we could use it for such bootstrapping?
If you checkout Ruby repository, you won't be able to build it without Ruby available. You should understand that the tarball we are using to build Ruby is already processed, therefore it does not need Ruby, but that could be perceived to be against guidelines to "build everything from source".
But my question was if building just miniruby for boostrap is not too much hastle, maybe it would be easier to build full featured ruby.
- Not sure if the miniruby configure options changes a lot. I have the
feeling that the build system tries to pull in even the parts which should be unused :/
- Would it be worth of cleaning up the "miniruby" directory, to have
more control about what could be used/included. I have diff of the stuff which is in tarball and what is the minimal set of files required to build miniruby at my hand, if anybody is interested.
Is upstream interested in such trimming? Seems like a worth-while thing to have. Or is it not upstreamable?
I don't think upstream is really interested. At several occasions I was told that Ruby build depends on Ruby and upstream is fine with it. Recent steps such as [1] are using Ruby even more then previously.
Anyway, thinking about this boostrap project over night and thinking about this question, I think that my next step should be trying to use the miniruby after `make clean`, which should bring the tarball content closer to upstream repository. I'll see what more is needed to build Ruby and I might consider reporting my findings upstream. But I am afraid this will require more time then I can spend on this.
So what did I miss? Of course I am interested in any feedback and especially in suggestions for improvement.
Is there a build? simple-koji-ci does not seem to run, same for tmt CI.
Is there any testing we can do to help?
Yes, forget about simple-koji-ci and get hands dirty with building and experimenting with .spec file yourself. This is not much about if the produced build of Ruby is right, but how to produce it.
Vít
Dne 11. 11. 20 v 9:35 Vít Ondruch napsal(a):
Anyway, thinking about this boostrap project over night and thinking about this question, I think that my next step should be trying to use the miniruby after `make clean`, which should bring the tarball content closer to upstream repository. I'll see what more is needed to build Ruby and I might consider reporting my findings upstream. But I am afraid this will require more time then I can spend on this.
So here is the next step:
https://src.fedoraproject.org/rpms/ruby/pull-request/72
https://src.fedoraproject.org/fork/vondruch/rpms/ruby/c/ecc88a455f7ffdfd3cd1...
So it is doable. I am still undecided, if building miniruby is worth of the hassle. Building full Ruby could shorten these ~35 lines [1] into `make` line. The downside would be slightly longer build time. The upside would be probably bigger robustness. I'll try that as a next step.
Vít
[1]: https://src.fedoraproject.org/rpms/ruby/pull-request/72#_2__65-101
ruby-sig@lists.fedoraproject.org
-
Jun Aruga -
Pavel Valena -
Vít Ondruch