Ian Pilcher wrote:

Something I think is missing....

I believe that there needs to be a clear strategy for the co-existence of Fedora gcj packages and JPackage "pure Java" packages. The current situation is a mess:

• If the JPackage repos are added to a FC4 yum configuration, yum will try to overwrite gcj packages with native packages, and vice versa.

• There's no way for a user to know what the effects of this, beyond performance will be.

etc...

Suggestions welcome :)

Ian Pilcher writes:

Gary Benson wrote:

...

Suggestions welcome :)

I'm not sure what the best solution is, but here are some alternatives to consider:

1. Change the names of gcj packages, so that they don't match the JPackage versions. For example, gcj-tomcat5 could "provide" tomcat5. It should be pretty easy to conditionalize this in the SPEC files.

2. Move the Fedora gcj packages into a separate repository from the rest of the distribution. This would at least allow people to create a "pure" JPackage system by simply disabling this repo.

This seems like a really bad idea because it doesn't allow gcj packages to provide dependencies for JPackages.

That's all I can think of for now. Perhaps others have ideas.

Andrew.

On Thu, 2005-09-15 at 14:56 +0100, Gary Benson wrote:

Ian Pilcher wrote:

...

I believe that there needs to be a clear strategy for the co-existence of Fedora gcj packages and JPackage "pure Java" packages. The current situation is a mess:

Suggestions welcome :)

Distribute JPackage packages "as is". For each JPP package shipped in Fedora, create a corresponding "<jppname>-bin" package containing the compiled library code.

RPM will know about the "-bin" dependencies, and those can depend on each other if needed. One would hope that the binary packages don't need any specific patches, but if they do, feed them up to JPackage. Maybe split JPP into not just "Free" and "Non-Free", but also "Free and works on Free compilers/runtimes". Work to get upstream projects like Apache to abstract code with Non-Free dependencies into separate Ant targets, facilitating the creation of sub-packages for RPM's also abstracting away Non-Free code.

Eventually work towards automating the binary builds from JPP SRPM's, with the ultimate goal of having a Yum repository which parallels the JPackage one, or is even included as part of JPP, containing "-bin" RPM's for all packages. Or you could create a whole Fedora specific Java repository to replace using JPP, and it could suck in the JPP srpm's, compile, and provide both noarch and arch results.

On Thursday 15 September 2005 12:52, Gary Benson wrote:

There are several reasons not to do this but the showstopper is that if someone does 'yum install jonas-examples' on an empty system they will not get the native code.

You may extend the RPM spec file with an optional header like the following:

Name: jonas-examples

... yadda yadda yadda ...

CanBeMassivelySpedUpBy: jonas-examples-compiled-by-gcj

With this in place, yum can now fetch jonas-examples-compiled-by-gcj if is available.

This requires changes to both rpm and yum. I have no idea how feasible this would be, but I don't remember this option having been discussed yet.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Vadim Nasardinov wrote:

You may extend the RPM spec file with an optional header like the following:

I thought rpm has something like `Suggests' now, but this won't force a package.

Anyway, I think having the only reason for binary java packages be that new users can't figure out how to install them is a pretty bad reason, since you could always force the install somehow (without modifying rpm) by inserting code into yum that matches *jpp* and checks for a native package to install on top of it.

- -- Sincerely,

David Walluck david@zarb.org

On Thursday 15 September 2005 12:24, Chris Hubick wrote:

Maybe split JPP into not just "Free" and "Non-Free", but also "Free and works on Free compilers/runtimes".

Although this would probably take more effort than anyone's is willing to invest at this point, I do like the general sound of this idea.

On Thursday 15 September 2005 13:27, Thomas Fitzsimmons wrote:

In other words, if our Fedora package versions always match or exceed their JPackage equivalents

That's a pretty big "if", is it not?

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Gary Benson wrote:

Couldn't have said it better myself. The only reason people are starting to get strange mixes of packages is that Fedora has lagged JPackage because there's not presently the manpower to keep them level.

There's no guarantee of this and, in fact, it is quite likely that jpackage versions will supercede FC4 versions (except there isn't much jpackage manpower right now, either). That is, unless you were to bump the epoch on every FC4 package as was done with eclipse (why the special case here I am not sure).

On the other hand, people want more compatibility. And I want less (ideally no) duplication. So I would again ask for some policy that separates native libs from the jars. Or at least maybe someone is willing to put some real thought into it rather than dismissing it outright. By the way, I say this having fallen victim to the exact same practices myself, where I have made patches and even full version updates to several packages and just have had no time to push these changes back to jpackage.org.

- -- Sincerely,

David Walluck david@zarb.org

Gary Benson wrote:

David Walluck wrote:

...

On the other hand, people want more compatibility. And I want less (ideally no) duplication. So I would again ask for some policy that separates native libs from the jars. Or at least maybe someone is willing to put some real thought into it rather than dismissing it outright.

Before fedora-devel-java-list was set up we had precisely this discussion internally, several times as it happens ;) If I answer you quickly it's not because I'm dismissing it but rather that I've already thought long and hard about it beforehand.

I've been trying to dig out the discussion all morning, but I keep getting sidetracked with email. So, if you'll bear with me a few moments longer...

Well, I'd forgotton just how much was said about it, but a brief summary is that there were three methods proposed:

1. Native code in the same rpms as the bytecode. 2. Native code in separate rpms to the bytecode. 3. Native code generated on user's machine.

Method 3, which hasn't been mentioned on this list, involved either %post scriptlets or some kind of background job like slocate has. It's main drawbacks were that native compilation is a hugely computationally intensive process, particularly with aggressive inter-class optimizations. Many target machines simply wouldn't have the RAM to do the job. Even were this not the case, it can frequently take hours, too long for a %post scriptlet, and using a background job would mean that there'd be periods when your native code was not in sync and your Java apps would run slowly.

Method 2, which is what you're proposing, has the following drawbacks over Method 1:

- There is no way at present to make rpm (yum, up2date, anaconda...) associate the native packages with the bytecode ones. Users would have to manually install them in the first instance.

- Installed native packages must Require their bytecode equivalents. Users could not upgrade a Fedora package with a JPackage one without manually uninstalling the native package first.

- The native package would not have access to the sources, and would not be able to generate a useful debuginfo package. This would render the packages undebuggable with gdb.

- There's no obvious way to automate the generation of the native packages. This causes the packagecount to more than double, and the non-atomic nature of it allows things to get out of sync and, for example, break rawhide.

But all this is somewhat tangential to the original point, which was that "[if] the JPackage repos are added to a FC4 yum configuration, yum will try to overwrite [Fedora] packages with [JPackage] packages..." Allowing the Fedora stack to be extended with JPackage packages was and is a key requirement. There's only one way I can see that would prevent the former while still allowing the latter, and that's for Fedora's packages to have the same EVR as the JPackage packages. Since that is something we cannot commit to, the problem will remain regardless of where the native code is packaged.

Gary

On Friday 16 September 2005 07:24, Gary Benson wrote:

a brief summary is that there were three methods proposed:

1. Native code in the same rpms as the bytecode.
2. Native code in separate rpms to the bytecode.
3. Native code generated on user's machine.

Method 2, which is what you're proposing, has the following drawbacks over Method 1:

• There is no way at present to make rpm (yum, up2date, anaconda...) associate the native packages with the bytecode ones. Users would have to manually install them in the first instance.

Yum can be changed to have a way to associate the native packages with the bytecode ones.

• Installed native packages must Require their bytecode equivalents. Users could not upgrade a Fedora package with a JPackage one without manually uninstalling the native package first.

Yum can be changed to handle this.

• The native package would not have access to the sources, and would not be able to generate a useful debuginfo package. This would render the packages undebuggable with gdb.

Can you elaborate? I thought native compilation worked by taking .class files as input. Where do the sources come in? Secondly, why can't you build native packages from the same SRPM that the binary non-native RPMs were built from? I don't believe this has been discussed on this list in sufficient detail.

• There's no obvious way to automate the generation of the native packages. This causes the packagecount to more than double, and the non-atomic nature of it allows things to get out of sync and, for example, break rawhide.

IIRC, David suggested repeatedly that JPackage would be more than happy to accept .spec files where native compilation logic was present conditionally and was off by default. I seem to recall vaguely this suggestion getting dismissed as impractical but don't remember any detailed explanation of why this may not be feasible.

The bottom line is, although the current situation is the best solution we could think of, it is not a good solution. The fact of the matter remains that Fedora and JPackage repositories don't mesh very well at this time. In view of this, why don't we explore the option of adding explicit knowledge of the Java packaging situation to Yum? If this turns out to be a dead-end, fine. At the very least, we will have explained, in a public forum, why smarter Yum is not the answer.

On Friday 16 September 2005 12:52, Andrew Haley wrote:

Vadim Nasardinov writes:

...

On Friday 16 September 2005 07:24, Gary Benson wrote:

...

a brief summary:

1. Native code in the same rpms as the bytecode.
2. Native code in separate rpms to the bytecode.

...

...

Method 2, which is what you're proposing, has the following drawbacks over Method 1:

...

...

• The native package would not have access to the sources, and would not be able to generate a useful debuginfo package. This would render the packages undebuggable with gdb.

Can you elaborate? I thought native compilation worked by taking .class files as input. Where do the sources come in?

You don't debug bytecode: you debug source. Debuginfo contains source.

The part I didn't understand was how specifically Method 2 would prevent you from generating appropriate debuginfo packages. I think the source of my confusion was that I misintepreted Gary's definition of "Method 2". I was thinking along the lines of David Walluck's long-standing proposal (I hope I'm not putting words in his mouth) to share exact same .spec files between Fedora and JPackage but have native-compilation-related spec logic conditionalized.

I am a little fuzzy on why sharing exact same .spec files (modulo minor distro-specific patches) is infeasible.

Even with Gary's definition of Method 2 (call it M2), it's not clear to me why debuginfo packages are impossible to generate. (I think M2 amounts effectively to having two SRPMs for the same application).

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Vadim Nasardinov wrote:

I am a little fuzzy on why sharing exact same .spec files (modulo minor distro-specific patches) is infeasible.

I think it was at my behest that gcj_support was added to some spec files. Anyway, the good news is that I have actually gone and taken a lot (if not all!) FC spec files and added `%if %{gcj_support}' in the appropriate places. I just have been strapped for time so I haven't done much of anything with these packages. Similarly, I really have very few additions I would like to add to eclipse (wrt jpackage), I just haven't had time to do it. So I don't blame Fedora here at all, but myself, since I've actually already done this work and just can't seem to find the time to do the last step.

Also, I obviously am not here to dictate Fedora policy, and we know that without Fedora, the eclipse 3.1 work may never have gotten done. The only thing I am in favor of, though, is a policy that might integrate better cross-distro and/or avoids duplication of work (since I am sure we all don't have the extra time).

In the end, I would prefer two separate spec files for each. It is actually possible to do something with the rpm build scripts to perhaps generate such a package 100% automatically (just as debuginfo packages can be automatically generated now). The inability of rpm to have a noarch subpackage of an arch package prevents what I'd call a near-ideal solution. Already we have aot-compile-rpm in the rpm build scripts but it's not being used. But I think this is a step in the right direction. Perhaps a second rpmbuild process could be launched on the automatically generated native spec file from the non-native build.

- -- Sincerely,

David Walluck david@zarb.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Chris Hubick wrote:

When I read that, my first thought is: It would be nice if the macro's used in JPP packages were generic/abstract enough that they didn't need to mention GCJ at all, and that the definitions of the 'compilation' macro could be modified to include doing .so builds in addition to the class files on platforms with GCJ...

It mentions gcj because it's for gcj only. And gcj could be autodetected by a macro too %(test -x %{_bindir}/gcj...). Remember that any jpackage.org build server is also a platform with native binaries, and probably has gcj installed. The existence of the gcj binary doesn't imply someone necessarily wants native packages.

Then I start to realize that it would probably need to be wedged into the Ant build file called by the macro. Then I start to think it would be nice if the Ant targets were generic enough to be able to do this abstraction...

Anthony Green has written gcjlib, but it is cumbersome to have to patch every single build.xml, which is why a script that does the same thing is generally preferred. But for upstream projects, it might make sense for them to adopt gcjlib.

- -- Sincerely,

David Walluck david@zarb.org

Vadim Nasardinov wrote:

The bottom line is, although the current situation is the best solution we could think of, it is not a good solution. The fact of the matter remains that Fedora and JPackage repositories don't mesh very well at this time.

Placing the native code in separate rpms will not solve this because the problem is not that Fedora rpms contain native code and JPackage ones do not. The problem is that Fedora and JPackage rpms share the same namespace. Ximian had problems with this five years ago, before they were even called Ximian. What is really needed is for some way to tell yum to ignore packages in jpackage.repo that exist already in fedora.repo. I doubt it's that simple though.

Cheers, Gary

David Walluck wrote:

Gary Benson wrote:

...

ones do not. The problem is that Fedora and JPackage rpms share the same namespace. Ximian had problems with this five years ago, before they were even called Ximian. What is really needed is for some way to tell yum to ignore packages in jpackage.repo that exist already in fedora.repo. I doubt it's that simple though.

But FC4 was said to be ``100% JPackage compatible''. Why couldn't updates be provided through JPackage?

Some issues take time to fix properly. The correct fix may not be immediately obvious, or there may be several options that require discussion in some upstream community. Wherever possible I will work around the issue locally so people aren't held up. There's probably 15-20 such issues in rawhide at the moment. Replacing Fedora packages with JPackage ones will lose patches like that.

Also, if a particular package that *is not* in Fedora requires an updated package (an older version of which *is* in Fedora), then one loses the ability to upgrade this package from JPackage as well.

Avoiding breakages without inhibiting things like this is difficult.

Cheers, Gary

Vadim Nasardinov wrote:

On Monday 19 September 2005 05:14, Gary Benson wrote:

...

What is really needed is for some way to tell yum to ignore packages in jpackage.repo that exist already in fedora.repo. I doubt it's that simple though.

Some of the possible choices are:

(a) fedora.repo always trumps jpackage.repo;

(b) jpackage.repo always trumps fedora.repo;

(c) the user has a choice of specifying either (a) or (b) as their default policy.

 People who use java stuff in FC exclusively with GCJ won't want
 to lose the native bits due to updates coming from jpackage.repo.
 People who use java packages primarily under a proprietary JVM
 will be happy to pull upgrades from jpackage.repo even if it
 means losing the .so bits;

(d) go wild and make the choice of repos configurable on a per-package basis.

Although I don't know how simple or difficult this is going to be, I doubt it will be much worse than the current situation. My initial conservative preference would be (c) with fedora.repo trumping jpackage.repo by default.

I like the sound of (c), though it's not just yum that needs to deal with this: anaconda and possibly up2date need to handle it too.

Cheers, Gary

Resurrecting an old thread as I'm catching up...

On 9/19/05, Vadim Nasardinov vadimn@redhat.com wrote:

...

What is really needed is for some way to tell yum to ignore packages in jpackage.repo that exist already in fedora.repo. I doubt it's that simple though.

(d) go wild and make the choice of repos configurable on a per-package basis.

My simple solution to this problem has been to add an exclude= line to my jpackage.repo file listing the names of all the Java packages that shipped with FC4:

[jpackage-generic] name=JPackage (free), generic mirrorlist=http://www.jpackage.org/jpackage_generic.txt failovermethod=priority gpgcheck=1 gpgkey=http://www.jpackage.org/jpackage.asc enabled=1 exclude=ant, ant-antlr, ant-apache-bcel, ant-apache-log4j, ant-apache-oro, ant-apache-regexp, ant-apache-resolver, ant-commons-logging, ...

and so on. The list of packages was built by grepping for [0-9]*jpp_[0-9]*fc from the list of the packages that shipped with FC4, then stripping off the version numbers.

When I first saw the problem mentioned on this list I was surprised that the jpackage.org http://jpackage.org site didn't suggest something like this to prevent JPackage packages from replacing the Fedora ones.

-Mark.

* David Walluck david@zarb.org [2005-09-16 05:26]:

That is, unless you were to bump the epoch on every FC4 package as was done with eclipse (why the special case here I am not sure).

This was totally unrelated and was because we had decided to ship 3.0.2 in FC4 at one point after having 3.1Mx in rawhide. We had to bump the epoch to go back. Then we decided to go with 3.1 in the end. Stupid? Yes.

Andrew