A new major version (2.0.0) of Java Packages Tools (javapackages) has just been released. Java Packages Tools is a collection of all kind tools, script and macros to support packaging Java.
Starting from version 2.0.0 Java Packages Tools includes functionality of JPackage Utilities (jpackage-utils) that is useful for Fedora. This will allow jpackage-utils package to be obsoleted and provided by javapackages-tools (see below for more detailed explanation).
%add_maven_depmap macro now injects pom.properties to every JAR file it processes. This will allow XMVn Subst to be used to replace any JAR files that are in system Maven repository, not only those built with Maven. It will allow xmvn-subst to be effectively used to replace bundled JARs with symlinks after package built, which will allow simplification of packages containing complex symlinking in %install sections (for example jboss-as) as well as many Eclipse plugins.
%add_to_maven_depmap and %update_maven_depmap macros were removed. These macros have been obsolete for long time. Their usage was preventing Fedora from implementing some features related to Maven metadata. Packages using these macros will have to be ported to use %add_maven_depmap or they will fail to build.
maven2jpp-mapdeps.xsl template has been removed. This template wasn't been used. XMvn has its own way of mapping dependencies. To best of my knowledge nothing in Fedora used this template.
Macros related to installation of icons and desktop files were removed. These legacy macros were out of scope of Java Packages Tools -- standard Fedora macros should be used instead.
14 new manual pages were added. These manpages document RPM macros (such ad %pom_* macros) and commands related to packaging Java. More manual pages are planned in future.
Documentation specific to JPackage and irrelevant to fedora was removed. Most notably this requires Jpackage policy -- in Fedora packaging guidelines should be followed instead.
Reasons for merging jpackage-utils into javapackages-tools ----------------------------------------------------------
In Fedora jpackage-utils was based on old version (year 2006) coming from JPackage. It included a number of fedora-specific patches which weren't accepted by upstream. Java Packages Tools was a project founded by Fedora developers to maintain Fedora-specific macros and various helper scripts. The most important reasons for merging them follow:
1) These two packages had the same purpose and were strong components -- jpackage-utils required javapackages-tools and vice versa.
2) jpackage-utils contained some legacy code which was preventing Fedora from implementing new features. This code couldn't be removed because heavy diversion from upstream is discouraged.
3) jpackage-utils didn't have update to new upstream version for 7 years -- it was de-facto maintained by Fedora developers. Removing custom patchsets from our packages allows for better separation of upstream code from RPM packaging and simplifies overall maintenance.
The new Java Package Tools haven't yet been built for rawhide, but I am going to do this today. The new jpackage-utils is going to obsolete and provide jpackage-utils.
On Thu, 2013-07-18 at 15:39 +0200, Mikolaj Izdebski wrote:
A new major version (2.0.0) of Java Packages Tools (javapackages) has just been released. Java Packages Tools is a collection of all kind tools, script and macros to support packaging Java.
Starting from version 2.0.0 Java Packages Tools includes functionality of JPackage Utilities (jpackage-utils) that is useful for Fedora. This will allow jpackage-utils package to be obsoleted and provided by javapackages-tools (see below for more detailed explanation).
%add_maven_depmap macro now injects pom.properties to every JAR file it processes. This will allow XMVn Subst to be used to replace any JAR files that are in system Maven repository, not only those built with Maven. It will allow xmvn-subst to be effectively used to replace bundled JARs with symlinks after package built, which will allow simplification of packages containing complex symlinking in %install sections (for example jboss-as) as well as many Eclipse plugins.
%add_to_maven_depmap and %update_maven_depmap macros were removed. These macros have been obsolete for long time. Their usage was preventing Fedora from implementing some features related to Maven metadata. Packages using these macros will have to be ported to use %add_maven_depmap or they will fail to build.
maven2jpp-mapdeps.xsl template has been removed. This template wasn't been used. XMvn has its own way of mapping dependencies. To best of my knowledge nothing in Fedora used this template.
Macros related to installation of icons and desktop files were removed. These legacy macros were out of scope of Java Packages Tools -- standard Fedora macros should be used instead.
14 new manual pages were added. These manpages document RPM macros (such ad %pom_* macros) and commands related to packaging Java. More manual pages are planned in future.
Thanks! This will be very useful indeed!
Cheers, Severin
On 07/18/2013 09:39 AM, Mikolaj Izdebski wrote:
%add_maven_depmap macro now injects pom.properties to every JAR file it processes. This will allow XMVn Subst to be used to replace any JAR
For some of our products, we may end up having jars in two (or more) separate locations where symlinking may not be possible. Because of this, I try to never modify the jars after %build. And if we (including Fedora) used an proper maven repository, then I suppose no one would do this as it would invalidate the current checksum maven files and so likely those would need to be regenerated by an rpm brp script.
So, my concern is that the jar is modified in the %install section. I guess for Fedora this may be safer, as I believe the %add_maven_depmap macro also installs the modified jar even though the modification may technically happen during the wrong build phase.
%add_to_maven_depmap and %update_maven_depmap macros were removed. These macros have been obsolete for long time. Their usage was preventing Fedora from implementing some features related to Maven metadata. Packages using these macros will have to be ported to use %add_maven_depmap or they will fail to build.
I am aware that Fedora has not been using the aggregate depmap file for a long time, but I am not sure what you men here. Can you elaborate?
I guess you have more manpower than I do. As I am just one person, I am looking at ways to produce various repos without having to rebuild every Java packages to pick up a new macro.
I think it would be better practice, at least, if a macro simply expanded to a call of a script. Then this code could be modified at any time. I am aware there's still a risk as old packages would have to be reinstalled.
I don't think that Fedora or RHEL has ever implemented the great functionality in Mandriva/Mageia's RPM file triggers (NB: I am not talking about %trigger).
- jpackage-utils didn't have update to new upstream version for 7 years
-- it was de-facto maintained by Fedora developers. Removing custom patchsets from our packages allows for better separation of upstream code from RPM packaging and simplifies overall maintenance.
I am not sure that this is accurate. There is a jpackage-utils 5.0.0 out there, although I am not sure it added any relevant features. Granted, there may not have been an official 5.0 release, but you make it sound as if no work had been done in the past seven years which is not the case.
On 07/21/2013 07:08 AM, David Walluck wrote:
On 07/18/2013 09:39 AM, Mikolaj Izdebski wrote:
%add_maven_depmap macro now injects pom.properties to every JAR file it processes. This will allow XMVn Subst to be used to replace any JAR
For some of our products, we may end up having jars in two (or more) separate locations where symlinking may not be possible. Because of this, I try to never modify the jars after %build. And if we (including Fedora) used an proper maven repository, then I suppose no one would do this as it would invalidate the current checksum maven files and so likely those would need to be regenerated by an rpm brp script.
JARs are modified only by files *not* built with Maven -- these files wouldn't have checksums generated. Even if we wanted to generate proper repository then checksums would have to be generated separately from Maven and that can easily be done after injecting Maven metadata.
So, my concern is that the jar is modified in the %install section. I guess for Fedora this may be safer, as I believe the %add_maven_depmap macro also installs the modified jar even though the modification may technically happen during the wrong build phase.
Files built with Maven already have all necessary metadata included and they are not repackaged. In this case metadata is added in `%build' section.
Besides that Java packaging never stricylt respected RPM build sections. Tests are often ran in %build instead of %check and some files are generated in %install and not %build (%add_to_maven_depmap and %jpackage_script are examples of file generation in %install). That theoretically happens wrong RPM build phases, but I think that changing this would only be a pointless effort.
%add_to_maven_depmap and %update_maven_depmap macros were removed. These macros have been obsolete for long time. Their usage was preventing Fedora from implementing some features related to Maven metadata. Packages using these macros will have to be ported to use %add_maven_depmap or they will fail to build.
I am aware that Fedora has not been using the aggregate depmap file for a long time, but I am not sure what you men here. Can you elaborate?
As I explained in my email, %add_maven_depmap now injects some additioinal metadata to JARs it processes. To have that metadata included in all system JARs known to Maven one would have either to modify %add_to_maven_depmap to inject that metadata too, or make sure that no package uses %add_to_maven_depmap.
Because %add_to_maven_depmap has been obsolete for quite long time and quite few packages still use it I decided to remove it so that during the next mass rebuild all packages would need to be migrated to %add_maven_depmap and could benefit from the new feature (the embedded pom.properties).
%update_maven_depmap was removed too as it was implemented to NOOP and it was usually used in conjunction with %add_to_maven_depmap -- both macro usages can be removed together in the same commit.
I guess you have more manpower than I do. As I am just one person, I am looking at ways to produce various repos without having to rebuild every Java packages to pick up a new macro.
I think it would be better practice, at least, if a macro simply expanded to a call of a script. Then this code could be modified at any time. I am aware there's still a risk as old packages would have to be reinstalled.
Unlike the original JPackage (%add_to_maven_depmap, %update_maven_depmap, %jpackage_script and others) *all* Fedora macros *do* expand to external scripts. %add_maven_depmap calls maven_depmap.py Python script (as processing XML in shell wouldn't be very nice). There are 20 other Fedora macros (14 %pom_* macros and 6 %mvn_* macros) which are simple wrappers for shell scripts.
- jpackage-utils didn't have update to new upstream version for 7 years
-- it was de-facto maintained by Fedora developers. Removing custom patchsets from our packages allows for better separation of upstream code from RPM packaging and simplifies overall maintenance.
I am not sure that this is accurate. There is a jpackage-utils 5.0.0 out there, although I am not sure it added any relevant features. Granted, there may not have been an official 5.0 release, but you make it sound as if no work had been done in the past seven years which is not the case.
This doesn't change the fact that there was no update for very long time and that Fedora maintained its own set of patches. A separate project (javapackages) has been created to develop Fedora-specific macros while keep jpackage-utils close to what JPackage uses in hope that our changes could be synchronized in future. But that didn't happen (yet).
If there is a willingness from JPackage side I am all for merging our efforts. Red Hat has licensed the javapackages project under the same terms as JPackage (BSD license) precisely because it would allow to merge our changes easily back to JPackage.
On 07/22/2013 12:40 AM, Mikolaj Izdebski wrote:
JARs are modified only by files *not* built with Maven -- these files wouldn't have checksums generated. Even if we wanted to generate proper
Right, of course. I'm sorry that I missed this. Otherwise, they should already contain the pom.xml and pom.properties files. I even use this fact myself in order to tell whether maven has been used to build the official jar on maven central or not.
Besides that Java packaging never stricylt respected RPM build sections. Tests are often ran in %build instead of %check and some files are
Actually, %check doesn't always work due to various technical assumptions, so I tend to purposely never use it.
generated in %install and not %build (%add_to_maven_depmap and %jpackage_script are examples of file generation in %install). That theoretically happens wrong RPM build phases, but I think that changing this would only be a pointless effort.
When we need to create a file that technically is not created by the build, we can create it directly in %{buildroot}, which has to be done in %install. Otherwise, I believe it is best to do it in %prep, but usually never %build.
Packagers don't always follow the best policies, and I have even made the mistake myself. But, now we are taking a lot of the manual work away from the packagers which should make it less likely to make these kind of packaging errors.
If there is a willingness from JPackage side I am all for merging our efforts. Red Hat has licensed the javapackages project under the same terms as JPackage (BSD license) precisely because it would allow to merge our changes easily back to JPackage.
Actually, I do agree on the level of inactivity at present. I am a bit philosophically apposed to some of the macros, but I am not sure there's much of a point to bring that up here. In short, modifying poms can lead to lazier packaging, e.g., the stripping out many dependencies instead of bothering to build them. My other concern is that it makes us move away from the RPM principle of using patches by instead performing a lot of trickery in %prep rather than fixing it at the source level.
On 07/22/2013 07:46 AM, David Walluck wrote:
On 07/22/2013 12:40 AM, Mikolaj Izdebski wrote:
If there is a willingness from JPackage side I am all for merging our efforts. Red Hat has licensed the javapackages project under the same terms as JPackage (BSD license) precisely because it would allow to merge our changes easily back to JPackage.
Actually, I do agree on the level of inactivity at present. I am a bit philosophically apposed to some of the macros, but I am not sure there's much of a point to bring that up here. In short, modifying poms can lead to lazier packaging, e.g., the stripping out many dependencies instead of bothering to build them. My other concern is that it makes us move away from the RPM principle of using patches by instead performing a lot of trickery in %prep rather than fixing it at the source level.
For sure plain patches have their advantages. They can be viewed more easily, are known to almost everybody and can be upstreamed easily.
The original design of macros for POM modification was a bit different from what's currently implemented. I was really annoyed by the necessity of rebasing patches with almost every upstream release so I embedded some instructions how the patch was generated in the patch itself. This allowed me to rebase patches by just re-running these steps against newer upstream version.
Finally I decided to go for the macro solution as it's present today as this reduces noise in git logs and it's more consistent with the principle that generated files should not be checked into repositories. It's sill possible (but a little non-obvious) to use the POM editor to produce patches which would be checked into git.
java-devel@lists.fedoraproject.org
-
David Walluck -
Mikolaj Izdebski -
Severin Gehwolf