I've started working on packaging scuttlebot for Fedora. I see that we now have a Fedora package for every nodejs module. This makes it easy to map directories in node_modules to package names - however, it means submitting hundreds of packages to ever get scuttlebot submitted.
I'm wondering if there is a better way. A node module typically corresponds to a .o file in a C library (with exceptions like libsodium). It is like having a separate package for every function in glibc. Suppose we did this:
1) a nodejs-stdlib that includes all the common modules (a list to be argued over at length :-) ). There is no penalty other than a small amount of disk space for unused modules - just like with a C library.
2) other multi-module systems are combined - usually including all modules with the same first word. For instance, all the pull stream modules begin with 'pull': pull-abortable, pull-box-stream, pull-cat, pull-cont, etc. This would become nodejs-pull, and include all the pull modules.
When the package name matches the first word of the module name, then determining the package is still easy. When that is not the case, as with the proposed nodejs-stdlib, then dnf can still search for npm(...)
On Fri, Aug 4, 2017 at 6:48 AM, Stuart D Gathman stuart@gathman.org wrote:
I've started working on packaging scuttlebot for Fedora. I see that we now have a Fedora package for every nodejs module. This makes it easy to map directories in node_modules to package names - however, it means submitting hundreds of packages to ever get scuttlebot submitted.
I'm wondering if there is a better way. A node module typically corresponds to a .o file in a C library (with exceptions like libsodium). It is like having a separate package for every function in glibc. Suppose we did this:
a nodejs-stdlib that includes all the common modules (a list to be argued over at length :-) ). There is no penalty other than a small amount of disk space for unused modules - just like with a C library.
other multi-module systems are combined - usually including all modules with the same first word. For instance, all the pull stream modules begin with 'pull': pull-abortable, pull-box-stream, pull-cat, pull-cont, etc. This would become nodejs-pull, and include all the pull modules.
When the package name matches the first word of the module name, then determining the package is still easy. When that is not the case, as with the proposed nodejs-stdlib, then dnf can still search for npm(...)
Although this looks tempting, it's only looking at half the problem. Versions Nodejs modules get updated all the time, at different rates for different packages. If you had just one package for many modules, it would be getting updated at an alarming rate.
Troy
On Fri, Aug 4, 2017 at 10:43 AM Troy Dawson tdawson@redhat.com wrote:
On Fri, Aug 4, 2017 at 6:48 AM, Stuart D Gathman stuart@gathman.org wrote:
I've started working on packaging scuttlebot for Fedora. I see that we
now have a Fedora package for every nodejs module. This makes it easy to map directories in node_modules to package names - however, it means submitting hundreds of packages to ever get scuttlebot submitted.
I'm wondering if there is a better way. A node module typically
corresponds to a .o file in a C library (with exceptions like libsodium). It is like having a separate package for every function in glibc. Suppose we did this:
- a nodejs-stdlib that includes all the common modules (a list to be
argued over at length :-) ). There is no penalty other than a small amount of disk space for unused modules - just like with a C library.
- other multi-module systems are combined - usually including all
modules with the same first word. For instance, all the pull stream modules begin with 'pull': pull-abortable, pull-box-stream, pull-cat, pull-cont, etc. This would become nodejs-pull, and include all the pull modules.
When the package name matches the first word of the module name, then
determining the package is still easy. When that is not the case, as with the proposed nodejs-stdlib, then dnf can still search for npm(...)
Although this looks tempting, it's only looking at half the problem. Versions Nodejs modules get updated all the time, at different rates for different packages. If you had just one package for many modules, it would be getting updated at an alarming rate.
I think the better approach is carefully-controlled bundling in Fedora. As of a couple years ago, it is now permissible to bundle software together in Fedora if it meets certain conditions: 1) If the dependency is already packaged in Fedora and this software is compatible with that version, then this software must link against the unbundled version. 2) If the dependency is not yet packaged in Fedora but is likely to be useful to large amounts of Fedora software, it is strongly encouraged that it be packaged separately. 3) Other dependencies MAY be carried internally by the package that needs them, but that package MUST include `Provides: bundled(npm(modulename)) = version`
Rule 3) is so that if there is a security vulnerability in npm(modulename), we can find any and all software that is required to be updated.
I haven't had any time to work on it, but I'd very much like to develop an automatic RPM dependency generator that will recurse down the node_modules directories, read their package.json files and automatically create those Provides. If anyone else wants to take a crack at doing that, it would be an immense help.
On 08/09/2017 08:06 AM, Stephen Gallagher wrote:
On Fri, Aug 4, 2017 at 10:43 AM Troy Dawson <tdawson@redhat.com mailto:tdawson@redhat.com> wrote:
Although this looks tempting, it's only looking at half the problem. Versions Nodejs modules get updated all the time, at different rates for different packages. If you had just one package for many modules, it would be getting updated at an alarming rate.
Point taken.
I think the better approach is carefully-controlled bundling in Fedora. As of a couple years ago, it is now permissible to bundle software together in Fedora if it meets certain conditions:
- If the dependency is already packaged in Fedora and this software
is compatible with that version, then this software must link against the unbundled version. 2) If the dependency is not yet packaged in Fedora but is likely to be useful to large amounts of Fedora software, it is strongly encouraged that it be packaged separately. 3) Other dependencies MAY be carried internally by the package that needs them, but that package MUST include `Provides: bundled(npm(modulename)) = version`
Rule 3) is so that if there is a security vulnerability in npm(modulename), we can find any and all software that is required to be updated.
I haven't had any time to work on it, but I'd very much like to develop an automatic RPM dependency generator that will recurse down the node_modules directories, read their package.json files and automatically create those Provides. If anyone else wants to take a crack at doing that, it would be an immense help.
That sounds workable. Thanks! There are 502 npm dependencies of scuttlebot, 196 of which are already in Fedora. I shudder to think of the Requires: list and the Provides: list!!
On Wed, Aug 9, 2017 at 8:06 AM Stephen Gallagher sgallagh@redhat.com wrote:
I haven't had any time to work on it, but I'd very much like to develop an automatic RPM dependency generator that will recurse down the node_modules directories, read their package.json files and automatically create those Provides. If anyone else wants to take a crack at doing that, it would be an immense help.
A big thank-you needs to go to Tomas Tomacek for https://pagure.io/nodejs-packaging/pull-request/2 which is now in effect on Fedora 28 (Rawhide) builds.
So now any Node.js package that is built in Fedora will automatically have `Provides: bundled(nodejs-foo)` added automatically.
I was wondering, could we use bundling and clean up the mess that is current nodejs stack in Fedora?
To be more precise, there are currently over 2000 packages and over 500 nodejs-* is available bugs, that nobody has time to fix and/or feels like fixing, because updating one module might/will break several others.
For the sake of maintainability, I'd like to leave as little packages as possible.
On Thu, Sep 14, 2017 at 6:45 PM, Stephen Gallagher sgallagh@redhat.com wrote:
On Wed, Aug 9, 2017 at 8:06 AM Stephen Gallagher sgallagh@redhat.com wrote:
I haven't had any time to work on it, but I'd very much like to develop an automatic RPM dependency generator that will recurse down the node_modules directories, read their package.json files and automatically create those Provides. If anyone else wants to take a crack at doing that, it would be an immense help.
A big thank-you needs to go to Tomas Tomacek for https://pagure.io/nodejs-packaging/pull-request/2 which is now in effect on Fedora 28 (Rawhide) builds.
So now any Node.js package that is built in Fedora will automatically have `Provides: bundled(nodejs-foo)` added automatically.
nodejs mailing list -- nodejs@lists.fedoraproject.org To unsubscribe send an email to nodejs-leave@lists.fedoraproject.org
That sounds like a Change Proposal for F28.
So your idea would be to mass-retire all but the most common or security-sensitive NPMs from Fedora and rely on bundling in most cases?
That will also mean coming up with a strategy for removing those unbundled packages on updates. Thoughts on that?
On Wed, Sep 20, 2017, 4:59 AM Zuzana Svetlikova zsvetlik@redhat.com wrote:
I was wondering, could we use bundling and clean up the mess that is current nodejs stack in Fedora?
To be more precise, there are currently over 2000 packages and over 500 nodejs-* is available bugs, that nobody has time to fix and/or feels like fixing, because updating one module might/will break several others.
For the sake of maintainability, I'd like to leave as little packages as possible.
On Thu, Sep 14, 2017 at 6:45 PM, Stephen Gallagher sgallagh@redhat.com wrote:
On Wed, Aug 9, 2017 at 8:06 AM Stephen Gallagher sgallagh@redhat.com wrote:
I haven't had any time to work on it, but I'd very much like to develop an automatic RPM dependency generator that will recurse down the node_modules directories, read their package.json files and automatically create those Provides. If anyone else wants to take a crack at doing that, it would be an immense help.
A big thank-you needs to go to Tomas Tomacek for https://pagure.io/nodejs-packaging/pull-request/2 which is now in effect on Fedora 28 (Rawhide) builds.
So now any Node.js package that is built in Fedora will automatically have `Provides: bundled(nodejs-foo)` added automatically.
nodejs mailing list -- nodejs@lists.fedoraproject.org To unsubscribe send an email to nodejs-leave@lists.fedoraproject.org
nodejs mailing list -- nodejs@lists.fedoraproject.org To unsubscribe send an email to nodejs-leave@lists.fedoraproject.org
Binary compiled ones probably also need to be kept.
Presumably this also means identifying "top level" packages that are an end goal in themselves and need to be kept.
Tom
On 20/09/17 11:11, Stephen Gallagher wrote:
That sounds like a Change Proposal for F28.
So your idea would be to mass-retire all but the most common or security-sensitive NPMs from Fedora and rely on bundling in most cases?
That will also mean coming up with a strategy for removing those unbundled packages on updates. Thoughts on that?
On Wed, Sep 20, 2017, 4:59 AM Zuzana Svetlikova <zsvetlik@redhat.com mailto:zsvetlik@redhat.com> wrote:
I was wondering, could we use bundling and clean up the mess that is current nodejs stack in Fedora? To be more precise, there are currently over 2000 packages and over 500 nodejs-* is available bugs, that nobody has time to fix and/or feels like fixing, because updating one module might/will break several others. For the sake of maintainability, I'd like to leave as little packages as possible. On Thu, Sep 14, 2017 at 6:45 PM, Stephen Gallagher <sgallagh@redhat.com <mailto:sgallagh@redhat.com>> wrote: On Wed, Aug 9, 2017 at 8:06 AM Stephen Gallagher <sgallagh@redhat.com <mailto:sgallagh@redhat.com>> wrote: I haven't had any time to work on it, but I'd very much like to develop an automatic RPM dependency generator that will recurse down the node_modules directories, read their package.json files and automatically create those Provides. If anyone else wants to take a crack at doing that, it would be an immense help. A big thank-you needs to go to Tomas Tomacek for https://pagure.io/nodejs-packaging/pull-request/2 which is now in effect on Fedora 28 (Rawhide) builds. So now any Node.js package that is built in Fedora will automatically have `Provides: bundled(nodejs-foo)` added automatically. _______________________________________________ nodejs mailing list -- nodejs@lists.fedoraproject.org <mailto:nodejs@lists.fedoraproject.org> To unsubscribe send an email to nodejs-leave@lists.fedoraproject.org <mailto:nodejs-leave@lists.fedoraproject.org> _______________________________________________ nodejs mailing list -- nodejs@lists.fedoraproject.org <mailto:nodejs@lists.fedoraproject.org> To unsubscribe send an email to nodejs-leave@lists.fedoraproject.org <mailto:nodejs-leave@lists.fedoraproject.org>
nodejs mailing list -- nodejs@lists.fedoraproject.org To unsubscribe send an email to nodejs-leave@lists.fedoraproject.org
nodejs@lists.fedoraproject.org
-
Stephen Gallagher -
Stuart D Gathman -
Stuart Gathman
-
Tom Hughes -
Troy Dawson -
Zuzana Svetlikova