Hello all,
I hope someone can help me on this one. Probably I am doing something wrong, but I cannot figure out what I do wrong:
I am building a customised CD based on FC6 RPMs+custom RPMs.
The problem I see is that the directory os/<prodname> contains all RPMS including all dependencies, but the directory os-disc1/<prodname> only contain a strange subset of the RPMs. I have 10 custom RPMs, but the os-disc1 directory contains only 31, and I expect a total of 330 RPMs. Notice that the entire RPM-collection in the os dir is about 340 MB, which properly fits on 1 CD. After the build only 1 iso is created of about 150 MB.
So, calculating dependencies based on my comps.xml works well, but splitting the tree into CD's ignores a lot of RPMs.
Does anyone here has an idea of what I might have forgotten to configure, or which error I make?
Note: even the glibc rpms are not there in the CD-image... Therefor anaconda fails during installation (extraction phase of the RPMs), as it cannot find some localisation files.
Kind Regards,
Remy Bohmer
On Wednesday 31 January 2007 06:40, Remy Bohmer wrote:
Hello all,
I hope someone can help me on this one. Probably I am doing something wrong, but I cannot figure out what I do wrong:
I am building a customised CD based on FC6 RPMs+custom RPMs.
The problem I see is that the directory os/<prodname> contains all RPMS including all dependencies, but the directory os-disc1/<prodname> only contain a strange subset of the RPMs. I have 10 custom RPMs, but the os-disc1 directory contains only 31, and I expect a total of 330 RPMs. Notice that the entire RPM-collection in the os dir is about 340 MB, which properly fits on 1 CD. After the build only 1 iso is created of about 150 MB.
How many discs are you asking for in your config file? Also, what groups are they in within comps? splittree does ordering based on comps groups, could have something to do with this.
Hello Jesse,
In my config file there is only 1 disc. In my comps file there are 4 groups, namely pungi, Core (=copied from FC6 Comps), and 2 custom groups. See attached my configuration files.
Kind Regards,
Remy
2007/1/31, Jesse Keating jkeating@redhat.com:
On Wednesday 31 January 2007 06:40, Remy Bohmer wrote:
Hello all,
I hope someone can help me on this one. Probably I am doing something wrong, but I cannot figure out what I do wrong:
I am building a customised CD based on FC6 RPMs+custom RPMs.
The problem I see is that the directory os/<prodname> contains all RPMS including all dependencies, but the directory os-disc1/<prodname> only contain a strange subset of the RPMs. I have 10 custom RPMs, but the os-disc1 directory contains only 31, and I expect a total of 330 RPMs. Notice that the entire RPM-collection in the os dir is about 340 MB, which properly fits on 1 CD. After the build only 1 iso is created of about 150 MB.
How many discs are you asking for in your config file? Also, what groups are they in within comps? splittree does ordering based on comps groups, could have something to do with this.
-- Jesse Keating Release Engineer: Fedora
-- Fedora-buildsys-list mailing list Fedora-buildsys-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
On Wednesday 31 January 2007 08:42, Remy Bohmer wrote:
Hello Jesse,
In my config file there is only 1 disc. In my comps file there are 4 groups, namely pungi, Core (=copied from FC6 Comps), and 2 custom groups. See attached my configuration files.
Just for snorts, name one of your groups "base". That and see what packageorder does on your tree. pypungi/pungi.py has the pkgorder command used.
Hello Jesse,
I named it Base, but unfortunately it makes no difference. Is it case sensitive? 'base' versus 'Base' ?
What did you mean with the line: 'That and see what packageorder does on your tree. pypungi/pungi.py has the pkgorder command used.' ? Should I run some command and check its output? I rebuilt everything with pung, and checked its output for now...
Do you have any other ideas?
Kind Regards,
Remy Bohmer
2007/1/31, Jesse Keating jkeating@redhat.com:
On Wednesday 31 January 2007 08:42, Remy Bohmer wrote:
Hello Jesse,
In my config file there is only 1 disc. In my comps file there are 4 groups, namely pungi, Core (=copied from FC6 Comps), and 2 custom groups. See attached my configuration files.
Just for snorts, name one of your groups "base". That and see what packageorder does on your tree. pypungi/pungi.py has the pkgorder command used.
-- Jesse Keating Release Engineer: Fedora
-- Fedora-buildsys-list mailing list Fedora-buildsys-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
On Wednesday 31 January 2007 11:58, Remy Bohmer wrote:
What did you mean with the line: 'That and see what packageorder does on your tree. pypungi/pungi.py has the pkgorder command used.' ? Should I run some command and check its output? I rebuilt everything with pung, and checked its output for now...
Use:
/usr/lib/anaconda-runtime/pkgorder /path/to/your/os <arch> Fedora
That will spit out a package order that is used by splittree to put your packages on various CDs. I'd be interested to see if all the packages in your compose make it.
pkgorder is a python script, so you can use python debugger (pdb) to see whats going on.
pkgorder is also part of anaconda-runtime, so any bugs found belong to the anaconda group, not pungi (:
Hello Jesse,
Thanks for your excellent help on this, I will try this pkgorder suggestion immediately tomorrow morning... It is evening here in the Netherlands, and I do not have the CD-build stuff at home... BTW: Do you prefer contact through this mailinglist, or IRC ?
Kind Regards,
Remy Bohmer
2007/1/31, Jesse Keating jkeating@redhat.com:
On Wednesday 31 January 2007 11:58, Remy Bohmer wrote:
What did you mean with the line: 'That and see what packageorder does on your tree. pypungi/pungi.py has the pkgorder command used.' ? Should I run some command and check its output? I rebuilt everything with pung, and checked its output for now...
Use:
/usr/lib/anaconda-runtime/pkgorder /path/to/your/os <arch> Fedora
That will spit out a package order that is used by splittree to put your packages on various CDs. I'd be interested to see if all the packages in your compose make it.
pkgorder is a python script, so you can use python debugger (pdb) to see whats going on.
pkgorder is also part of anaconda-runtime, so any bugs found belong to the anaconda group, not pungi (:
-- Jesse Keating Release Engineer: Fedora
-- Fedora-buildsys-list mailing list Fedora-buildsys-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
On Wednesday 31 January 2007 14:47, Remy Bohmer wrote:
Thanks for your excellent help on this, I will try this pkgorder suggestion immediately tomorrow morning... It is evening here in the Netherlands, and I do not have the CD-build stuff at home... BTW: Do you prefer contact through this mailinglist, or IRC ?
Given our timezone differences, mail is probably best (:
Hello Jesse,
Attached are the results of this command: /usr/lib/anaconda-runtime/pkgorder /home/me/cdbuildtree/results/myrelease/i386/os i386 myproduct
I replaced 'myproduct' with 'Fedora'. No differences. The produced list is exactly like the list of RPM's in my os-disc1- directory. Thus, much too short...
Does this give you some new ideas?
Kind Regards,
Remy Bohmer
2007/1/31, Jesse Keating jkeating@redhat.com:
On Wednesday 31 January 2007 11:58, Remy Bohmer wrote:
What did you mean with the line: 'That and see what packageorder does on your tree. pypungi/pungi.py has the pkgorder command used.' ? Should I run some command and check its output? I rebuilt everything with pung, and checked its output for now...
Use:
/usr/lib/anaconda-runtime/pkgorder /path/to/your/os <arch> Fedora
That will spit out a package order that is used by splittree to put your packages on various CDs. I'd be interested to see if all the packages in your compose make it.
pkgorder is a python script, so you can use python debugger (pdb) to see whats going on.
pkgorder is also part of anaconda-runtime, so any bugs found belong to the anaconda group, not pungi (:
-- Jesse Keating Release Engineer: Fedora
-- Fedora-buildsys-list mailing list Fedora-buildsys-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
On Thursday 01 February 2007 05:40, Remy Bohmer wrote:
Attached are the results of this command: /usr/lib/anaconda-runtime/pkgorder /home/me/cdbuildtree/results/myrelease/i386/os i386 myproduct
I replaced 'myproduct' with 'Fedora'. No differences. The produced list is exactly like the list of RPM's in my os-disc1- directory. Thus, much too short...
Does this give you some new ideas?
It would appear that pkgorder doesn't know about the other groups you have in your comps file perhaps. pkgorder is just a python script, I'd poke at it and see how it figures out the order, and see if perhaps you need to change your group names in comps.
Hi,
On 2/1/07, Jesse Keating jkeating@redhat.com wrote:
It would appear that pkgorder doesn't know about the other groups you have in your comps file perhaps. pkgorder is just a python script, I'd poke at it and see how it figures out the order, and see if perhaps you need to change your group names in comps.
After looking at pkgorder it seems it has a hardcoded list of groups it adds for resolving the order.
Maybe a 'fallback' to add any remaining groups that were not selected yet would be a good improvement of pkgorder? (and a solution to this problem).
I'll see if I can come-up with a patch to pkgorder that does this.
Best regards,
Jeroen Janssen
Hello All,
I renamed all my groups to 'core', 'base', 'base-x' and 'development-tools', and it does not make any difference... So, I believe it is not only related to the group naming in the comps file... I am going to debug this some more today.
If anyone has a good idea about this, please let me know.
Kind Regards,
Remy
2007/2/1, Jesse Keating jkeating@redhat.com:
On Thursday 01 February 2007 05:40, Remy Bohmer wrote:
Attached are the results of this command: /usr/lib/anaconda-runtime/pkgorder /home/me/cdbuildtree/results/myrelease/i386/os i386 myproduct
I replaced 'myproduct' with 'Fedora'. No differences. The produced list is exactly like the list of RPM's in my os-disc1- directory. Thus, much too short...
Does this give you some new ideas?
It would appear that pkgorder doesn't know about the other groups you have in your comps file perhaps. pkgorder is just a python script, I'd poke at it and see how it figures out the order, and see if perhaps you need to change your group names in comps.
-- Jesse Keating Release Engineer: Fedora
Hello All,
I got it! but, it is very strange... ;-))
I debugged the pkgorder script and found the following: * At the routine 'processTransaction()' the complete lists of dependencies is available, then it calls the routine printMatchingPkgs() to print the packages to standard, after it has checked that they are available. * The filemask that is passed to printMatchingPkgs() through 'fpattern', is build up as follows: <os-dir>/Product/package*. Where Product starts with a capital, but my product_name in the pungi configuration does NOT star with a capital. So, the product-dir in the os-dir does not start with a capital. So, no dependency files can be found, leaving a os-disc1 with just a few files in it...
I searched through the entire tree several times, but nowhere I have listed the productname with a capital, so this capital must be introduced by the tooling itself ! (Fedora also starts with a capital... Redhat also, and probably other distros using anaconda also?)
So, to be able to build a customised CD, you must use a product_name in Pungi that starts with a capital. I have changed this in my build environment and now it works!
I believe the problem is introduced by collecting the dependencies in the routine addGroups() -> ds.ResolveDeps() in the pkgorder script.
For me it is going to deep in the external tooling, which I do not fully understand yet how they work. Maybe someone else recognises this phenomenon and can pick it up from here? Or if someone can give me some tips on this, I can look further into it...
Kind Regards,
Remy Bohmer
2007/2/2, Remy Bohmer mythtv.bohmer@gmail.com:
Hello All,
I renamed all my groups to 'core', 'base', 'base-x' and 'development-tools', and it does not make any difference... So, I believe it is not only related to the group naming in the comps file... I am going to debug this some more today.
If anyone has a good idea about this, please let me know.
Kind Regards,
Remy
2007/2/1, Jesse Keating jkeating@redhat.com:
On Thursday 01 February 2007 05:40, Remy Bohmer wrote:
Attached are the results of this command: /usr/lib/anaconda-runtime/pkgorder /home/me/cdbuildtree/results/myrelease/i386/os i386 myproduct
I replaced 'myproduct' with 'Fedora'. No differences. The produced list is exactly like the list of RPM's in my os-disc1- directory. Thus, much too short...
Does this give you some new ideas?
It would appear that pkgorder doesn't know about the other groups you have in your comps file perhaps. pkgorder is just a python script, I'd poke at it and see how it figures out the order, and see if perhaps you need to change your group names in comps.
-- Jesse Keating Release Engineer: Fedora
Hello All,
I thought I got it... But this is not true... I have now the problem reversed :-(((
By changing the first letter of the product name to a capital, the result was that I have now a CD image of 331 packages, where I need 362. Before this change, I had 31 packages...
The problem is still pkgorder, at the same location as mentioned below.
For some reason pkgorder is really stubborn, and still mixes up the productname by changing the first letter of several packages. It does not matter where the packages come from, either from the orignal FC6-DVD , or from the FC6-Updates repo, or from my custom repo. Even if I build without my own repo, does not matter. Notice that a bug in pkgorder does not even tell anybody if it can find a package at all, it just continuous, resulting in an incomplete CD-image...
So, does anyone know how the location of a package is determined in pkgorder? (I mean the relative path inside a repo, with the product name)
It does NOT read it from Pungi configuration file (Tried to change it completely, no result) Tried to change the product name on the commandline of pkgorder -> no result either...
So, nowhere in my tree I can find the wrong productname, and neither does any change of any product_name setting help.
I really hope, someone can help me out here... I am starting to get quite desperate about these tools... :-((
Remy
2007/2/2, Remy Bohmer mythtv.bohmer@gmail.com:
Hello All,
I got it! but, it is very strange... ;-))
I debugged the pkgorder script and found the following:
- At the routine 'processTransaction()' the complete lists of
dependencies is available, then it calls the routine printMatchingPkgs() to print the packages to standard, after it has checked that they are available.
- The filemask that is passed to printMatchingPkgs() through
'fpattern', is build up as follows: <os-dir>/Product/package*. Where Product starts with a capital, but my product_name in the pungi configuration does NOT star with a capital. So, the product-dir in the os-dir does not start with a capital. So, no dependency files can be found, leaving a os-disc1 with just a few files in it...
I searched through the entire tree several times, but nowhere I have listed the productname with a capital, so this capital must be introduced by the tooling itself ! (Fedora also starts with a capital... Redhat also, and probably other distros using anaconda also?)
So, to be able to build a customised CD, you must use a product_name in Pungi that starts with a capital. I have changed this in my build environment and now it works!
I believe the problem is introduced by collecting the dependencies in the routine addGroups() -> ds.ResolveDeps() in the pkgorder script.
For me it is going to deep in the external tooling, which I do not fully understand yet how they work. Maybe someone else recognises this phenomenon and can pick it up from here? Or if someone can give me some tips on this, I can look further into it...
Kind Regards,
Remy Bohmer
2007/2/2, Remy Bohmer mythtv.bohmer@gmail.com:
Hello All,
I renamed all my groups to 'core', 'base', 'base-x' and 'development-tools', and it does not make any difference... So, I believe it is not only related to the group naming in the comps file... I am going to debug this some more today.
If anyone has a good idea about this, please let me know.
Kind Regards,
Remy
2007/2/1, Jesse Keating jkeating@redhat.com:
On Thursday 01 February 2007 05:40, Remy Bohmer wrote:
Attached are the results of this command: /usr/lib/anaconda-runtime/pkgorder /home/me/cdbuildtree/results/myrelease/i386/os i386 myproduct
I replaced 'myproduct' with 'Fedora'. No differences. The produced list is exactly like the list of RPM's in my os-disc1- directory. Thus, much too short...
Does this give you some new ideas?
It would appear that pkgorder doesn't know about the other groups you have in your comps file perhaps. pkgorder is just a python script, I'd poke at it and see how it figures out the order, and see if perhaps you need to change your group names in comps.
-- Jesse Keating Release Engineer: Fedora
On Monday 05 February 2007 11:53, Remy Bohmer wrote:
For some reason pkgorder is really stubborn, and still mixes up the productname by changing the first letter of several packages. It does not matter where the packages come from, either from the orignal FC6-DVD , or from the FC6-Updates repo, or from my custom repo. Even if I build without my own repo, does not matter. Notice that a bug in pkgorder does not even tell anybody if it can find a package at all, it just continuous, resulting in an incomplete CD-image...
So, does anyone know how the location of a package is determined in pkgorder? (I mean the relative path inside a repo, with the product name)
It does NOT read it from Pungi configuration file (Tried to change it completely, no result) Tried to change the product name on the commandline of pkgorder -> no result either...
So, nowhere in my tree I can find the wrong productname, and neither does any change of any product_name setting help.
I really hope, someone can help me out here... I am starting to get quite desperate about these tools... :-((
Er, Are you putting packages into different directories under os/ ? I'm pretty sure that the way pungi is designed, all your packages would (by default) go under os/Fedora/ ALL packages, regardless of what repository they come from. Are you getting multiple product dirs somehow? I'm seriously having a hard time figuring out how you're getting to this stage.
Hello Jesse,
What you mention is exactly what I want, except my product is not named Fedora, but something else... I will summarise the facts here below, hope this clears something up.
Are you putting packages into different directories under os/ ?
No, the /os/ is an intermediate buildresult of what pungi (with anaconda) is doing. I am not modifying it in any way. My goal is to use as many standard tooling as possible, without the need for me to modify the tools. (Except if there is a need for a bugfix)
So, only 1 directory with packages is there in /os/myproduct, not 2.
But after building the /os directory pungi calls at some point pkgorder and this one introduces a problem while building the os-disc1/myproduct directory. I can reproduce it easily while just running pkgorder on the /os tree that is generated by pungi, as you earlier suggested.
During build and debugging pkgorder, I have seen that processTransaction() searches for the relative path of a package. This path is mostly called os/Myproduct (Uppercase-M) instead of os/myproduct. (Lowercase-M) os/Myproduct (Uppercase-M) does not exist, so the package cannot be found. (the lacking 631 packages) Some packages are searched in os/myproduct (lowercase-M), and these are found correctly. (the 31 I had) Due to a bug in pkgorder this 'file-not-found' issue is not visible to the outside world. I have a patch for this, so that this it is at least visible that this happens.
As a test, I have renamed the product_name to Myproduct (Uppercase-M). If I do that I end up with 631 packages in /os-disc1, lacking the 31 I had while using the lowercase M.
The packages that are searched for, in the wrong directory, can come from any repo, i.e. fc6-dvd, fc6-updates, or my own custom repo, but always the same packages go wrong.
Are you getting multiple product dirs somehow?
I am thus NOT getting multiple product dirs in /os-disc1. It is not a destination (/os-disc1) problem but a package gathering problem from /os.
Do you have any idea how the relative-paths are resolved from the /os repository? I am certain that the cause is there somewhere, but I already spent 2 days debugging pkgorder with pdb, and it does not bring me any closer the root cause :-(( I do not understand the relation to yum/package-dictionaries etc. Notice that I have verified that dependency resolving works fine...
I really hope you can help me on this...
Kind Regards,
Remy
2007/2/5, Jesse Keating jkeating@redhat.com:
On Monday 05 February 2007 11:53, Remy Bohmer wrote:
For some reason pkgorder is really stubborn, and still mixes up the productname by changing the first letter of several packages. It does not matter where the packages come from, either from the orignal FC6-DVD , or from the FC6-Updates repo, or from my custom repo. Even if I build without my own repo, does not matter. Notice that a bug in pkgorder does not even tell anybody if it can find a package at all, it just continuous, resulting in an incomplete CD-image...
So, does anyone know how the location of a package is determined in pkgorder? (I mean the relative path inside a repo, with the product name)
It does NOT read it from Pungi configuration file (Tried to change it completely, no result) Tried to change the product name on the commandline of pkgorder -> no result either...
So, nowhere in my tree I can find the wrong productname, and neither does any change of any product_name setting help.
I really hope, someone can help me out here... I am starting to get quite desperate about these tools... :-((
Er, Are you putting packages into different directories under os/ ? I'm pretty sure that the way pungi is designed, all your packages would (by default) go under os/Fedora/ ALL packages, regardless of what repository they come from. Are you getting multiple product dirs somehow? I'm seriously having a hard time figuring out how you're getting to this stage.
-- Jesse Keating Release Engineer: Fedora
Hello All,
To finish this thread: Due to this bug, I believe that Pungi/Anaconda is not ready yet for building a really custom installation CD, derived from Fedora 6 and custom repositories. It took me a lot of time and finally I had to give it up.
I want to thank the people that helped me so far for their efforts. Unfortunately I am not going to be using it any further.
Kind Regards,
Remy Bohmer
2007/2/5, Remy Bohmer mythtv.bohmer@gmail.com:
Hello Jesse,
What you mention is exactly what I want, except my product is not named Fedora, but something else... I will summarise the facts here below, hope this clears something up.
Are you putting packages into different directories under os/ ?
No, the /os/ is an intermediate buildresult of what pungi (with anaconda) is doing. I am not modifying it in any way. My goal is to use as many standard tooling as possible, without the need for me to modify the tools. (Except if there is a need for a bugfix)
So, only 1 directory with packages is there in /os/myproduct, not 2.
But after building the /os directory pungi calls at some point pkgorder and this one introduces a problem while building the os-disc1/myproduct directory. I can reproduce it easily while just running pkgorder on the /os tree that is generated by pungi, as you earlier suggested.
During build and debugging pkgorder, I have seen that processTransaction() searches for the relative path of a package. This path is mostly called os/Myproduct (Uppercase-M) instead of os/myproduct. (Lowercase-M) os/Myproduct (Uppercase-M) does not exist, so the package cannot be found. (the lacking 631 packages) Some packages are searched in os/myproduct (lowercase-M), and these are found correctly. (the 31 I had) Due to a bug in pkgorder this 'file-not-found' issue is not visible to the outside world. I have a patch for this, so that this it is at least visible that this happens.
As a test, I have renamed the product_name to Myproduct (Uppercase-M). If I do that I end up with 631 packages in /os-disc1, lacking the 31 I had while using the lowercase M.
The packages that are searched for, in the wrong directory, can come from any repo, i.e. fc6-dvd, fc6-updates, or my own custom repo, but always the same packages go wrong.
Are you getting multiple product dirs somehow?
I am thus NOT getting multiple product dirs in /os-disc1. It is not a destination (/os-disc1) problem but a package gathering problem from /os.
Do you have any idea how the relative-paths are resolved from the /os repository? I am certain that the cause is there somewhere, but I already spent 2 days debugging pkgorder with pdb, and it does not bring me any closer the root cause :-(( I do not understand the relation to yum/package-dictionaries etc. Notice that I have verified that dependency resolving works fine...
I really hope you can help me on this...
Kind Regards,
Remy
2007/2/5, Jesse Keating jkeating@redhat.com:
On Monday 05 February 2007 11:53, Remy Bohmer wrote:
For some reason pkgorder is really stubborn, and still mixes up the productname by changing the first letter of several packages. It does not matter where the packages come from, either from the orignal FC6-DVD , or from the FC6-Updates repo, or from my custom repo. Even if I build without my own repo, does not matter. Notice that a bug in pkgorder does not even tell anybody if it can find a package at all, it just continuous, resulting in an incomplete CD-image...
So, does anyone know how the location of a package is determined in pkgorder? (I mean the relative path inside a repo, with the product name)
It does NOT read it from Pungi configuration file (Tried to change it completely, no result) Tried to change the product name on the commandline of pkgorder -> no result either...
So, nowhere in my tree I can find the wrong productname, and neither does any change of any product_name setting help.
I really hope, someone can help me out here... I am starting to get quite desperate about these tools... :-((
Er, Are you putting packages into different directories under os/ ? I'm pretty sure that the way pungi is designed, all your packages would (by default) go under os/Fedora/ ALL packages, regardless of what repository they come from. Are you getting multiple product dirs somehow? I'm seriously having a hard time figuring out how you're getting to this stage.
-- Jesse Keating Release Engineer: Fedora
buildsys@lists.fedoraproject.org
-
Jeroen Janssen -
Jesse Keating -
Remy Bohmer