I took a look at pungi and decided that with the state it is in, and my lack of deep understanding of it, I don't want to tackle converting it to DNF. So, in order to support the ability to independently move lorax to DNF and Python3 I've converted the library call to run lorax as a process.
I'm still undecided about moving lorax to DNF for F22 or not. I'll probably make a decision on that after tomorrow's Anaconda DNF test day.
I *am* going to make the switch for rawhide on Friday though, so these changes will need to be in the rawhide version of pungi soon.
I've done multiple test runs and things look good, at least on x86_64. The only issues I have right now are that the boot.iso is about 200M larger than the F21 one, and DNF doesn't log transaction decisions yet so it's hard to track down what's getting pulled in by what.
Brian C. Lane (3): Close child fds when using subprocess Call lorax as a process not a library Add the option to pass a custom path for the multilib config files
src/bin/pungi.py | 3 ++ src/pypungi/__init__.py | 83 +++++++++++++++++++++++++++---------------------- src/pypungi/config.py | 1 + src/pypungi/multilib.py | 52 +++++++++++++++++++++---------- src/pypungi/util.py | 5 +-- 5 files changed, 88 insertions(+), 56 deletions(-)
--- src/pypungi/util.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/pypungi/util.py b/src/pypungi/util.py index 0a2ea11..800cf7f 100644 --- a/src/pypungi/util.py +++ b/src/pypungi/util.py @@ -24,7 +24,8 @@ def _doRunCommand(command, logger, rundir='/tmp', output=subprocess.PIPE, error=
logger.info("Running %s" % subprocess.list2cmdline(command))
- p1 = subprocess.Popen(command, cwd=rundir, stdout=output, stderr=error, universal_newlines=True, env=env) + p1 = subprocess.Popen(command, cwd=rundir, stdout=output, stderr=error, universal_newlines=True, env=env, + close_fds=True) (out, err) = p1.communicate()
if out:
Doing this allows lorax to move to DNF (and Python3) without needing to wait for pungi to be updated. --- src/pypungi/__init__.py | 82 ++++++++++++++++++++++++++----------------------- src/pypungi/util.py | 2 +- 2 files changed, 45 insertions(+), 39 deletions(-)
diff --git a/src/pypungi/__init__.py b/src/pypungi/__init__.py index c1318ce..0e4b316 100644 --- a/src/pypungi/__init__.py +++ b/src/pypungi/__init__.py @@ -20,7 +20,6 @@ import os import re import shutil import sys -import gzip import pypungi.util import pprint import lockfile @@ -29,7 +28,6 @@ import urlgrabber.progress import subprocess import createrepo import ConfigParser -import pylorax from fnmatch import fnmatch
import arch as arch_module @@ -1371,59 +1369,67 @@ class Pungi(pypungi.PungiBase): def doBuildinstall(self): """Run lorax on the tree."""
- # the old ayum object has transaction data that confuse lorax, reinit. - self._inityum() + cmd = ["lorax"] + cmd.extend(["--workdir", self.workdir]) + cmd.extend(["--logfile", os.path.join(self.config.get('pungi', 'destdir'), 'logs/lorax.log')])
- # Add the repo in the destdir to our yum object - self._add_yum_repo('ourtree', - 'file://%s' % self.topdir, - cost=10) - - product = self.config.get('pungi', 'name') - version = self.config.get('pungi', 'version') - release = '%s %s' % (self.config.get('pungi', 'name'), self.config.get('pungi', 'version')) + try: + # Convert url method to a repo + self.ksparser.handler.repo.methodToRepo() + except: + pass
- variant = self.config.get('pungi', 'flavor') - bugurl = self.config.get('pungi', 'bugurl') - isfinal = self.config.get('pungi', 'isfinal') + for repo in self.ksparser.handler.repo.repoList: + if repo.mirrorlist: + # The not bool() thing is because pykickstart is yes/no on + # whether to ignore groups, but yum is a yes/no on whether to + # include groups. Awkward. + cmd.extend(["--mirrorlist", repo.mirrorlist]) + else: + cmd.extend(["--source", repo.baseurl])
- volid = self._shortenVolID() - workdir = self.workdir - outputdir = self.topdir + # Add the repo in the destdir to our yum object + cmd.extend(["--source", "file://%s" % self.topdir]) + cmd.extend(["--product", self.config.get('pungi', 'name')]) + cmd.extend(["--version", self.config.get('pungi', 'version')]) + cmd.extend(["--release", "%s %s" % (self.config.get('pungi', 'name'), self.config.get('pungi', 'version'))]) + if self.config.get('pungi', 'flavor'): + cmd.extend(["--variant", self.config.get('pungi', 'flavor')]) + cmd.extend(["--bugurl", self.config.get('pungi', 'bugurl')]) + if self.config.get('pungi', 'isfinal'): + cmd.append("--isfinal") + cmd.extend(["--volid", self._shortenVolID()])
# on ppc64 we need to tell lorax to only use ppc64 packages so that the media will run on all 64 bit ppc boxes if self.tree_arch == 'ppc64': - self.ayum.arch.setup_arch('ppc64') - self.ayum.compatarch = 'ppc64' + cmd.extend(["--buildarch", "ppc64"]) elif self.tree_arch == 'ppc64le': - self.ayum.arch.setup_arch('ppc64le') - self.ayum.compatarch = 'ppc64le' + cmd.extend(["--buildarch", "ppc64le"])
# Only supported mac hardware is x86 make sure we only enable mac support on arches that need it - if self.tree_arch in ['x86_64']: - if self.config.getboolean('pungi','nomacboot'): - domacboot = False - else: - domacboot = True + if self.tree_arch in ['x86_64'] and not self.config.getboolean('pungi','nomacboot'): + cmd.append("--macboot") else: - domacboot = False + cmd.append("--nomacboot")
- # run the command - lorax = pylorax.Lorax() try: - conf_file = self.config.get('lorax', 'conf_file') - lorax.configure(conf_file=conf_file) + cmd.extend(["--conf", self.config.get('lorax', 'conf_file')]) except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): - lorax.configure() + pass
try: - installpkgs = self.config.get('lorax', 'installpkgs').split(" ") + cmd.extend(["--installpkgs", self.config.get('lorax', 'installpkgs')]) except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): - installpkgs = None + pass + + # Allow the output directory to exist. + cmd.append("--force") + + # MUST be last in the list + cmd.append(self.topdir)
- lorax.run(self.ayum, product=product, version=version, release=release, - variant=variant, bugurl=bugurl, isfinal=isfinal, domacboot=domacboot, - workdir=workdir, outputdir=outputdir, volid=volid, installpkgs=installpkgs) + self.logger.info(" ".join(cmd)) + pypungi.util._doRunCommand(cmd, self.logger)
# write out the tree data for snake self.writeinfo('tree: %s' % self.mkrelative(self.topdir)) diff --git a/src/pypungi/util.py b/src/pypungi/util.py index 800cf7f..a60b78d 100644 --- a/src/pypungi/util.py +++ b/src/pypungi/util.py @@ -34,7 +34,7 @@ def _doRunCommand(command, logger, rundir='/tmp', output=subprocess.PIPE, error= if p1.returncode != 0: logger.error("Got an error from %s" % command[0]) logger.error(err) - raise OSError, "Got an error from %s: %s" % (command[0], err) + raise OSError, "Got an error (%d) from %s: %s" % (p1.returncode, command[0], err)
def _link(local, target, logger, force=False): """Simple function to link or copy a package, removing target optionally."""
The default is /usr/share/pungi/multilib/, pass --multilibconf to override this.
This also adds multilib.init() so that an import of multilib doesn't immediately setup the classes.
(cherry picked from commit 234524296fd53871aed64690cf6a7d5849ca154a) --- src/bin/pungi.py | 3 +++ src/pypungi/__init__.py | 1 + src/pypungi/config.py | 1 + src/pypungi/multilib.py | 52 ++++++++++++++++++++++++++++++++++--------------- 4 files changed, 41 insertions(+), 16 deletions(-)
diff --git a/src/bin/pungi.py b/src/bin/pungi.py index ca7ced8..70e6efd 100755 --- a/src/bin/pungi.py +++ b/src/bin/pungi.py @@ -272,6 +272,9 @@ if __name__ == '__main__': parser.add_option("-i", "--installpkgs", default=[], action="append", metavar="STRING", help="Package glob for lorax to install before runtime-install.tmpl runs. (may be listed multiple times)") + parser.add_option("--multilibconf", default=None, type="string", + action="callback", callback=set_config, callback_args=(config, ), + help="Path to multilib conf files. Default is /usr/share/pungi/multilib/")
parser.add_option("-c", "--config", dest="config", help='Path to kickstart config file') diff --git a/src/pypungi/__init__.py b/src/pypungi/__init__.py index 0e4b316..3240f59 100644 --- a/src/pypungi/__init__.py +++ b/src/pypungi/__init__.py @@ -106,6 +106,7 @@ class PungiBase(object):
def __init__(self, config): self.config = config + multilib.init(self.config.get('pungi', 'multilibconf'))
# ARCH setup self.tree_arch = self.config.get('pungi', 'arch') diff --git a/src/pypungi/config.py b/src/pypungi/config.py index 617ae18..46b9ed8 100644 --- a/src/pypungi/config.py +++ b/src/pypungi/config.py @@ -29,6 +29,7 @@ class Config(SafeConfigParser): self.set('pungi', 'sourcedir', 'source') self.set('pungi', 'debugdir', 'debug') self.set('pungi', 'isodir', 'iso') + self.set('pungi', 'multilibconf', '/usr/share/pungi/multilib/') self.set('pungi', 'relnotefilere', 'GPL README-BURNING-ISOS-en_US.txt ^RPM-GPG') self.set('pungi', 'relnotedirre', '') self.set('pungi', 'relnotepkgs', 'fedora-release fedora-release-notes') diff --git a/src/pypungi/multilib.py b/src/pypungi/multilib.py index 6719573..153d32a 100755 --- a/src/pypungi/multilib.py +++ b/src/pypungi/multilib.py @@ -92,6 +92,9 @@ class MultilibMethodBase(object): """a base class for multilib methods""" name = "base"
+ def __init__(self, config_path): + self.config_path = config_path + def select(self, po): raise NotImplementedError
@@ -141,10 +144,11 @@ class RuntimeMultilibMethod(MultilibMethodBase): """pre-defined paths to libs""" name = "runtime"
- def __init__(self, **kwargs): - self.blacklist = read_lines_from_file("/usr/share/pungi/multilib/runtime-blacklist.conf") - self.whitelist = read_lines_from_file("/usr/share/pungi/multilib/runtime-whitelist.conf") - self.patterns = expand_runtime_patterns(read_runtime_patterns_from_file("/usr/share/pungi/multilib/runtime-patterns.conf")) + def __init__(self, *args, **kwargs): + super(RuntimeMultilibMethod, self).__init__(*args, **kwargs) + self.blacklist = read_lines_from_file(self.config_path+"runtime-blacklist.conf") + self.whitelist = read_lines_from_file(self.config_path+"runtime-whitelist.conf") + self.patterns = expand_runtime_patterns(read_runtime_patterns_from_file(self.config_path+"runtime-patterns.conf"))
def select(self, po): if self.skip(po): @@ -186,8 +190,10 @@ class RuntimeMultilibMethod(MultilibMethodBase):
class FileMultilibMethod(MultilibMethodBase): """explicitely defined whitelist and blacklist""" - def __init__(self, **kwargs): - self.name = "file" + name = "file" + + def __init__(self, *args, **kwargs): + super(FileMultilibMethod, self).__init__(*args, **kwargs) whitelist = kwargs.pop("whitelist", None) blacklist = kwargs.pop("blacklist", None) self.whitelist = self.read_file(whitelist) @@ -212,8 +218,10 @@ class FileMultilibMethod(MultilibMethodBase):
class KernelMultilibMethod(MultilibMethodBase): """kernel and kernel-devel""" - def __init__(self, **kwargs): - self.name = "kernel" + name = "kernel" + + def __init__(self, *args, **kwargs): + super(KernelMultilibMethod, self).__init__(*args, **kwargs)
def select(self, po): if self.is_kernel_or_kernel_devel(po): @@ -223,8 +231,10 @@ class KernelMultilibMethod(MultilibMethodBase):
class YabootMultilibMethod(MultilibMethodBase): """yaboot on ppc""" - def __init__(self, **kwargs): - self.name = "yaboot" + name = "yaboot" + + def __init__(self, *args, **kwargs): + super(YabootMultilibMethod, self).__init__(*args, **kwargs)
def select(self, po): if po.arch in ["ppc"]: @@ -237,9 +247,10 @@ class DevelMultilibMethod(MultilibMethodBase): """all -devel and -static packages""" name = "devel"
- def __init__(self, **kwargs): - self.blacklist = read_lines_from_file("/usr/share/pungi/multilib/devel-blacklist.conf") - self.whitelist = read_lines_from_file("/usr/share/pungi/multilib/devel-whitelist.conf") + def __init__(self, *args, **kwargs): + super(DevelMultilibMethod, self).__init__(*args, **kwargs) + self.blacklist = read_lines_from_file(self.config_path+"devel-blacklist.conf") + self.whitelist = read_lines_from_file(self.config_path+"devel-whitelist.conf")
def select(self, po): if self.skip(po): @@ -267,9 +278,17 @@ class DevelMultilibMethod(MultilibMethodBase):
DEFAULT_METHODS = ["devel", "runtime"] METHOD_MAP = {} -for cls in (AllMultilibMethod, DevelMultilibMethod, FileMultilibMethod, KernelMultilibMethod, NoneMultilibMethod, RuntimeMultilibMethod, YabootMultilibMethod): - method = cls() - METHOD_MAP[method.name] = method + +def init(config_path="/usr/share/pungi/multilib/"): + global METHOD_MAP + + if not config_path.endswith("/"): + config_path += "/" + + for cls in (AllMultilibMethod, DevelMultilibMethod, FileMultilibMethod, KernelMultilibMethod, + NoneMultilibMethod, RuntimeMultilibMethod, YabootMultilibMethod): + method = cls(config_path) + METHOD_MAP[method.name] = method
def po_is_multilib(po, methods): @@ -384,6 +403,7 @@ def main(): if not opts.tmpdir: tmpdir = tempfile.mkdtemp(prefix="multilib_")
+ init() nvra_list = do_multilib(opts.arch, opts.method, opts.repos, tmpdir, opts.logfile) for nvra, method in nvra_list: print "MULTILIB(%s): %s" % (method, nvra)
On Wed, 11 Feb 2015 16:13:20 -0800 "Brian C. Lane" bcl@redhat.com wrote:
I took a look at pungi and decided that with the state it is in, and my lack of deep understanding of it, I don't want to tackle converting it to DNF. So, in order to support the ability to independently move lorax to DNF and Python3 I've converted the library call to run lorax as a process.
I'm still undecided about moving lorax to DNF for F22 or not. I'll probably make a decision on that after tomorrow's Anaconda DNF test day.
I would rather not change this for Fedora 22.
I *am* going to make the switch for rawhide on Friday though, so these changes will need to be in the rawhide version of pungi soon.
Rawhide soon will be switching to the pungi 4 development branch and changing dramatically how we make things. porting that to python 3 and dnf is very high up on the list of things to do.
I've done multiple test runs and things look good, at least on x86_64. The only issues I have right now are that the boot.iso is about 200M larger than the F21 one, and DNF doesn't log transaction decisions yet so it's hard to track down what's getting pulled in by what.
Seems like an extra reason not to switch f22 to using dnf for lorax
Dennis
On Wed, 2015-02-11 at 16:13 -0800, Brian C. Lane wrote:
I took a look at pungi and decided that with the state it is in, and my lack of deep understanding of it, I don't want to tackle converting it to DNF. So, in order to support the ability to independently move lorax to DNF and Python3 I've converted the library call to run lorax as a process.
We're likely going to move to the new pungi (aka distill) at some point anyway, would that change anything?
On Wednesday, February 11, 2015 04:13:20 PM Brian C. Lane wrote:
I took a look at pungi and decided that with the state it is in, and my lack of deep understanding of it, I don't want to tackle converting it to DNF. So, in order to support the ability to independently move lorax to DNF and Python3 I've converted the library call to run lorax as a process.
I'm still undecided about moving lorax to DNF for F22 or not. I'll probably make a decision on that after tomorrow's Anaconda DNF test day.
I *am* going to make the switch for rawhide on Friday though, so these changes will need to be in the rawhide version of pungi soon.
I've done multiple test runs and things look good, at least on x86_64. The only issues I have right now are that the boot.iso is about 200M larger than the F21 one, and DNF doesn't log transaction decisions yet so it's hard to track down what's getting pulled in by what.
Brian C. Lane (3): Close child fds when using subprocess Call lorax as a process not a library Add the option to pass a custom path for the multilib config files
src/bin/pungi.py | 3 ++ src/pypungi/__init__.py | 83 +++++++++++++++++++++++++++---------------------- src/pypungi/config.py | 1 + src/pypungi/multilib.py | 52 +++++++++++++++++++++---------- src/pypungi/util.py | 5 +-- 5 files changed, 88 insertions(+), 56 deletions(-)
Thanks
with some modification I have applied to pungi-4-devel branch
Dennis
On Thu, Mar 12, 2015, at 12:43 PM, Dennis Gilmore wrote:
with some modification I have applied to pungi-4-devel branch
I took a look at this, and I feel things would be nicer if pungi lost the logic for generating the CD/DVD, and instead passed configuration parameters to lorax. Then pungi would be a lot smaller.
Well, I guess it's tricky as lorax doesn't do source CDs, so it'd still have to exist there. But it could at least avoid a lot of the architecture dependencies, etc.
buildsys@lists.fedoraproject.org
-
Brian C. Lane -
Colin Walters -
Dennis Gilmore -
Mathieu Bridon