On Tue, Nov 04, 2014 at 11:01:08PM -0800, Adam Williamson wrote:
On Tue, 2014-11-04 at 23:33 -0600, Dennis Gilmore wrote:
On Mon, 03 Nov 2014 07:22:06 -0800 Adam Williamson awilliam@redhat.com wrote:
On Mon, 2014-11-03 at 08:35 -0500, Matthew Miller wrote:
Adam Williamson awilliam@redhat.com wrote:
This isn't about the trees, but about the repository packages we created that point to the trees, as part of the alpha attempt to do productized netinsts.
Ah, yes. In that case, +1. However, it has the unfortunate side-effect of making it harder to count productized repo access without direct dnf/yum support.
The repos weren't designed to be installed in any case, only used during installation.
umm we can not remove them, anaconda expects there to be a repo called fedora-server or fedora-cloud due to how the compose is done and that is not changing.
why not? we added it in a hurry for alpha, we can remove it easily enough now. I don't think the anaconda guys want the memory overhead of the useless extra repos, they seemed in favour of removing them when I mentioned it in #anaconda.
CCing bcl. IIRC the change either way is a one-liner to pyanaconda/packaging/__init__.py which we've already ping-ponged back and forth like twice.
Which change, I'm losing track.
We should certainly NOT be shipping f21 with all the repos installed and enabled, it slows down the installation and chews up memory unnecessarily.
they need to stick around for f21, we can look at doing different things in f22. We need to work with the anaconda guys to see what we can do and what changes they will need to make.
IIRC the 'include all the enabled repos' hack was a quick fix because nobody had sorted out how to make just the right ones get included during the compose time.
IMHO this is a releng problem. Give anaconda the right repos with the right names and it will do the right thing. When you run lorax/pungi and you want to customize the repos you need to install them in /etc/yum.repos.d/ so that lorax will copy them to the image.
Also, this discussion should be on anaconda-devel, not a cc list. So I've added it to the cc list. I don't have any particular authority in this, just strong opinions :)
On 11/05/2014 10:27 AM, Brian C. Lane wrote:
On Tue, Nov 04, 2014 at 11:01:08PM -0800, Adam Williamson wrote:
On Tue, 2014-11-04 at 23:33 -0600, Dennis Gilmore wrote:
On Mon, 03 Nov 2014 07:22:06 -0800 Adam Williamson awilliam@redhat.com wrote:
On Mon, 2014-11-03 at 08:35 -0500, Matthew Miller wrote:
Adam Williamson awilliam@redhat.com wrote:
This isn't about the trees, but about the repository packages we created that point to the trees, as part of the alpha attempt to do productized netinsts.
Ah, yes. In that case, +1. However, it has the unfortunate side-effect of making it harder to count productized repo access without direct dnf/yum support.
The repos weren't designed to be installed in any case, only used during installation.
umm we can not remove them, anaconda expects there to be a repo called fedora-server or fedora-cloud due to how the compose is done and that is not changing.
why not? we added it in a hurry for alpha, we can remove it easily enough now. I don't think the anaconda guys want the memory overhead of the useless extra repos, they seemed in favour of removing them when I mentioned it in #anaconda.
CCing bcl. IIRC the change either way is a one-liner to pyanaconda/packaging/__init__.py which we've already ping-ponged back and forth like twice.
Which change, I'm losing track.
We should certainly NOT be shipping f21 with all the repos installed and enabled, it slows down the installation and chews up memory unnecessarily.
they need to stick around for f21, we can look at doing different things in f22. We need to work with the anaconda guys to see what we can do and what changes they will need to make.
IIRC the 'include all the enabled repos' hack was a quick fix because nobody had sorted out how to make just the right ones get included during the compose time.
IMHO this is a releng problem. Give anaconda the right repos with the right names and it will do the right thing. When you run lorax/pungi and you want to customize the repos you need to install them in /etc/yum.repos.d/ so that lorax will copy them to the image.
Also, this discussion should be on anaconda-devel, not a cc list. So I've added it to the cc list. I don't have any particular authority in this, just strong opinions :)
OK, I am sure you guys know what you are talking about but it is a little vague to me. Just what are all these repos?
If a repo is rawhide then it should be only the "rolling release" "rawhide". If it is the release under development (e.g., F21) then it should be the F21 branch plus updates-testing (plain updates is empty). If it is production, then it should be "everything" plus an install option of updates.
Or, has something completely different been done? Is there a repo for fedora-server, another for fedora-workstation and yet another for fedora-cloud. How do the Live Installs fit if this is the case?
While I intend to switch form using netinstall plus kickstart to doing installs based on a Live Image, I will be rolling my own. Therefore, I guess I will be creating the GOVOF Product :-) [That is, Gene's Own Version Of Fedora]
The difference will be package selection. I have yet to determine which packages to include at install time as oppose to those packages post-installed. One big thing for me is that I want to have the system called Fedora and not Generic and all packages should be available in a repository.
BTW, I have become impressed with the Live Install since it greatly reduces the time necessary to perform the install.
Anyway, I am just trying to understand how/what things will be different with Fedora 21 as compared to previous releases.
Gene
On Wed, 2014-11-05 at 14:43 -0500, Gene Czarcinski wrote:
OK, I am sure you guys know what you are talking about but it is a little vague to me. Just what are all these repos?
We've cleaned this up now, but just to write it down one more time for the record:
The repos in question are the fedora-workstation.repo , fedora-server.repo and fedora-cloud.repo repos that were packaged in fedora-repos-anaconda around the time of Alpha.
The *original* plan was that there would be Server, Workstation and Cloud network install images that would, by default, offer *only* the package groups those products considered part of their product.
The design for achieving this was that each of those images would have the fedora-(product).repo for that Product as its base repository. Those repos are/were configured to use mirrormanager with a URL like:
https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-workstation-21&...
which was intended at pre-release time to point to either the latest TC/RC in staging, e.g.:
https://dl.fedoraproject.org/pub/alt/stage/21_Alpha_RC1/Workstation/x86_64/o...
and/or to the current frozen milestone tree, e.g.:
https://dl.fedoraproject.org/pub/fedora/linux/releases/test/21-Alpha/Worksta...
(I don't recall if we ever actually nailed down precisely when the mirrormanager redirect would point to which, but I guess it doesn't matter too much now).
Those trees indeed contained only the package groups each Product wanted (according to the kickstarts used to create them).
For Alpha we actually implemented all that, so there was a Workstation network install image that used fedora-workstation.repo as its base repository and a Server network install image that used fedora-server.repo as its base repository.
However, it turned out that didn't work quite as intended. anaconda is by default quite 'greedy' when it comes to repositories - without an explicit configuration it'll use any valid ones it finds (in /etc/anaconda.repos.d) as supplementary repositories. It reads comps info from supplementary repositories. So although the 'Product-specific' network install images used the correct pruned repository as their *base* repository, they still offered the full range of package groups, because they found them in fedora.repo , which was also present on the network install images.
Removing fedora.repo from the images would've been a bit tricky but probably possible somehow or other, but when I tested to check that that would be enough to solve the problem, it turned out it wouldn't. anaconda would also use fedora-updates.repo as a supplementary repository in all cases, and fedora-updates-testing.repo as a supplementary repo prior to release.
Prior to release, updates-testing usually has a full set of comps data, and post-release, updates usually does. So even if you removed fedora.repo from the images, the group list wasn't filtered as expected, because anaconda would get the groups data from one of the updates repos.
We spent a bit of time kicking around ways to "solve" that but there really aren't any very good ones. It's not even as simple as 'how can we remove the updates repositories from the images', because we actually *want* them to be available for people to get updated packages during install.
So what we wound up doing instead was saying "you know what, no-one really *wants* these Product-specific network installs in any case. Everyone seems to be happy with us just officially shipping the Server one, with Server as the *default* package group but all the other package groups available, so the Server image also works for doing network installs of Workstation and Cloud. We want to make network install *possible* for particular use cases for those products, but it's not the primary distribution method and doesn't need a clever Product-specific image." (Really, what we're now calling the 'Server network install image' is pretty much exactly what the plain old 'network install image' was for all previous releases).
So for Alpha and Beta we just left in all the crap we'd done to try and make the images Product-specific, but said it was totally fine that it didn't work. For Beta we dropped the Workstation installer tree from the mirrors entirely, and we fixed comps so that Server is the default package group.
This *works*, but it leaves around a bunch of crap we don't really need. We have a line in lorax which pulls in the 'product-specific' repositories to the network install images, even though we don't need them any more - so when you do an Alpha or Beta network install anaconda is actually setting up all of fedora-server.repo , fedora-workstation.repo and fedora-cloud.repo , even though they're entirely useless and it'd be just fine using only fedora.repo . We also have to keep the Product repositories set up in mirrormanager, or else network installs will start failing because they can't configure the base repository (which is still the Product-specific repository).
So what bcl just did yesterday was stop lorax from pulling in the product-specific repositories when composing images any more, and re-instate a change from very early Alpha days, before we actually put the Product-specific repos in place, which tells anaconda to use 'fedora' as the base repository for Fedora installs (as it did before the whole Product kerfuffle). That's what this change is:
https://git.fedorahosted.org/cgit/anaconda.git/commit/?id=9b96d5f6d3f1d72896...
productName(), if you trace out the code, ultimately turns out to be 'the value of PRODUCT in /.buildstamp'. Which for a Product-ized Fedora, is "Fedora-(Product)". e.g. for Server, it's "Fedora-Server". So, when the line reads:
DEFAULT_REPOS = [productName.lower(), "rawhide"]
it makes DEFAULT_REPOS a list containing the strings 'fedora-server' and 'rawhide', and that means anaconda will try to use a repository named 'fedora-server' as its base repo, and if that doesn't work it'll try and use 'rawhide', and if that doesn't work it'll go sit in a corner and cry. i.e., when the line looks like that, we're expecting there to be a 'Product-ized' repo for anaconda to use.
When instead the line reads:
DEFAULT_REPOS = [productName.split('-')[0].lower(), "rawhide"]
it makes DEFAULT_REPOS a list containing the strings 'fedora' and 'rawhide' (because we split "Fedora-Server" with - as the separator, take the first of the strings that results from the split, and lowercase it). anaconda will try to use the repository named 'fedora' as its base repository. i.e., when the line looks like that, we're *not* looking for a Product-ized repo, we're just using the universal, non-Product-y 'fedora' repository. This is what was used as the base repo in previous releases - before we had Products, the PRODUCT value in /.buildstamp was just "Fedora", so the simpler version of the line would produce 'fedora'.
(Happily we don't have to worry too much about RHEL or CentOS here, because there's no such thing as a 'default' network install for those - if you boot their netinst images, they don't successfully discover mirrors automatically. You have to explicitly provide a source, interactively, on the command line, or in your kickstart.)
I guess it gets slightly confusing because in order to produce more or less the *same* behaviour as F20 we have to *change* anaconda (change the DEFAULT_REPOS line), whereas if we *don't* change the DEFAULT_REPOS line we get *different* behaviour from F20. But that's all that's going on.
On Wed, 2014-11-05 at 14:43 -0500, Gene Czarcinski wrote:
OK, I am sure you guys know what you are talking about but it is a little vague to me. Just what are all these repos?
We've cleaned this up now, but just to write it down one more time for the record:
The repos in question are the fedora-workstation.repo , fedora-server.repo and fedora-cloud.repo repos that were packaged in fedora-repos-anaconda around the time of Alpha.
The *original* plan was that there would be Server, Workstation and Cloud network install images that would, by default, offer *only* the package groups those products considered part of their product.
The design for achieving this was that each of those images would have the fedora-(product).repo for that Product as its base repository. Those repos are/were configured to use mirrormanager with a URL like:
https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-workstation-21&...
which was intended at pre-release time to point to either the latest TC/RC in staging, e.g.:
https://dl.fedoraproject.org/pub/alt/stage/21_Alpha_RC1/Workstation/x86_64/o...
and/or to the current frozen milestone tree, e.g.:
https://dl.fedoraproject.org/pub/fedora/linux/releases/test/21-Alpha/Worksta...
(I don't recall if we ever actually nailed down precisely when the mirrormanager redirect would point to which, but I guess it doesn't matter too much now).
Those trees indeed contained only the package groups each Product wanted (according to the kickstarts used to create them).
For Alpha we actually implemented all that, so there was a Workstation network install image that used fedora-workstation.repo as its base repository and a Server network install image that used fedora-server.repo as its base repository.
However, it turned out that didn't work quite as intended. anaconda is by default quite 'greedy' when it comes to repositories - without an explicit configuration it'll use any valid ones it finds (in /etc/anaconda.repos.d) as supplementary repositories. It reads comps info from supplementary repositories. So although the 'Product-specific' network install images used the correct pruned repository as their *base* repository, they still offered the full range of package groups, because they found them in fedora.repo , which was also present on the network install images.
Removing fedora.repo from the images would've been a bit tricky but probably possible somehow or other, but when I tested to check that that would be enough to solve the problem, it turned out it wouldn't. anaconda would also use fedora-updates.repo as a supplementary repository in all cases, and fedora-updates-testing.repo as a supplementary repo prior to release.
Prior to release, updates-testing usually has a full set of comps data, and post-release, updates usually does. So even if you removed fedora.repo from the images, the group list wasn't filtered as expected, because anaconda would get the groups data from one of the updates repos.
We spent a bit of time kicking around ways to "solve" that but there really aren't any very good ones. It's not even as simple as 'how can we remove the updates repositories from the images', because we actually *want* them to be available for people to get updated packages during install.
So what we wound up doing instead was saying "you know what, no-one really *wants* these Product-specific network installs in any case. Everyone seems to be happy with us just officially shipping the Server one, with Server as the *default* package group but all the other package groups available, so the Server image also works for doing network installs of Workstation and Cloud. We want to make network install *possible* for particular use cases for those products, but it's not the primary distribution method and doesn't need a clever Product-specific image." (Really, what we're now calling the 'Server network install image' is pretty much exactly what the plain old 'network install image' was for all previous releases).
So for Alpha and Beta we just left in all the crap we'd done to try and make the images Product-specific, but said it was totally fine that it didn't work. For Beta we dropped the Workstation installer tree from the mirrors entirely, and we fixed comps so that Server is the default package group.
This *works*, but it leaves around a bunch of crap we don't really need. We have a line in lorax which pulls in the 'product-specific' repositories to the network install images, even though we don't need them any more - so when you do an Alpha or Beta network install anaconda is actually setting up all of fedora-server.repo , fedora-workstation.repo and fedora-cloud.repo , even though they're entirely useless and it'd be just fine using only fedora.repo . We also have to keep the Product repositories set up in mirrormanager, or else network installs will start failing because they can't configure the base repository (which is still the Product-specific repository).
So what bcl just did yesterday was stop lorax from pulling in the product-specific repositories when composing images any more, and re-instate a change from very early Alpha days, before we actually put the Product-specific repos in place, which tells anaconda to use 'fedora' as the base repository for Fedora installs (as it did before the whole Product kerfuffle). That's what this change is:
https://git.fedorahosted.org/cgit/anaconda.git/commit/?id=9b96d5f6d3f1d72896...
productName(), if you trace out the code, ultimately turns out to be 'the value of PRODUCT in /.buildstamp'. Which for a Product-ized Fedora, is "Fedora-(Product)". e.g. for Server, it's "Fedora-Server". So, when the line reads:
DEFAULT_REPOS = [productName.lower(), "rawhide"]
it makes DEFAULT_REPOS a list containing the strings 'fedora-server' and 'rawhide', and that means anaconda will try to use a repository named 'fedora-server' as its base repo, and if that doesn't work it'll try and use 'rawhide', and if that doesn't work it'll go sit in a corner and cry. i.e., when the line looks like that, we're expecting there to be a 'Product-ized' repo for anaconda to use.
When instead the line reads:
DEFAULT_REPOS = [productName.split('-')[0].lower(), "rawhide"]
it makes DEFAULT_REPOS a list containing the strings 'fedora' and 'rawhide' (because we split "Fedora-Server" with - as the separator, take the first of the strings that results from the split, and lowercase it). anaconda will try to use the repository named 'fedora' as its base repository. i.e., when the line looks like that, we're *not* looking for a Product-ized repo, we're just using the universal, non-Product-y 'fedora' repository. This is what was used as the base repo in previous releases - before we had Products, the PRODUCT value in /.buildstamp was just "Fedora", so the simpler version of the line would produce 'fedora'.
(Happily we don't have to worry too much about RHEL or CentOS here, because there's no such thing as a 'default' network install for those - if you boot their netinst images, they don't successfully discover mirrors automatically. You have to explicitly provide a source, interactively, on the command line, or in your kickstart.)
I guess it gets slightly confusing because in order to produce more or less the *same* behaviour as F20 we have to *change* anaconda (change the DEFAULT_REPOS line), whereas if we *don't* change the DEFAULT_REPOS line we get *different* behaviour from F20. But that's all that's going on.
On Fri, Nov 07, 2014 at 02:01:35PM -0800, Adam Williamson wrote:
So what we wound up doing instead was saying "you know what, no-one really *wants* these Product-specific network installs in any case. Everyone seems to be happy with us just officially shipping the Server one, with Server as the *default* package group but all the other package groups available, so the Server image also works for doing network installs of Workstation and Cloud. We want to make network install *possible* for particular use cases for those products, but it's not the primary distribution method and doesn't need a clever Product-specific image." (Really, what we're now calling the 'Server network install image' is pretty much exactly what the plain old 'network install image' was for all previous releases).
I think if we decide to stay with this for F22, we should probably look at just calling it the "generic network install image" (and maybe asking Base WG if they want to own it). Or it's possible that the cloud/server/workstation WGs will really want to get into further-differentiated install defaults, in which case having separate ones _might_ make sense again -- although we could also just document to use updates=http://mirror.somewhere.fedora/foo/workstation/product.img if people want that.
----- Original Message -----
On Fri, Nov 07, 2014 at 02:01:35PM -0800, Adam Williamson wrote:
So what we wound up doing instead was saying "you know what, no-one really *wants* these Product-specific network installs in any case. Everyone seems to be happy with us just officially shipping the Server one, with Server as the *default* package group but all the other package groups available, so the Server image also works for doing network installs of Workstation and Cloud. We want to make network install *possible* for particular use cases for those products, but it's not the primary distribution method and doesn't need a clever Product-specific image." (Really, what we're now calling the 'Server network install image' is pretty much exactly what the plain old 'network install image' was for all previous releases).
I think if we decide to stay with this for F22, we should probably look at just calling it the "generic network install image" (and maybe asking Base WG if they want to own it).
I can bring it to Base WG meeting as a topic next time.
Jaroslav
On 11/10/2014 04:16 AM, Jaroslav Reznik wrote:
----- Original Message -----
On Fri, Nov 07, 2014 at 02:01:35PM -0800, Adam Williamson wrote:
So what we wound up doing instead was saying "you know what, no-one really *wants* these Product-specific network installs in any case. Everyone seems to be happy with us just officially shipping the Server one, with Server as the *default* package group but all the other package groups available, so the Server image also works for doing network installs of Workstation and Cloud. We want to make network install *possible* for particular use cases for those products, but it's not the primary distribution method and doesn't need a clever Product-specific image." (Really, what we're now calling the 'Server network install image' is pretty much exactly what the plain old 'network install image' was for all previous releases).
I think if we decide to stay with this for F22, we should probably look at just calling it the "generic network install image" (and maybe asking Base WG if they want to own it).
I can bring it to Base WG meeting as a topic next time.
If you take a look over at the kickstart mailing list, I have posted a little writeup explaining how to use netinst to perform a kickstart install of a Live Image. You can have the kickstart and squashfs.img files on the network or you can put the netinstall, kickstart, and squashfs.img files on a usb-stick or combinations of the previous.
I believe that the network install iso should not be referred to as the "Generic Network Install Image' but instead it shoud be the Fedora Network Install Image. This image by itself really does nothing (except maybe come up in rescue mode). The delivery "payload" (installed system) can be anything from the workstation product to GOVOF (Gene's Own Version Of Fedora:-) ) and will likely be the tool for kickstart installs. The installed product may be a Fedora Product, a Fedora Nonproduct, or a Generic Product but I believe the tool itself should have the Fedora Brand on it.
Gene
Gene
On Mon, Nov 10, 2014 at 09:08:01AM -0500, Gene Czarcinski wrote:
I believe that the network install iso should not be referred to as the "Generic Network Install Image' but instead it shoud be the Fedora Network Install Image. This image by itself really does nothing (except maybe come up in rescue mode). The delivery "payload" (installed system) can be anything from the workstation product to GOVOF (Gene's Own Version Of Fedora:-) ) and will likely be the tool for kickstart installs. The installed product may be a Fedora Product, a Fedora Nonproduct, or a Generic Product but I believe the tool itself should have the Fedora Brand on it.
Sounds right to me, yeah.
anaconda-devel@lists.stg.fedoraproject.org