Hi everyone,
As you probably know, we had a FESCo meeting the other day, where we raised our "work in progress" proposal for Fedora ARM Primary Arch status in F18. This is a work in progress, and isn't final yet. We meant to ping you about now, but probably should have done so even before posting that draft so you don't feel left out :) Note that we know Fedora 18 is an optimistic goal, but this is Fedora, and there will of course be a Fedora 19, and 20, and so on if it takes that.
Why are we doing this? We think it's consistent with Fedora's mission statement. After all, Fedora is all about driving technology, and if you want to see where the buzz and excitement is, any IT magazine will tell you that ARM is the "next big thing". We think it already is pretty big, and it's growing. In a few years, it's going to be just amazing. We can get there as a Secondary if we have to, but we think it's time to start looking at changing that. Unlike other secondaries, ARM is growing rapidly, and it's cheap. You can buy an ARM board for less than an order of magnitude cheaper than a cheap Intel box.
All this awesomeness does mean you get some compromises. For example, typical Fedora ARM users don't run Anaconda today, they dd a pre-built image onto an SD Card and boot that (and we're working on getting these built using the standard "appliance" like Fedora tooling). In the coming months though, we'll see ARM servers with PXE and TFTP. In the future, you'll have ARM servers booting using UEFI and smelling increasingly like x86 over time. But it's not just server. The Raspberry Pi is going to be - *is* - blowing everyone away. It reminds me of being a kid and getting a BBC Master/B computer and being able to write programs and understand how the computer actually worked. As Eben says, he's trying to rekindle that kind of feeling kids used to have. We have made good choices to keep armv5 support around and that means many Pi users will be running Fedora by *default*. In the future, the next OLPC device (as you know) will join the Pi in being ARM based.
Anyway. Here is the proposal we're working on:
https://fedoraproject.org/wiki/Features/FedoraARM
I want to get some specific input on a few questions if I may, and then also ask you that you please let me/us know of any concerns/input:
0). Hardware. We are making certain plans for ARM hardware to be made available for Fedora developers, in addition to the existing FAS-based approach we have in Seneca today. What developer hardware would you consider that you would want/need in order to be able to support ARM as a target in Fedora? Do we need to buy each of you an ARM board? :)
1). Kernel sub-packages. 32-bit ARM is a bit of a zoo, and everyone knows it. It's being worked on by Deepak's ARM kernel team at Linaro, but well, nobody is kidding around in thinking we're at a single zimage stage quite yet. That being the case, and for general sanity and supportability, we do not intend to target kernels for every ARM platform in existence. Instead, we propose a small number of kernels (e.g. for omap, tegra, versatile, pi, and Calxeda highbank) as our upper limit, all derived from pure-upstream sources. We currently build a default of the versatile kernel (for Versatile Express, the ARM built development platform that qemu also provides a simulation thereof) as our fallback because it represents real hardware, and is qemu-able.
We want to be able to build a number of official ARM kernels (which will shrink over time), and for those weird and whacky things that nobody is even remotely calling mainline Fedora ARM, we're sure third parties will provide independent non-Fedora bits for the adventurous. Anyway, back to our official Fedora plan. What we don't want to do is disrupt your workflows by taking massively longer to build a kernel. Right now, Koji tells me an x86 build takes as little as 12 minutes, whereas an ARM build (on non-Enterprise hardware, the kind that will be replaced with real early 32-bit ARM server builders as part of this) takes 4 hours today and will hopefully fall to 1.5, but still won't be instantaneous. And that won't give us all the variants either. So we have some possibilities:
- We build every ARM kernel every time - We build e.g. versatile in general and periodically build the other kernels by use of a SPEC file macro knob
I think the second is probably a non-starter for you. So assuming, the first is the preferred option, then my question becomes:
- How long is acceptable for a kernel build to take?
Now, a trivial SPEC file or general non-arch kernel bug is likely to fail on x86 well before it fails on ARM. That will of course take care of many generic build issues that will fail a parent Koji build quickly. And we're looking to base the official Fedora ARM builder hardware in Phoenix on the first generation 32-bit servers, and these will have up to 288 cores in 2U. Several of those should provide enough builders that we don't get to a point of not accepting additional builds. Still, that's great, but pipelining doesn't really solve the fundamental speed issue. So, another option is that we modify Koji to submit sub-package tasks across multiple builders. i.e. all of the ARM subpackages (and this would happen for x86 variants too) would get submitted to builders at the same time, rather than linearly. It's a lot of work, but it's doable. Especially if it's the only option. It comes down to how long you guys think is the longest you are willing to wait for an overall all-arch build of the kernel to take in Koji.
To be clear, nothing in here is concerned with 64-bit yet. Those crazy core counts I'm talking about in servers we'll use for building are all happening this year, in the next 6 months. They'll be individually coherent quad core Cortex-A9-like systems, like your Android tablet on steroids, but not (yet) taking on x86. We don't ever expect ARM to take on Intel head-to-head for performance, they're in it for density and power saving, amongst other things. Next year, we'll see A15 based servers. And the following year or thereabouts, I suspect we'll see the first ARMv8 (AArch64) servers in testing. At that point, you're looking at individual coherence domains of 8 cores+ (and it's a lot +), at a few GHz, and so on. So we'll get you higher performing builders :)
2). Impact. What does making ARM a primary arch mean to you? Not what do we think it means, but what do you think it means to you in terms of:
- How will this positively or negatively impact your role? - What level of disruption are you willing to accept? - Are you willing to make any changes to workflow?
We don't intend for this to be a trainwreck. If it's not ready, it won't be PA, period. But we want to know how flexible you guys are willing to be as we figure this out. If you want to wait until we have total parity, single zimage, and we're just like x86, that is good to know (and discuss) right now :)
I'm sure there's other stuff we can discuss. So let's do that :) If we have useful input on IRC, I'll followup to this thread, but I like the idea of doing it by email to keep a record of the discussion.
Thanks for reading - whew! I am...verbose.
Jon.
Hi,
So, another option is that we modify Koji to submit sub-package tasks across multiple builders. i.e. all of the ARM subpackages (and this would happen for x86 variants too) would get submitted to builders at the same time, rather than linearly. It's a lot of work, but it's doable.
Hmm, that wouldn't be subpackages I guess, but different variants of the same source rpm, i.e. rpmbuild --define "variant={tegra,omap,...}" kernel.spec
Then have the specfile pick the kernel config + package name based on %variant.
Could be useful to parallelize other packages too:
qemu for example could be splitted by target arch, so you'll have variant={x86,arm,ppc,sparc,mips,...}. Wouldn't work as good as for the kernel though as there are common files for all archs, are compiled once with todays subpackages but would compiled multiple times with variants.
Maybe gcc could profit too? I remember there is (was?) a huge gcc build time difference depending on whenever java language support was enabled or not. So "./configure --languages=%{variant}" could possibly speed up gcc build time by building all langs in parallel?
cheers, Gerd
On Wed, Mar 21, 2012 at 10:13:45PM -0400, Jon Masters wrote:
Just a note here: These are pretty much just my thoughts with a kernel maintainer hat on. I'm not speaking for the whole team.
Anyway. Here is the proposal we're working on:
https://fedoraproject.org/wiki/Features/FedoraARM
I want to get some specific input on a few questions if I may, and then also ask you that you please let me/us know of any concerns/input:
0). Hardware. We are making certain plans for ARM hardware to be made available for Fedora developers, in addition to the existing FAS-based approach we have in Seneca today. What developer hardware would you consider that you would want/need in order to be able to support ARM as a target in Fedora? Do we need to buy each of you an ARM board? :)
I think that depends on a number of things. I would have put this question last if I were making the list to be honest. However I do appreciate that you're actively thinking about how to increase the ARM developer community ;)
1). Kernel sub-packages. 32-bit ARM is a bit of a zoo, and everyone knows it. It's being worked on by Deepak's ARM kernel team at Linaro, but well, nobody is kidding around in thinking we're at a single zimage stage quite yet. That being the case, and for general sanity and supportability, we do not intend to target kernels for every ARM platform in existence. Instead, we propose a small number of kernels (e.g. for omap, tegra, versatile, pi, and Calxeda highbank) as our upper limit, all derived from pure-upstream sources. We currently build a default of the versatile kernel (for Versatile Express, the ARM built development platform that qemu also provides a simulation thereof) as our fallback because it represents real hardware, and is qemu-able.
We want to be able to build a number of official ARM kernels (which will shrink over time), and for those weird and whacky things that nobody is even remotely calling mainline Fedora ARM, we're sure third parties will provide independent non-Fedora bits for the adventurous. Anyway, back to our official Fedora plan. What we don't want to do is disrupt your workflows by taking massively longer to build a kernel. Right now, Koji tells me an x86 build takes as little as 12 minutes, whereas an ARM build (on non-Enterprise hardware, the kind that will be replaced with real early 32-bit ARM server builders as part of this) takes 4 hours today and will hopefully fall to 1.5, but still won't be instantaneous. And that won't give us all the variants either. So we have some possibilities:
- We build every ARM kernel every time
- We build e.g. versatile in general and periodically build the other
kernels by use of a SPEC file macro knob
I think the second is probably a non-starter for you. So assuming, the first is the preferred option, then my question becomes:
The second option isn't necessarily a non-starter. It depends on how it's done. If the macro was in place to enable additional board kernels and was toggled ON from Beta->release, then it might be acceptable. You would leave it off during the ramp up to Beta to limit impact perhaps. Not great, but not entirely unresonable and somewhat along the lines of how we handle the debug options for the kernel during development.
My biggest requirement in this regard is that whatever ARM kernels are built are done from a single SRPM in a single koji build invocation. If more kernels are needed, that's a new SRPM and new koji build. I don't want to see someone trying to rebuild an SRPM already built in koji to enable more kernels and shove those into the repo from some side location.
- How long is acceptable for a kernel build to take?
I'm not sure there's a hard and fast timeframe. Even the x86 builds vary for things like building release vs. debug kernels. Pretty sure we'd know when we see something that would be bad though. For example:
http://arm.koji.fedoraproject.org/koji/buildinfo?buildID=58088 http://arm.koji.fedoraproject.org/koji/buildinfo?buildID=56442
Those are probably way too long. This one:
http://arm.koji.fedoraproject.org/koji/buildinfo?buildID=59980
seems totally acceptable to me, but looking at what it produced I would gather it's mostly useless for the ARM project (perhaps just the qemu thing?). I'm not sure building only the qemu kernel is a great way to go either, both from a ARM _and_ kernel perspective. It might be better to build that + 1 board from each armv5tel and armv7hl. I know that gets into "which do we build" but at least it allows people to test on their hardware. I looked, but didn't seem to find a build configured like that. It would be interesting to see what the time looks like there.
Now, a trivial SPEC file or general non-arch kernel bug is likely to fail on x86 well before it fails on ARM. That will of course take care of many generic build issues that will fail a parent Koji build quickly.
Yes. However, waiting for a day for your build to complete only to have it be canceled because some ARM variant died is one of the larger concerns.
The time issue mostly becomes important when it starts negatively impacting our ability to get fixes out, particularly security fixes. If we have to wait 8 hours to get a build for a CVE fix, that effectively means we're not going to get it out until the following day in a lot of cases.
The other time sensitive issue is during a rebase of the kernel in rawhide. Not sensitive in the same way as above, but along the lines of "oh hell. ARM doesn't build in the latest merge window kernel I'm trying to build and it took 12 hours to find that out." That kind of thing is going to seriously wear on the patience of whoever happens to be working on rawhide that month. I realize there's no great solution to that, but it is a concern.
issue. So, another option is that we modify Koji to submit sub-package tasks across multiple builders. i.e. all of the ARM subpackages (and this would happen for x86 variants too) would get submitted to builders at the same time, rather than linearly. It's a lot of work, but it's doable. Especially if it's the only option. It comes down to how long you guys think is the longest you are willing to wait for an overall all-arch build of the kernel to take in Koji.
That sounds like you'd need to both modify Koji to handle that, and rework package spec files to somehow tell Koji "hey, we're going to be X variants from this SRPM". I'm not sure the kernel.spec lends itself to easily farming out builds to separate machines at the moment, since it just loops through all variants it needs to build and does them in a serial fashion (e.g. i686, i686-debug, i686-PAE, i686-PAEdebug).
That option is interesting, but I don't consider it feasible for f18 at all and I would even be surprised if koji was able to handle it for f19.
2). Impact. What does making ARM a primary arch mean to you? Not what do we think it means, but what do you think it means to you in terms of:
- How will this positively or negatively impact your role?
Positive: Broader Fedora dominance, I guess. Maybe some new learning. Negative: More bugs, more platforms to support, more time spent.
That might be pretty black and white, but at least I came up with _some_ positives. Honestly, if ARM is handled well upstream and by the ARM team, it might not impact us more than waiting a bit for builds. If it _isn't_ handled well upstream it can quickly become a nightmare.
- What level of disruption are you willing to accept?
Can you elaborate on disruption? We already spend time mucking around with ARM configs during rebase (not intelligently mind you), and I don't think we grumbled too much about it. Aside from more bugs, is there something else you were thinking?
- Are you willing to make any changes to workflow?
I think we're pretty open minded. We might not be as gung-ho about something as the owners, but if the changes aren't unreasonable we tend to evaluate them fairly. It's going to depend on what's proposed.
I'll counter with a question to you. Who (as in name(s)) is going to be the main ARM kernel person? My ARM knowledge is limited at best, I don't think Dave has much at all, and I'm not sure about Justin but I'd be surprised if he was a closet ARM expert. Do you have people in mind to handle the HW specific issues that pop up so we can assign bugs to them?
We don't intend for this to be a trainwreck. If it's not ready, it won't be PA, period. But we want to know how flexible you guys are willing to be as we figure this out. If you want to wait until we have total parity, single zimage, and we're just like x86, that is good to know (and discuss) right now :)
Personally, I don't think it has to be one vmlinux + 18 FDTs to be suitable. A smaller set of board specific kernels is probably doable, depending on some of the factors mentioned above. I see that as similar to how we built e.g. ppc, ppc-smp, ppc64, ppc64-kdump. Nobody particularly _liked_ doing that but we lived.
josh
Hey Josh,
Good to talk with you...where are you based now anyway, OOI? I ought to know this, but perhaps that neuron is temporarily unavailable.
On 03/22/2012 09:36 AM, Josh Boyer wrote:
On Wed, Mar 21, 2012 at 10:13:45PM -0400, Jon Masters wrote:
0). Hardware. We are making certain plans for ARM hardware to be made available for Fedora developers, in addition to the existing FAS-based approach we have in Seneca today. What developer hardware would you consider that you would want/need in order to be able to support ARM as a target in Fedora? Do we need to buy each of you an ARM board? :)
I think that depends on a number of things. I would have put this question last if I were making the list to be honest. However I do appreciate that you're actively thinking about how to increase the ARM developer community ;)
Ah yes, but I do need to sweeten the pot somehow :) In all seriousness, though, we're not opposed to buying the kernel team ARM boards if it will mean you're able to play with stuff and help with feedback.
- We build every ARM kernel every time
- We build e.g. versatile in general and periodically build the other
kernels by use of a SPEC file macro knob
I think the second is probably a non-starter for you. So assuming, the first is the preferred option, then my question becomes:
The second option isn't necessarily a non-starter. It depends on how it's done. If the macro was in place to enable additional board kernels and was toggled ON from Beta->release, then it might be acceptable. You would leave it off during the ramp up to Beta to limit impact perhaps. Not great, but not entirely unresonable and somewhat along the lines of how we handle the debug options for the kernel during development.
Excellent. This was my original thought precisely. However, I think Matthew has a strong reservation against this. Therefore, I would like to know what the collective consensus is on this one?
My biggest requirement in this regard is that whatever ARM kernels are built are done from a single SRPM in a single koji build invocation. If more kernels are needed, that's a new SRPM and new koji build. I don't want to see someone trying to rebuild an SRPM already built in koji to enable more kernels and shove those into the repo from some side location.
Right. How right you are. This actually did get suggested in my team. I have tried to shoot it down as a crazy idea and I now have this wonderful email of yours to back up that opinion :) Two things I think are immutable on this topic:
1). We have only one kernel package. One SPEC/SRPM. Others might make non-Fedora kernels for $random, and that's fine, but that's not official. For example, we won't pull in non-upstream patches for Nvidia systems to have better framebuffer support, but those are headed up and indeed we had a chat with them about it recently. Meanwhile, if they want to host a separate testing kernel with those (open source) patches, they can be free to do so as anyone else is able to.
2). There is only one official tagged build at a time. We don't re-issue builds or do some other nonsense.
- How long is acceptable for a kernel build to take?
I'm not sure there's a hard and fast timeframe. Even the x86 builds vary for things like building release vs. debug kernels. Pretty sure we'd know when we see something that would be bad though. For example:
http://arm.koji.fedoraproject.org/koji/buildinfo?buildID=58088 http://arm.koji.fedoraproject.org/koji/buildinfo?buildID=56442
Those are probably way too long. This one:
Right, those are what you get when you build all of the possibilities we might enable. That would be the worst worst case build time for a post-beta/whatever kernel if we split it as discussed above. Note:
1). Koji could be modified to issue those in parallel, so the real build time would be similar to the following one you liked more.
2). The Enterprise hardware even this year will be at least twice as fast as that worst case time. Current suped-up cellphone hardware has crappy cache sizes, etc. The first gen of 32-bit servers will have 4MB cache sizes, 4 cores, and up to 4GB of RAM, which is much improved. The second generation will take that memory over 8GB and the cores will be a lot higher performance. That's next year. The year after, we can start to look at 64-bit systems. We can run a 32-bit userspace on a 64-bit system so we can use those beefier builders later on.
http://arm.koji.fedoraproject.org/koji/buildinfo?buildID=59980
seems totally acceptable to me, but looking at what it produced I would gather it's mostly useless for the ARM project (perhaps just the qemu thing?).
Yes. That's the minimum kernel we discussed. That's "versatile" as in Versatile Express, the marketing-ish name of the FPGA development system ARM produce that can support many of their different cores. We have a physical VE with the hardware from a server semiconductor already in place prior to fabrication for example. The thing is, Ve is both real hardware and the default for qemu.
I'm not sure building only the qemu kernel is a great way to go either, both from a ARM _and_ kernel perspective. It might be better to build that + 1 board from each armv5tel and armv7hl.
That would be fine too. I mean, we could compromise and choose a set of qemu+obvious board as the minimum and then turn on the rest at certain points in the cycle. Whatever works for you guys.
I know that gets into "which do we build" but at least it allows people to test on their hardware. I looked, but didn't seem to find a build configured like that. It would be interesting to see what the time looks like there.
Ok. That will be useful data to have. We can look at that.
Now, a trivial SPEC file or general non-arch kernel bug is likely to fail on x86 well before it fails on ARM. That will of course take care of many generic build issues that will fail a parent Koji build quickly.
Yes. However, waiting for a day for your build to complete only to have it be canceled because some ARM variant died is one of the larger concerns.
Absolutely. Clearly, a day is nuts. But it sounds like a few hours for ARM isn't as big a deal. Would 2 or 4 hours be your upper limit? etc.?
issue. So, another option is that we modify Koji to submit sub-package tasks across multiple builders. i.e. all of the ARM subpackages (and this would happen for x86 variants too) would get submitted to builders at the same time, rather than linearly. It's a lot of work, but it's doable. Especially if it's the only option. It comes down to how long you guys think is the longest you are willing to wait for an overall all-arch build of the kernel to take in Koji.
That sounds like you'd need to both modify Koji to handle that, and rework package spec files to somehow tell Koji "hey, we're going to be X variants from this SRPM". I'm not sure the kernel.spec lends itself to easily farming out builds to separate machines at the moment, since it just loops through all variants it needs to build and does them in a serial fashion (e.g. i686, i686-debug, i686-PAE, i686-PAEdebug).
That option is interesting, but I don't consider it feasible for f18 at all and I would even be surprised if koji was able to handle it for f19.
Right. That's more of a if we have to, but also something we could go in with the expectation of doing. For example, if you could live with 2-4 hours build time for ARM kernels (or whatever you guys decide) for now, and then we try to parallelize and make that go away later on. Or maybe it's a non-starter for the other guys to wait that long for a build so they'll say they need this kind of Koji thing as a gate.
2). Impact. What does making ARM a primary arch mean to you? Not what do we think it means, but what do you think it means to you in terms of:
- How will this positively or negatively impact your role?
Positive: Broader Fedora dominance, I guess. Maybe some new learning. Negative: More bugs, more platforms to support, more time spent.
That might be pretty black and white, but at least I came up with _some_ positives. Honestly, if ARM is handled well upstream and by the ARM team, it might not impact us more than waiting a bit for builds. If it _isn't_ handled well upstream it can quickly become a nightmare.
Yep. We're counting on the level of upstream investment, Linaro involvement, and all of that goodness to help us out.
- What level of disruption are you willing to accept?
Can you elaborate on disruption? We already spend time mucking around with ARM configs during rebase (not intelligently mind you), and I don't think we grumbled too much about it. Aside from more bugs, is there something else you were thinking?
Well, it's really whether you're ok with that kind of thing sometimes. We can't promise to be perfect, and in fact we can promise crap like that will come up...so it's really whether you can live with that.
- Are you willing to make any changes to workflow?
I think we're pretty open minded. We might not be as gung-ho about something as the owners, but if the changes aren't unreasonable we tend to evaluate them fairly. It's going to depend on what's proposed.
Cool.
I'll counter with a question to you. Who (as in name(s)) is going to be the main ARM kernel person? My ARM knowledge is limited at best, I don't think Dave has much at all, and I'm not sure about Justin but I'd be surprised if he was a closet ARM expert. Do you have people in mind to handle the HW specific issues that pop up so we can assign bugs to them?
In fact we do. We have a Red Hat FY13 hire for our ARM team who will represent us in upstream kernel work. That's mostly for v8 but I would expect (given the person we have in mind) that they will also handle 32-bit. I'll be able to share names soon. Beyond that, we have others in the team internally and in the community who can help debug issues.
We don't intend for this to be a trainwreck. If it's not ready, it won't be PA, period. But we want to know how flexible you guys are willing to be as we figure this out. If you want to wait until we have total parity, single zimage, and we're just like x86, that is good to know (and discuss) right now :)
Personally, I don't think it has to be one vmlinux + 18 FDTs to be suitable. A smaller set of board specific kernels is probably doable, depending on some of the factors mentioned above. I see that as similar to how we built e.g. ppc, ppc-smp, ppc64, ppc64-kdump. Nobody particularly _liked_ doing that but we lived.
Right. To me, it's about having a plan to kill off the separate flavors. As long as we don't do what $other_distro does and consider it hunky dory having a bazillion ARM kernels, and go in aggressively wanting to cut it down to one in the end, I think we could live. But that's my opinion, and I'm not you guys.
Thanks for the feedback, you rock!
Jon.
On Thu, Mar 22, 2012 at 11:01:55AM -0400, Jon Masters wrote:
I'm not sure building only the qemu kernel is a great way to go either, both from a ARM _and_ kernel perspective. It might be better to build that + 1 board from each armv5tel and armv7hl.
That would be fine too. I mean, we could compromise and choose a set of qemu+obvious board as the minimum and then turn on the rest at certain points in the cycle. Whatever works for you guys.
What might be useful is documenting how you plan to monitor how many people are actually using the various images, and at what point you decide to cut off life support. If we're spending an hour building an image that just a dozen people are downloading, it's not really a good use of resources.
This isn't an ARM specific thing btw, we'd *love* to kill off support for < 686-PAE, but sadly there are more than a few dozen users there.
Yes. However, waiting for a day for your build to complete only to have it be canceled because some ARM variant died is one of the larger concerns.
Absolutely. Clearly, a day is nuts. But it sounds like a few hours for ARM isn't as big a deal. Would 2 or 4 hours be your upper limit? etc.?
4 hours would really be the absolute maximum. A security issue coming in on a Friday afternoon doesn't mean someone has to give up their weekend working on builds/updates. (Yes, this is Fedora, no guarantees blahblah, but we try our best to turnaround CVEs asap).
Another reason is that for rawhide, we frequently get a build a day done. (sometimes multiple). To catch the patches Linus has pushed during the day, it's not uncommon for one of us to do a build at the end of the work day to end up in the next days rawhide. If it takes too long, we miss the compose window.
Can you elaborate on disruption? We already spend time mucking around with ARM configs during rebase (not intelligently mind you), and I don't think we grumbled too much about it. Aside from more bugs, is there something else you were thinking?
Well, it's really whether you're ok with that kind of thing sometimes. We can't promise to be perfect, and in fact we can promise crap like that will come up...so it's really whether you can live with that.
It would be good to at least have people going over our guesses at config options etc. Our educated guesses may not be the best choices, and right now they seem to be sort of ignored..
$ git annotate config-arm-generic | grep FIXME ae6da83b (Dave Jones 2011-12-30 22:23:07 -0500 200)# FIXME: Guesses, need checking
On the subject of configs, I'd really like to see arm taking advantage of the config-generic infrastructure. Having to add options in multiple places on a rebase sucks.
At that point, I'd like to review some of the config decisions arm has made which differ from every other platform.
Dave
kernel@lists.fedoraproject.org
-
Dave Jones -
Gerd Hoffmann -
Jon Masters -
Josh Boyer