It's been eight years since koji 1.0. In that time Koji has grown a lot, but always incrementally and with great care to avoid breaking compatibility. Over the years, we've found numerous things that we wanted to add or change, but that we dismissed as too big, too complicated, or too invasive.
Bumping the major release number gives use the freedom to shake things up a bit. Koji 2.0 is about making major changes, otherwise it would just be koji 1.13.
Koji 2.0 will take some time. We will maintain Koji 1.x until 2.0 is sufficiently stable. Some features (e.g. content generators) will also appear in 1.x.
The list below is probably not complete, it is ambitious, and it is certainly open to discussion. If you are a Koji user, or otherwise invested in Koji, then you are encouraged to join in. I expect there will be a lot to say, so I've created a new mailing list devoted specifically to Koji development.
https://lists.fedorahosted.org/mailman/listinfo/koji-devel
Also, apologies for the tense summary listing. I can certain expound on any of these as necessary. I hope this suffices to start some conversation.
= High level goals =
• better documentation • more community involvement • refactor/modernize code • more modular design ∘ content generators ∘ broader, better plugin framework • better support for different types of build processes • better support for for different types build output • make hard-wired restrictions more configurable • easier to deploy • better qa process • better release process
= Highlights/Major changes =
• python3 support ∘ the bulk of the code will target python 2.6 + python-six ∘ we'll create a basic client lib for older systems (e.g rhel5 clients/builders) • drop xmlrpc in favor a json based rpc • build namespaces ∘ allow for use cases that require multiple builds of the same NVR (or NVRA) • refactor task scheduling • extend content generator support ∘ content generators will land in 1.x fairly soon, but in 2.0 they will be more integral ∘ refactor kojid to use content generator calls ∘ (possibly) tighter integration in the db • unify handling of rpms with other build types in the db ∘ e.g. unify rpminfo and archiveinfo tables • support different ways of building (possibly via content generators) • utilize jsonb fields in postgres • modular auth ∘ make it easier to add new auth mechanisms ∘ support openid auth • improve plugins ∘ make the plugin framework cleaner and more pythonic ∘ support plugins in cli and web ui • improve/update web ui ∘ drop cheetah templates in favor of python-jinja2 ∘ more parity with cli ∘ history browsing ∘ landing page search or history ∘ support plugins • change how tag pkglists (and blocking) work • refactor package ownership • refactor uploads • more flexible gc
= Yet more changes =
• store all task requests by named args ∘ (for ease of inspection) • get rid of tagBuild tasks • drop odd event refererences in favor of timestamps • streamlined cli options • marker files for many things on disk • more history data • drop modpython support • policy code ∘ more robust syntax ‣ test negation ‣ OR ‣ parentheses ‣ quoted strings ∘ multiple result policies (non-terminal actions) ∘ all-matches policy (needed for scheduler?) ∘ break action (breaks out of nesting) ∘ stop action (halts processing of an all-matches policy)
Subject: Koji 2.0 planning
That sounds great! One thing I'd love to see (still being quite new to koji) is improved consistency in the CLI. It's not bad now, but being less than perfect requires that I almost always seek help.
For example, for every NOUN provide: add-NOUN, edit-NOUN, list-NOUN (and/or show-NOUN), remove-NOUN. If I were implementing this, I would subclass all from base classes and in cases where something cannot be permitted, e.g., removal for database constraints/integrity, have the base class provide a default behavior saying something like "This is not possible due; not because of authorization but by design." Such extra commands might seem like clutter, but they also would help in being somewhat self-documenting. "koji help" could be shortened drastically by mentioning something like:
Maintenance Commands: {add,edit,list,show,remove}-NOUN where NOUN can be any of: group, pkg, user....
Just my $0.02. Regardless, I look forward to the progress. -- John Florian
On 06/02/2015 02:18 PM, John Florian wrote:
Subject: Koji 2.0 planning
That sounds great! One thing I'd love to see (still being quite new to koji) is improved consistency in the CLI. It's not bad now, but being less than perfect requires that I almost always seek help.
For example, for every NOUN provide: add-NOUN, edit-NOUN, list-NOUN (and/or show-NOUN), remove-NOUN. If I were implementing this, I would subclass all from base classes and in cases where something cannot be permitted, e.g., removal for database constraints/integrity, have the base class provide a default behavior saying something like "This is not possible due; not because of authorization but by design." Such extra commands might seem like clutter, but they also would help in being somewhat self-documenting. "koji help" could be shortened drastically by mentioning something like:
Maintenance Commands: {add,edit,list,show,remove}-NOUN where NOUN can be any of: group, pkg, user....
Good idea. We could certainly use more consistency there. Similarly with the api calls.
Just my $0.02. Regardless, I look forward to the progress.
<snip great things>
Here's a list of things I have, some of them may be impractical, but I thought I would thow them out there:
One thing I always wanted, but was a dream was configurable and dynamic builders. By that I mean a way you could configure:
- create new builder - destroy idle builder - max builders to spawn - max idle builders - max builders at startup - builder type per channel (so you could have larger/smaller/non virt) etc.
And then have libvirt and nova and ec2 backends, so you could have a koji hub that just spins up build vm's on virthosts or cloud instances or amazon for it's needs. That would allow you to be better about using resources and not need to have a bunch of dedicated builders sitting around doing not much all the time and also be able to add short term capacity for things like mass rebuilds. You could also perhaps set things to make a new builder for every build.
Another thing that would be great IMHO would be some kind of reorg of the database and packages in the filesystem so it's possible/easy to archive things. Right now you have to have everything mixed under the same mount, if there was some way to mark things as archived you could move them to slower/different storage.
More metrics built in would be nice. Some kind of report on hits and downloads per build, etc.
kevin
On 06/02/2015 11:43 AM, Mike McLean wrote:
It's been eight years since koji 1.0. In that time Koji has grown a lot, but always incrementally and with great care to avoid breaking compatibility. Over the years, we've found numerous things that we wanted to add or change, but that we dismissed as too big, too complicated, or too invasive.
Bumping the major release number gives use the freedom to shake things up a bit. Koji 2.0 is about making major changes, otherwise it would just be koji 1.13.
Koji 2.0 will take some time. We will maintain Koji 1.x until 2.0 is sufficiently stable. Some features (e.g. content generators) will also appear in 1.x.
The list below is probably not complete, it is ambitious, and it is certainly open to discussion. If you are a Koji user, or otherwise invested in Koji, then you are encouraged to join in. I expect there will be a lot to say, so I've created a new mailing list devoted specifically to Koji development.
https://lists.fedorahosted.org/mailman/listinfo/koji-devel
Also, apologies for the tense summary listing. I can certain expound on any of these as necessary. I hope this suffices to start some conversation.
= High level goals =
• better documentation • more community involvement • refactor/modernize code • more modular design ∘ content generators ∘ broader, better plugin framework • better support for different types of build processes • better support for for different types build output • make hard-wired restrictions more configurable • easier to deploy • better qa process • better release process
better tests and improve the test coverage. better reports.
= Highlights/Major changes =
• python3 support ∘ the bulk of the code will target python 2.6 + python-six ∘ we'll create a basic client lib for older systems (e.g rhel5 clients/builders) • drop xmlrpc in favor a json based rpc • build namespaces ∘ allow for use cases that require multiple builds of the same NVR (or NVRA) • refactor task scheduling • extend content generator support ∘ content generators will land in 1.x fairly soon, but in 2.0 they will be more integral ∘ refactor kojid to use content generator calls ∘ (possibly) tighter integration in the db • unify handling of rpms with other build types in the db ∘ e.g. unify rpminfo and archiveinfo tables • support different ways of building (possibly via content generators) • utilize jsonb fields in postgres • modular auth ∘ make it easier to add new auth mechanisms ∘ support openid auth • improve plugins ∘ make the plugin framework cleaner and more pythonic ∘ support plugins in cli and web ui • improve/update web ui ∘ drop cheetah templates in favor of python-jinja2 ∘ more parity with cli ∘ history browsing ∘ landing page search or history ∘ support plugins • change how tag pkglists (and blocking) work • refactor package ownership • refactor uploads • more flexible gc
refactor db access api, introduce an ORM (SQLAlchemy for example) to simplify the code writing too much raw DB-API invocations and SQLs. However, whatever ORM is adopted, it must be careful to use, or it may hurt the performance easily.
= Yet more changes =
• store all task requests by named args ∘ (for ease of inspection) • get rid of tagBuild tasks • drop odd event refererences in favor of timestamps • streamlined cli options • marker files for many things on disk • more history data • drop modpython support • policy code ∘ more robust syntax ‣ test negation ‣ OR ‣ parentheses ‣ quoted strings ∘ multiple result policies (non-terminal actions) ∘ all-matches policy (needed for scheduler?) ∘ break action (breaks out of nesting) ∘ stop action (halts processing of an all-matches policy)
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
Regards, Chenxiong Qi
On Tue, Jun 2, 2015 at 4:43 AM, Mike McLean mikem@redhat.com wrote:
It's been eight years since koji 1.0. In that time Koji has grown a lot, but always incrementally and with great care to avoid breaking compatibility. Over the years, we've found numerous things that we wanted to add or change, but that we dismissed as too big, too complicated, or too invasive.
Bumping the major release number gives use the freedom to shake things up a bit. Koji 2.0 is about making major changes, otherwise it would just be koji 1.13.
Koji 2.0 will take some time. We will maintain Koji 1.x until 2.0 is sufficiently stable. Some features (e.g. content generators) will also appear in 1.x.
The list below is probably not complete, it is ambitious, and it is certainly open to discussion. If you are a Koji user, or otherwise invested in Koji, then you are encouraged to join in. I expect there will be a lot to say, so I've created a new mailing list devoted specifically to Koji development.
https://lists.fedorahosted.org/mailman/listinfo/koji-devel
Also, apologies for the tense summary listing. I can certain expound on any of these as necessary. I hope this suffices to start some conversation.
= High level goals =
• better documentation • more community involvement • refactor/modernize code • more modular design ∘ content generators ∘ broader, better plugin framework • better support for different types of build processes • better support for for different types build output • make hard-wired restrictions more configurable • easier to deploy • better qa process • better release process
= Highlights/Major changes =
• python3 support ∘ the bulk of the code will target python 2.6 + python-six ∘ we'll create a basic client lib for older systems (e.g rhel5 clients/builders) • drop xmlrpc in favor a json based rpc • build namespaces ∘ allow for use cases that require multiple builds of the same NVR (or NVRA) • refactor task scheduling • extend content generator support ∘ content generators will land in 1.x fairly soon, but in 2.0 they will be more integral ∘ refactor kojid to use content generator calls ∘ (possibly) tighter integration in the db • unify handling of rpms with other build types in the db ∘ e.g. unify rpminfo and archiveinfo tables • support different ways of building (possibly via content generators) • utilize jsonb fields in postgres • modular auth ∘ make it easier to add new auth mechanisms ∘ support openid auth
I'd like to add to this more granular access control, groups etc. It would be great if every task could be assigned groups and each group could be given things like Read, Execute, Execute scratch, Delete, Tag etc style of ACLs.
One of the requests we get on occasion is the ability for someone to run image scratch builds, at the moment to do that you need to be admin as there's no ability for non admin to do image builds, but we'd likely want enough granularity to differentiate between scratch versus official builds.
Peter
On Jun 07 14:44, Peter Robinson wrote:
On Tue, Jun 2, 2015 at 4:43 AM, Mike McLean mikem@redhat.com wrote:
It's been eight years since koji 1.0. In that time Koji has grown a lot, but always incrementally and with great care to avoid breaking compatibility. Over the years, we've found numerous things that we wanted to add or change, but that we dismissed as too big, too complicated, or too invasive.
Bumping the major release number gives use the freedom to shake things up a bit. Koji 2.0 is about making major changes, otherwise it would just be koji 1.13.
Koji 2.0 will take some time. We will maintain Koji 1.x until 2.0 is sufficiently stable. Some features (e.g. content generators) will also appear in 1.x.
The list below is probably not complete, it is ambitious, and it is certainly open to discussion. If you are a Koji user, or otherwise invested in Koji, then you are encouraged to join in. I expect there will be a lot to say, so I've created a new mailing list devoted specifically to Koji development.
https://lists.fedorahosted.org/mailman/listinfo/koji-devel
Also, apologies for the tense summary listing. I can certain expound on any of these as necessary. I hope this suffices to start some conversation.
= High level goals =
• better documentation • more community involvement • refactor/modernize code • more modular design ∘ content generators ∘ broader, better plugin framework • better support for different types of build processes • better support for for different types build output • make hard-wired restrictions more configurable • easier to deploy • better qa process • better release process
= Highlights/Major changes =
• python3 support ∘ the bulk of the code will target python 2.6 + python-six ∘ we'll create a basic client lib for older systems (e.g rhel5 clients/builders) • drop xmlrpc in favor a json based rpc • build namespaces ∘ allow for use cases that require multiple builds of the same NVR (or NVRA) • refactor task scheduling • extend content generator support ∘ content generators will land in 1.x fairly soon, but in 2.0 they will be more integral ∘ refactor kojid to use content generator calls ∘ (possibly) tighter integration in the db • unify handling of rpms with other build types in the db ∘ e.g. unify rpminfo and archiveinfo tables • support different ways of building (possibly via content generators) • utilize jsonb fields in postgres • modular auth ∘ make it easier to add new auth mechanisms ∘ support openid auth
I'd like to add to this more granular access control, groups etc. It would be great if every task could be assigned groups and each group could be given things like Read, Execute, Execute scratch, Delete, Tag etc style of ACLs.
+1
One of the requests we get on occasion is the ability for someone to run image scratch builds, at the moment to do that you need to be admin as there's no ability for non admin to do image builds, but we'd likely want enough granularity to differentiate between scratch versus official builds.
Read/NoRead ACLs would nicely enable people doing builds of sources under embargo.
Peter
--Brian
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 06/01/2015 11:43 PM, Mike McLean wrote:
It's been eight years since koji 1.0. In that time Koji has grown a lot, but always incrementally and with great care to avoid breaking compatibility. Over the years, we've found numerous things that we wanted to add or change, but that we dismissed as too big, too complicated, or too invasive.
Bumping the major release number gives use the freedom to shake things up a bit. Koji 2.0 is about making major changes, otherwise it would just be koji 1.13.
Koji 2.0 will take some time. We will maintain Koji 1.x until 2.0 is sufficiently stable. Some features (e.g. content generators) will also appear in 1.x.
The list below is probably not complete, it is ambitious, and it is certainly open to discussion. If you are a Koji user, or otherwise invested in Koji, then you are encouraged to join in. I expect there will be a lot to say, so I've created a new mailing list devoted specifically to Koji development.
https://lists.fedorahosted.org/mailman/listinfo/koji-devel
Also, apologies for the tense summary listing. I can certain expound on any of these as necessary. I hope this suffices to start some conversation.
= High level goals =
• better documentation • more community involvement • refactor/modernize code • more modular design ∘ content generators ∘ broader, better plugin framework • better support for different types of build processes • better support for for different types build output • make hard-wired restrictions more configurable • easier to deploy • better qa process • better release process
= Highlights/Major changes =
• python3 support ∘ the bulk of the code will target python 2.6 + python-six ∘ we'll create a basic client lib for older systems (e.g rhel5 clients/builders) • drop xmlrpc in favor a json based rpc • build namespaces ∘ allow for use cases that require multiple builds of the same NVR (or NVRA) • refactor task scheduling • extend content generator support ∘ content generators will land in 1.x fairly soon, but in 2.0 they will be more integral ∘ refactor kojid to use content generator calls ∘ (possibly) tighter integration in the db • unify handling of rpms with other build types in the db ∘ e.g. unify rpminfo and archiveinfo tables • support different ways of building (possibly via content generators) • utilize jsonb fields in postgres • modular auth ∘ make it easier to add new auth mechanisms ∘ support openid auth • improve plugins ∘ make the plugin framework cleaner and more pythonic ∘ support plugins in cli and web ui • improve/update web ui ∘ drop cheetah templates in favor of python-jinja2 ∘ more parity with cli ∘ history browsing ∘ landing page search or history ∘ support plugins • change how tag pkglists (and blocking) work • refactor package ownership • refactor uploads • more flexible gc
= Yet more changes =
• store all task requests by named args ∘ (for ease of inspection) • get rid of tagBuild tasks • drop odd event refererences in favor of timestamps • streamlined cli options • marker files for many things on disk • more history data • drop modpython support • policy code ∘ more robust syntax ‣ test negation ‣ OR ‣ parentheses ‣ quoted strings ∘ multiple result policies (non-terminal actions) ∘ all-matches policy (needed for scheduler?) ∘ break action (breaks out of nesting) ∘ stop action (halts processing of an all-matches policy)
I would like to see 2 features in Koji: - - know how to build installation media - - managed signed repositories of RPM content
If those existed, and we overloaded the use of package groups with comps data, that would virtually eliminate the need for mash, pungi, distill, and puddle. A compose would become a series of brew commands to write out signed repos and installation media.
Is that an oversimplification, or a real goal?
- - Jay
Any thoughts on pulp for an external repo manager rather than doing it in koji?
Pat
On 06/01/2015 10:43 PM, Mike McLean wrote:
It's been eight years since koji 1.0. In that time Koji has grown a lot, but always incrementally and with great care to avoid breaking compatibility. Over the years, we've found numerous things that we wanted to add or change, but that we dismissed as too big, too complicated, or too invasive.
Bumping the major release number gives use the freedom to shake things up a bit. Koji 2.0 is about making major changes, otherwise it would just be koji 1.13.
Koji 2.0 will take some time. We will maintain Koji 1.x until 2.0 is sufficiently stable. Some features (e.g. content generators) will also appear in 1.x.
The list below is probably not complete, it is ambitious, and it is certainly open to discussion. If you are a Koji user, or otherwise invested in Koji, then you are encouraged to join in. I expect there will be a lot to say, so I've created a new mailing list devoted specifically to Koji development.
https://lists.fedorahosted.org/mailman/listinfo/koji-devel
Also, apologies for the tense summary listing. I can certain expound on any of these as necessary. I hope this suffices to start some conversation.
= High level goals =
• better documentation • more community involvement • refactor/modernize code • more modular design ∘ content generators ∘ broader, better plugin framework • better support for different types of build processes • better support for for different types build output • make hard-wired restrictions more configurable • easier to deploy • better qa process • better release process
= Highlights/Major changes =
• python3 support ∘ the bulk of the code will target python 2.6 + python-six ∘ we'll create a basic client lib for older systems (e.g rhel5 clients/builders) • drop xmlrpc in favor a json based rpc • build namespaces ∘ allow for use cases that require multiple builds of the same NVR (or NVRA) • refactor task scheduling • extend content generator support ∘ content generators will land in 1.x fairly soon, but in 2.0 they will be more integral ∘ refactor kojid to use content generator calls ∘ (possibly) tighter integration in the db • unify handling of rpms with other build types in the db ∘ e.g. unify rpminfo and archiveinfo tables • support different ways of building (possibly via content generators) • utilize jsonb fields in postgres • modular auth ∘ make it easier to add new auth mechanisms ∘ support openid auth • improve plugins ∘ make the plugin framework cleaner and more pythonic ∘ support plugins in cli and web ui • improve/update web ui ∘ drop cheetah templates in favor of python-jinja2 ∘ more parity with cli ∘ history browsing ∘ landing page search or history ∘ support plugins • change how tag pkglists (and blocking) work • refactor package ownership • refactor uploads • more flexible gc
= Yet more changes =
• store all task requests by named args ∘ (for ease of inspection) • get rid of tagBuild tasks • drop odd event refererences in favor of timestamps • streamlined cli options • marker files for many things on disk • more history data • drop modpython support • policy code ∘ more robust syntax ‣ test negation ‣ OR ‣ parentheses ‣ quoted strings ∘ multiple result policies (non-terminal actions) ∘ all-matches policy (needed for scheduler?) ∘ break action (breaks out of nesting) ∘ stop action (halts processing of an all-matches policy)
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
buildsys@lists.fedoraproject.org
-
Brian Stinson -
Chenxiong Qi -
Jay Greguske -
John Florian -
Kevin Fenzi -
Mike McLean -
Pat Riehecky -
Peter Robinson