Hi all,
I would like to start working on the replacement application for PDC in Fedora. So far I have been mostly looking at the technical side of the application and how it will be setup in our infrastructure.
Below is a list of the design decisions I am looking to take. I would really value some feedback on this in order to see if this is realistic or not.
# Technical stack * Python 3 (just making it obvious) * Django 1.11 LTS (supported until at least April 2020) [0] * DRF 3.8.2 (latest version) Using Django and DRF (Django REST Framework) should allow us to reuse some of PDC code.
# Application architecture * 1 backend service used to update the PDC database (currently pdc-updater [1]) * 1 frontend service serving the API HTTP endpoint and the HTML documentation of these endpoint. * 1 database
# Deployment architecture * Backend service runs in Openshift (need access to the fedora-messaging bus) * Frontend service runs in Openshift * Database runs in on Fedora Infrastructure database hosts.
# Development Workflow * I would like to have the possibility for these services to be automatically deployed on our Openshift instances (stg and prod) using the Github build trigger [2] available in Openshift. An idea to achieve this is to use Openshift S2I (source-to-image) [3] feature and use PyPi to install the application dependencies. In order to control which dependencies we are using and pulling from PyPi we could use devpi [4] as a caching proxy.
Thanks, Clément
[0] - https://www.djangoproject.com/download/ [1] - https://github.com/fedora-infra/pdc-updater [2] - https://docs.openshift.com/online/dev_guide/builds/triggering_builds.html#gi... [3] - https://github.com/openshift/source-to-image [4] - https://devpi.net/docs/devpi/devpi/latest/%2Bd/index.html
On 09/11/2018 11:48 AM, Clement Verna wrote:
Hi all,
I would like to start working on the replacement application for PDC in Fedora. So far I have been mostly looking at the technical side of the application and how it will be setup in our infrastructure.
Below is a list of the design decisions I am looking to take. I would really value some feedback on this in order to see if this is realistic or not.
# Technical stack * Python 3 (just making it obvious) * Django 1.11 LTS (supported until at least April 2020) [0] * DRF 3.8.2 (latest version) Using Django and DRF (Django REST Framework) should allow us to reuse some of PDC code.
# Application architecture * 1 backend service used to update the PDC database (currently pdc-updater [1]) * 1 frontend service serving the API HTTP endpoint and the HTML documentation of these endpoint. * 1 database
# Deployment architecture * Backend service runs in Openshift (need access to the fedora-messaging bus) * Frontend service runs in Openshift * Database runs in on Fedora Infrastructure database hosts.
# Development Workflow * I would like to have the possibility for these services to be automatically deployed on our Openshift instances (stg and prod) using the Github build trigger [2] available in Openshift. An idea to achieve this is to use Openshift S2I (source-to-image) [3] feature and use PyPi to install the application dependencies. In order to control which dependencies we are using and pulling from PyPi we could use devpi [4] as a caching proxy.
This all sounds perfectly great to me... everything should be there for this with one exception: We really want to have some check in place for s2i so that it checks license, so we don't accidentally push out something thats not under a open source license. This doesn't need to be a blocker, but it would be great to get in place soon.
kevin
everything should be there for this with one exception: We really want to have some check in place for s2i so that it checks license, so we don't accidentally push out something thats not under a open source license. This doesn't need to be a blocker, but it would be great to get in place soon.
I could see that as an integration test in PDC, or have a regular (or evented) job on the devpi host that would check the licences of all cached (and thus requested) packages. The downside of doing it on devpi is that we won't know directly which app has requested the nonfree package. Since all dependencies are already available locally and the license is in the package metadata (PKG-INFO file), a script running in the integration testsuite wouldn't even need internet access.
I can write a POC if you want.
A.
On Wed, 12 Sep 2018 at 09:40, Aurelien Bompard abompard@fedoraproject.org wrote:
everything should be there for this with one exception: We really want to have some check in place for s2i so that it checks license, so we don't accidentally push out something thats not under a open source license. This doesn't need to be a blocker, but it would be great to get in place soon.
I could see that as an integration test in PDC, or have a regular (or evented) job on the devpi host that would check the licences of all cached (and thus requested) packages.
I like the CI test idea, a little bit like when we tests that the code base is pep8 compliant or the test coverage in above 90%. There are a couple of python packages that could be useful to help with that [0] [1].
[0] https://github.com/dhatim/python-license-check [1] https://github.com/raimon49/pip-licenses
The downside of doing it on devpi is that we won't know directly which app has requested the nonfree package. Since all dependencies are already available locally and the license is in the package metadata (PKG-INFO file), a script running in the integration testsuite wouldn't even need internet access.
I can write a POC if you want.
A. _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
I like the CI test idea, a little bit like when we tests that the code base is pep8 compliant or the test coverage in above 90%. There are a couple of python packages that could be useful to help with that [0] [1].
[0] https://github.com/dhatim/python-license-check [1] https://github.com/raimon49/pip-licenses
Pretty cool, I've implemented python-license-check for fedora-messaging, it's quite trivial: https://github.com/fedora-infra/fedora-messaging/pull/68
A.
On 09/12/2018 03:44 AM, Aurelien Bompard wrote:
I like the CI test idea, a little bit like when we tests that the code base is pep8 compliant or the test coverage in above 90%. There are a couple of python packages that could be useful to help with that [0] [1].
[0] https://github.com/dhatim/python-license-check [1] https://github.com/raimon49/pip-licenses
Pretty cool, I've implemented python-license-check for fedora-messaging, it's quite trivial: https://github.com/fedora-infra/fedora-messaging/pull/68
Nice! I like this idea too... it means the issue will be found closer to the upstream app, so it can be fixed quicker than if it got merged, released, and pushed into openshift.
We just need to make sure all the apps we deploy that use pip/s2i use this method.
kevin
On 09/12/2018 04:01 AM, Clement Verna wrote:
This looks useful, thanks for sharing!
Hello,
could we use this?
https://github.com/PostgREST/postgrest
we could trim down PDC to just db + REST API (+ git backend syncing through hooks?)
It's written in Haskell (!!) and it looks really like a useful piece of technology.
We could drop Django completely and have git for any write changes, which means we will automatically get:
- version control - acls - public interface
for free without any future maintenance.
Only the writeable/interesting parts of PDC would be in DistGit.
I don't know what will be stored in PDC at this point but I think just db + REST + git would be a suitable stack for a high-level release-engineering-like settings.
clime
P.S. I would personally be happy to help with this setup.
On Wed, Sep 12, 2018 at 8:57 PM Randy Barlow bowlofeggs@fedoraproject.org wrote:
On 09/12/2018 04:01 AM, Clement Verna wrote:
This looks useful, thanks for sharing!
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Mon, 24 Sep 2018 at 11:30, Michal Novotny clime@redhat.com wrote:
Hello,
could we use this?
While I don't have a strong preference for one or the other, the main reason we decided to use DRF is to be able to reuse some of code from PDC ( https://github.com/product-definition-center/product-definition-center). We are currently working on the releases endpoint ( https://github.com/fedora-infra/fpdc) and to be fair we did not reuse to much code so far, but I expect that it will be the case when working on components and composes endpoint.
we could trim down PDC to just db + REST API (+ git backend syncing through hooks?)
It's written in Haskell (!!) and it looks really like a useful piece of technology.
We could drop Django completely and have git for any write changes, which means we will automatically get:
- version control
- acls
- public interface
for free without any future maintenance.
Only the writeable/interesting parts of PDC would be in DistGit.
I don't know what will be stored in PDC at this point but I think just db + REST + git would be a suitable stack for a high-level release-engineering-like settings.
clime
P.S. I would personally be happy to help with this setup.
On Wed, Sep 12, 2018 at 8:57 PM Randy Barlow bowlofeggs@fedoraproject.org wrote:
On 09/12/2018 04:01 AM, Clement Verna wrote:
This looks useful, thanks for sharing!
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to
infrastructure-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Mon, Sep 24, 2018 at 2:43 PM Clement Verna cverna@fedoraproject.org wrote:
On Mon, 24 Sep 2018 at 11:30, Michal Novotny clime@redhat.com wrote:
Hello,
could we use this?
While I don't have a strong preference for one or the other, the main reason we decided to use DRF is to be able to reuse some of code from PDC (https://github.com/product-definition-center/product-definition-center). We are currently working on the releases endpoint (https://github.com/fedora-infra/fpdc) and to be fair we did not reuse to much code so far, but I expect that it will be the case when working on components and composes endpoint.
I guess it mainly depends on the people doing the work (that means you and the rest of the pack).
clime
we could trim down PDC to just db + REST API (+ git backend syncing through hooks?)
It's written in Haskell (!!) and it looks really like a useful piece of technology.
We could drop Django completely and have git for any write changes, which means we will automatically get:
- version control
- acls
- public interface
for free without any future maintenance.
Only the writeable/interesting parts of PDC would be in DistGit.
I don't know what will be stored in PDC at this point but I think just db + REST + git would be a suitable stack for a high-level release-engineering-like settings.
clime
P.S. I would personally be happy to help with this setup.
On Wed, Sep 12, 2018 at 8:57 PM Randy Barlow bowlofeggs@fedoraproject.org wrote:
On 09/12/2018 04:01 AM, Clement Verna wrote:
This looks useful, thanks for sharing!
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
infrastructure@lists.fedoraproject.org
-
Aurelien Bompard -
Clement Verna -
Kevin Fenzi -
Michal Novotny -
Randy Barlow