Hi, I would like to ask if it would be possible to allocate some storage space that would be publically available (eq. fedorapeople.org, or similar) we could use store logs from our CI so that external contributors can access them.
Why is this needed ? Our CI runs in internal vm hosts and internal open stack, both if which is not accessbile from the public internet, has no public IP or domain. At the same time we would like the make te test results available, so that for example an external contributor can see the results of tests we run on their pull request.
What tests does the Anaconda CI run ? - unit tests & Pylint we run on all pull requests for Anaconda and related projects - periodically run kickstart tests[0], a battery of 130+ integration test, where each test is basically a full kickstart-configured installation run
What kind of environment would be needed ? Basically just a simple file/static html storage would be needed, where we can SCP the log files to. The general idea is that tests results would be uploaded to this space whem the respective tests finish & there would a periodic cleanup task that would keep last N logs & remove old logs.
What ammount of storage space would be needed for this ? While the unit tests & pylint runs we do on Pull requests don't produce that much data, the kickstart test suite is in a different class.
Each of the 130+ tests is a full installation run, which produces a sizeable ammount of data as we gather a substantial ammount of logs from each test (anaconda logs, dnf logs, lvm & storage logs). This data is important to debug all the various issues that can happen in what is essentially a full instrumented installation run.
Our measurements have shown that each full kickstart test run (~130 tests) produces ~800 MB of logs. Therefore we would like to request 16 GB of space to be available so that we can store ~10 last tests results from kickstart test, a reasonable number of PR test results & have some room to spare.
Would this be possible ? Requesting 16 GB might seem like a lot but I'm afraid we can't reasonably reduce the space requirements for logs from kickstart tests runs without at the same time also making test runs harder to debug.
Looking for your feedback & don't be afraid to ask any questions about this proposal. :)
Best Wishes Martin Kolman
On 06/20/2018 06:15 AM, Martin Kolman wrote:
Hi, I would like to ask if it would be possible to allocate some storage space that would be publically available (eq. fedorapeople.org, or similar) we could use store logs from our CI so that external contributors can access them.
Sure, it should be.
Why is this needed ? Our CI runs in internal vm hosts and internal open stack, both if which is not accessbile from the public internet, has no public IP or domain. At the same time we would like the make te test results available, so that for example an external contributor can see the results of tests we run on their pull request.
What tests does the Anaconda CI run ?
- unit tests & Pylint we run on all pull requests for Anaconda and related projects
- periodically run kickstart tests[0], a battery of 130+ integration test, where each test is basically a full kickstart-configured installation run
What kind of environment would be needed ? Basically just a simple file/static html storage would be needed, where we can SCP the log files to. The general idea is that tests results would be uploaded to this space whem the respective tests finish & there would a periodic cleanup task that would keep last N logs & remove old logs.
What ammount of storage space would be needed for this ? While the unit tests & pylint runs we do on Pull requests don't produce that much data, the kickstart test suite is in a different class.
Each of the 130+ tests is a full installation run, which produces a sizeable ammount of data as we gather a substantial ammount of logs from each test (anaconda logs, dnf logs, lvm & storage logs). This data is important to debug all the various issues that can happen in what is essentially a full instrumented installation run.
Our measurements have shown that each full kickstart test run (~130 tests) produces ~800 MB of logs. Therefore we would like to request 16 GB of space to be available so that we can store ~10 last tests results from kickstart test, a reasonable number of PR test results & have some room to spare.
Would this be possible ? Requesting 16 GB might seem like a lot but I'm afraid we can't reasonably reduce the space requirements for logs from kickstart tests runs without at the same time also making test runs harder to debug.
Looking for your feedback & don't be afraid to ask any questions about this proposal. :)
Yep. we can do this (at least) two ways:
1. We can use a fedorapeople group space. Indeed, there is already an anaconda space there: https://fedorapeople.org/groups/anaconda/
PROs: already there, has ipv6, anyone in fas group can upload. CONS: not backed up or mirrored, not in phx2, so might be slower to upload depending on where you are uploading from.
2. an alt space under dl.fedoraproject.org/pub/alt PROs: is on netapp space so it gets snapshotted and mirrored. CONS: no ipv6, needs setup, more restricted what accounts can upload to it.
Would you like to just try the fedorapeople space first? It looks like anyone in the 'gitanaconda' group should have write to the /project/anaconda/ space there. If you prefer a new group we can set that up too.
Also it looks like there's a alt anaconda space: https://dl.fedoraproject.org/pub/alt/anaconda/ but only bcl is setup to write there. we could add another person or two there, but wouldn't want to add big groups or automated stuff.
kevin
On Thu, 2018-06-21 at 08:36 -0700, Kevin Fenzi wrote:
On 06/20/2018 06:15 AM, Martin Kolman wrote:
Hi, I would like to ask if it would be possible to allocate some storage space that would be publically available (eq. fedorapeople.org, or similar) we could use store logs from our CI so that external contributors can access them.
Sure, it should be.
Nice! Thanks in advance! :)
Why is this needed ? Our CI runs in internal vm hosts and internal open stack, both if which is not accessbile from the public internet, has no public IP or domain. At the same time we would like the make te test results available, so that for example an external contributor can see the results of tests we run on their pull request.
What tests does the Anaconda CI run ?
- unit tests & Pylint we run on all pull requests for Anaconda and related projects
- periodically run kickstart tests[0], a battery of 130+ integration test, where each test is basically a full kickstart-configured installation run
What kind of environment would be needed ? Basically just a simple file/static html storage would be needed, where we can SCP the log files to. The general idea is that tests results would be uploaded to this space whem the respective tests finish & there would a periodic cleanup task that would keep last N logs & remove old logs.
What ammount of storage space would be needed for this ? While the unit tests & pylint runs we do on Pull requests don't produce that much data, the kickstart test suite is in a different class.
Each of the 130+ tests is a full installation run, which produces a sizeable ammount of data as we gather a substantial ammount of logs from each test (anaconda logs, dnf logs, lvm & storage logs). This data is important to debug all the various issues that can happen in what is essentially a full instrumented installation run.
Our measurements have shown that each full kickstart test run (~130 tests) produces ~800 MB of logs. Therefore we would like to request 16 GB of space to be available so that we can store ~10 last tests results from kickstart test, a reasonable number of PR test results & have some room to spare.
Would this be possible ? Requesting 16 GB might seem like a lot but I'm afraid we can't reasonably reduce the space requirements for logs from kickstart tests runs without at the same time also making test runs harder to debug.
Looking for your feedback & don't be afraid to ask any questions about this proposal. :)
Yep. we can do this (at least) two ways:
- We can use a fedorapeople group space. Indeed, there is already an
anaconda space there: https://fedorapeople.org/groups/anaconda/
PROs: already there, has ipv6, anyone in fas group can upload. CONS: not backed up or mirrored, not in phx2, so might be slower to upload depending on where you are uploading from.
- an alt space under dl.fedoraproject.org/pub/alt
PROs: is on netapp space so it gets snapshotted and mirrored. CONS: no ipv6, needs setup, more restricted what accounts can upload to it.
Would you like to just try the fedorapeople space first? It looks like anyone in the 'gitanaconda' group should have write to the /project/anaconda/ space there. If you prefer a new group we can set that up too.
Yeah, that looks good: - lack of backup/mirroring should not be a problem (the results are more or less ephemeral anyway & we can easily back them up internally if needed) - upload speed should not be an issue for the PR test logs & we plan to publish daily logs from kickstart test runs, so even if there was a ~hour delay for uploding the logs it should not make much difference for once-a-day run.
Using the existing gitanaconda group should be fine as well, thanks. :)
BTW, how would you recommend to handle the automated log uploads to this space ?
Is there some interface where we can set a SSH public key, so that the test runner can upload the logs there or something similar ?
A related cron job running somewhere on our side would would SSH in regularly to cleanup the files to keep only the N remaining results.
Also it looks like there's a alt anaconda space: https://dl.fedoraproject.org/pub/alt/anaconda/ but only bcl is setup to write there. we could add another person or two there, but wouldn't want to add big groups or automated stuff.
kevin
On 06/22/2018 09:38 AM, Martin Kolman wrote:
Yeah, that looks good:
- lack of backup/mirroring should not be a problem (the results are more or less
ephemeral anyway & we can easily back them up internally if needed)
- upload speed should not be an issue for the PR test logs & we plan to publish daily logs
from kickstart test runs, so even if there was a ~hour delay for uploding the logs it should not make much difference for once-a-day run.
Using the existing gitanaconda group should be fine as well, thanks. :)
BTW, how would you recommend to handle the automated log uploads to this space ?
Is there some interface where we can set a SSH public key, so that the test runner can upload the logs there or something similar ?
Other groups (cockpit at least) have created a fas user. I'd suggest something with 'bot' in the name like 'anaconda-bot'. Then login and set it's ssh key and add it to the group.
A related cron job running somewhere on our side would would SSH in regularly to cleanup the files to keep only the N remaining results.
Sounds good. You could also manage them on your side and rsync --delete, but whatever works.
kevin
infrastructure@lists.fedoraproject.org
-
Kevin Fenzi -
Martin Kolman