FYI - We are running out of space on copr-be.cloud.fedoraproject.org:
/dev/vdc1 6.4T 6.4T 75G 99% /var/lib/copr/public_html
This is where repositories are stored. We likely run out of space till Monday. Therefore I will add some space to this volume from openstack - I plan to add 500 GB.
We have been working on a task which should results in smaller footprint - to delete EOLed chroots with build artifacts (with opt-out option).
Miroslav
On 10/12/18 3:50 AM, Miroslav Suchý wrote:
FYI - We are running out of space on copr-be.cloud.fedoraproject.org:
/dev/vdc1 6.4T 6.4T 75G 99% /var/lib/copr/public_html
This is where repositories are stored. We likely run out of space till Monday. Therefore I will add some space to this volume from openstack - I plan to add 500 GB.
ok. How much do we have left on that device?
We have been working on a task which should results in smaller footprint - to delete EOLed chroots with build artifacts (with opt-out option).
Cool. That should help.
Thanks!
kevin
Dne 12.10.2018 v 22:11 Kevin Fenzi napsal(a):
How much do we have left on that device?
At the end I added 1TB. Now we have:
/dev/vdc1 7.5T 6.2T 1.3T 83% /var/lib/copr/public_html
I attached the graph showing the history (sorry for the gap in 2015,2016). But the end of the story is that we consumed 2 TB during last year and we are accelerating.
So my question is - can Copr consume additional cca 3 TB of storage next year from fedora-infra resources? Or should I start planning to get money for a storage?
Miroslav
On 10/16/18 12:08 AM, Miroslav Suchý wrote:
Dne 12.10.2018 v 22:11 Kevin Fenzi napsal(a):
How much do we have left on that device?
At the end I added 1TB. Now we have:
/dev/vdc1 7.5T 6.2T 1.3T 83% /var/lib/copr/public_html
I attached the graph showing the history (sorry for the gap in 2015,2016). But the end of the story is that we consumed 2 TB during last year and we are accelerating.
Purging old eol data should slow that? Any idea how much?
So my question is - can Copr consume additional cca 3 TB of storage next year from fedora-infra resources? Or should I start planning to get money for a storage?
We should definitely plan for it. We should coordinate and decide who is asking for it (so only one of us does). :)
We are going to be evaluating some new storage for koji archiving (old releases in koji). If that works out well, we may be able to look at that same vendor for cloud storage moving forward.
So, lets keep this in mind, and revisit later in the year when we are making budgets for next year and also when we have more info about this new storage vendor and if they could meet our needs.
Thanks for the email!
kevin
Dne 18.10.2018 v 21:08 Kevin Fenzi napsal(a):
Purging old eol data should slow that? Any idea how much?
du tells me that:
fedora-21-* 82 GB fedora-22-* 132 GB fedora-23-* 241 GB fedora-24-* 343 GB fedora-25-* 438 GB fedora-26-* 775 GB
epel-5-* 9.7 GB
and I have some more time to check: epel-6-* which is 132 GB
Miroslav
Dne 22.10.2018 v 18:18 Miroslav Suchý napsal(a):
Dne 18.10.2018 v 21:08 Kevin Fenzi napsal(a):
Purging old eol data should slow that? Any idea how much?
du tells me that:
fedora-21-* 82 GB fedora-22-* 132 GB fedora-23-* 241 GB fedora-24-* 343 GB fedora-25-* 438 GB fedora-26-* 775 GB
epel-5-* 9.7 GB
and I have some more time to check: epel-6-* which is 132 GB
epel-7-* 531 GB fedora-27-* 615 GB fedora-28-* 612 GB fedora-29-* 335 GB
We are working on a code which will keep EOLed chroot as opt-in. I.e., owners will get email if they want to keep it. Unless they confirm, we will delete them. When this will be done, we can delete up to 2TB of data. However, in the next 12 months we will like consume 2.5 TB of data - assuming the same grow as in past. And additional 5 TB in following year. [*] This assume the same set or architectures - but `mock --force-arch` allows us to build for all primary architectures. The only limitation is a storage now. I would love to add aarch64 soon. For the records I queried how much all ppc64le consumes:
*-ppc64le 307GB intel arch have the rest, i.e., 5.5 TB
[*] Please mind that forecast is hard - especially about a future.
Miroslav
On Tue, 23 Oct 2018 at 04:51, Miroslav Suchý msuchy@redhat.com wrote:
Dne 22.10.2018 v 18:18 Miroslav Suchý napsal(a):
Dne 18.10.2018 v 21:08 Kevin Fenzi napsal(a):
Purging old eol data should slow that? Any idea how much?
du tells me that:
fedora-21-* 82 GB fedora-22-* 132 GB fedora-23-* 241 GB fedora-24-* 343 GB fedora-25-* 438 GB fedora-26-* 775 GB
epel-5-* 9.7 GB
and I have some more time to check: epel-6-* which is 132 GB
epel-7-* 531 GB fedora-27-* 615 GB fedora-28-* 612 GB fedora-29-* 335 GB
We are working on a code which will keep EOLed chroot as opt-in. I.e., owners will get email if they want to keep it. Unless they confirm, we will delete them. When this will be done, we can delete up to 2TB of data. However, in the next 12 months we will like consume 2.5 TB of data - assuming the same grow as in past. And additional 5 TB in following year. [*] This assume the same set or architectures - but `mock --force-arch` allows us to build for all primary architectures. The only limitation is a storage now. I would love to add aarch64 soon. For the records I queried how much all ppc64le consumes:
*-ppc64le 307GB intel arch have the rest, i.e., 5.5 TB
[*] Please mind that forecast is hard - especially about a future.
Miroslav
We will need to work on budgets then with someone looking to pay for this storage. We would need to plan out for 4 years of growth so let us say 20TB of storage without backups.
infrastructure@lists.fedoraproject.org