On Mon, 2018-04-23 at 00:27 -0400, Neal Gompa wrote:
On Tue, Apr 17, 2018 at 3:05 PM, Jonathan Dieter jdieter@gmail.com wrote:
I'm assuming that you're referring here to getting zchunk packaged into Fedora. I'd really like to finalize the file format (we're close, but I still need a good way of storing signatures in it) and the download API before releasing it into Fedora proper.
I'm looking forward to this!
I've updated the file format to allow for multiple signatures, updated the zchunk code to recognize the existence of a signature (while still not checking it), and have released as zchunk-0.3.0 in COPR. I've also added in 32-bits of flags that we can use to extend the format in a backwards-compatible way.
The current zchunk format description is at: https://github.com/jdieter/zchunk/blob/master/zchunk_format.txt
I would recommend using the dicts mentioned above as they give me over
40% space savings for both other.xml.zck and primary.xml.zck. Do please let me know if you run into any problems.
Are those dictionaries Fedora specific? If so, how can other distributions generate similar ones? If not, still, how were they made? :)
They were generated from Fedora metadata, but they should help with any distribution's repodata. I generated them by splitting a few day's worth of metadata along package boundaries, stripping out any checksums, and then running zstd --train * on the directory containing the split metadata. The script I used is available at https://www.jdieter.net/downloads/zchunk-dicts/split.py, and I hope to write up proper instructions at some point.
Jonathan