Hellishly Slow Level 13 Deflate Compression
34 points
4 days ago
| 4 comments
| kirill.korins.ky
| HN
jbosh
1 hour ago
[-]
I love it. So much in computers is trade offs and this was a fun read exploring it.

It would be interesting to see some economics of what 8,000% increase in encoding time takes to make that money back in terms of storage or bandwidth. I also wonder how brotli/lzma would compare here. Are there some obscene modes on those that had similar results?

reply
userbinator
46 minutes ago
[-]
I also wonder how brotli/lzma would compare here.

Far better, just like anything else based on arithmetic coding. The main distinction here is that the output can still be decompressed with a standard Inflate implementation.

reply
a_t48
1 hour ago
[-]
zstd has higher level modes. Default is -3. I saw a good tradeoff between compression speed and ratio up to -9 or so. From -20 to -22 it will use much more memory and IIRC can have downstream effects on decompression speed. I'm using -9 for my container registry and plan to recompress at a higher level for commonly accessed base layers, as well as give customers a button that lets them pay a bit more to do it themselves.
reply
loeg
31 minutes ago
[-]
To be a little pedantic, the usual zstd levels are positive integers (1-22 default 3). The negative integers denote "fast" modes with worse compression (there are only a few of these).
reply
Someone
28 minutes ago
[-]
So, what’s the effect on memory usage?

And for decompression, the effect on memory usage and timings?

reply
tobijdc
57 minutes ago
[-]
There is also zopfli and it's decadent ECT that allow for more extreme tradeoffs.
reply
userbinator
26 minutes ago
[-]
It's interesting to see just how far Deflate can be taken, and to know that even after decades there is still some (admittedly tiny) room for improvement. Optimal LZ is well-known, and so is static Huffman, but their combination creates some additional inefficiencies(opportunities).

...and of course it's written by someone with a Russian name, and has that characteristic style common to many other articles about data compression.

reply