Connections in Math: the two kinds of random
29 points
4 hours ago
| 6 comments
| stillthinking.net
| HN
AnotherGoodName
2 hours ago
[-]
This is under-taught in comp sci so I'll say it in all caps for visibility. THERE IS NO KNOWN GENERIC WAY TO MEASURE INFORMATION ENTROPY!!!!

Seriously. There isn't. You might say 'but measure the number of 1's vs 0's' and i'll just reply with '101010101010 repeated'. So you up it and start saying ok maybe measure frequencies of pairwise bits then and i just reply by cycling 00,01,10,11 endlessly etc. Frequency counts and any other measure you can think of doesn't actually work.

The root definition will state it's just based on the properties of the prediction but... what model do you use for the prediction? This gets to the heart of the matter;

The measure of information entropy itself is actually equivalent to creating on omniscient oracle. It requires being able to ask the question "Oh great oracle, what's the shortest program that can reproduce X"? or equivalently "What's the probability of this data assuming i always have the most perfect model to predict it?". You then calculate entropy based on that oracles answer.

Kolmorogorov, AI and information entropy are all pointing to the same thing. The reason people get confused by the fact that the frequencies of base 10 digits is the same for digits of pi and a truly random sequence is just because the counts of base 10 digits isn't a measure of entropy at all. Measuring entropy literally requires an omniscient oracle.

reply
amluto
6 minutes ago
[-]
> This is under-taught in comp sci so I'll say it in all caps for visibility. THERE IS NO KNOWN GENERIC WAY TO MEASURE INFORMATION ENTROPY!!!!

Both your comment and the OP seem to conflate a distribution with a sample from a distribution.

A distribution over strings has entropy. A single string can have a Kolmogorov complex but does not have entropy in the Shannon sense per se.

When you write a compression algorithm, you are often effectively building an algorithm to look at a single sample from an unknown distribution (the input data) and trying to invent a distribution that is both easy to describe and that produces the input data with respectably large probability. And then you output the description of the distribution and enough data to identify the sample in question, and the latter takes space that is roughly the entropy of the distribution you just made up.

> "What's the probability of this data assuming i always have the most perfect model to predict it?". You then calculate entropy based on that oracles answer.

Careful here. The most perfect model may be the one that outputs the data in question with probability one, and the entropy is exactly zero.

If you want information theory to give you meaningful answers, you need to ask it more carefully specified questions.

reply
contravariant
4 hours ago
[-]
I think the explanation of entropy's blind spot is a bit off. It's not actually a problem for entropy if something is generated by a rule, you can calculate entropy for things like the continuous fractions for instance, with an easy rule to generate them for any particular number. Likewise for decimal expansions.

The real blind spot is that entropy is meaningless for a specific sequence, you can't really ask about the entropy of pi if you don't have a theory for how the numbers are generated. Sure if it is pick a uniformly random real number between 0 and 10 then both files have equivalent entropy, but sending pi is also vanishingly unlikely.

There's actually a more subtle way in which this is a blind spot, which takes a bit more machinery. You can define entropy for an ergodic system, which could be considered a kind of mathematical RNG. Now as it turns out this provides a way to generate something almost equivalent to a particular distribution except that this argument only holds for most starting points not all. A direct example would be how pi generates a perfectly fine random distribution of digits (we think) but something like 1/3 does not.

reply
pcael
4 hours ago
[-]
thats interesting, and maybe beyond my current knowledge, I will certainly look into it. About the entropy being a property of a distribution, thats totally correct and I need to fix the post. Thanks.
reply
zzless
3 hours ago
[-]
The choice of pi as a number whose sequence of digits is random is a bit of a weak point of the argument in the post. It is not even known whether every digit 0-9 appears infinitely many times in pi (a weak version of the normal number conjecture). So we do not really know (but strongly believe) that the sequence of pi digits would truly appear random. This of course does not detract from beautiful arguments and the general ideas in this post.
reply
tristenharr
4 hours ago
[-]
Lately I’ve felt Kolmogorov complexity is an unfair measurement because it takes for granted your underlying programming language as treats it as zero cost. In theory you could create a custom language and embed the program as data and “compress” a large random sequence with a better Kolmogorov complexity for that specific language than Pi, simply by not exposing the ability in the language to even work with Pi. I think what’s maybe more interesting is when you take into account the work of Dr. Futamura and the idea of Jones Optimality and view things through that lens.
reply
zzless
3 hours ago
[-]
His definition of Kolmogorov complexity is a bit loose. The rigorous definition uses Turing machines (or Minsky, or Post, or some sort of lambda expression, etc.) so the size is something specific. Different versions of complexity defined this way may give different values but have the same properties and asymptotics so one might just as well stick with the Turing kind. Chaitin's theorem (about the limit of Kolmogorov's complexity being just entropy) holds for all versions as well.
reply
adgjlsfhk1
1 hour ago
[-]
it's not just that they have the same asymptomatic. once you do the radix conversion (e.g. base 10 has log(10)/log(2) times more symbols), any 2 definitions are only off from each other by a constant
reply
AnotherGoodName
1 hour ago
[-]
You always include the measurement of things needed to run the program too.

It's a bit like how benchmarks of compression utilities should always include the size of the utility itself. Otherwise someone can just submit a program with a dictionary of 256 common benchmark files for compression and claim "it compresses them to a single byte" :)

reply
pcael
3 hours ago
[-]
Does that solve the issue? You can always ask yourself if you can embedd something smaller or not? Kolmogorov is just comparing things.. plus, in order to specifically point to pi in the languages internal table, you will need complexity as large as your representation of pi.
reply
hyperhello
4 hours ago
[-]
The thing that frustrates me about this argument is that there is no shortest program that produces pi. You need a computer to run it, which is massive non compressed data, or a human to calculate stuff, an uncountable amount of entropy.

I see that the irrational pi has a smooth distribution of digits and a file full of zeroes is compressible, but they are both sort of magically part of a world that does not run programs and thus not quite different in a practical sense.

Just my thoughts and sorry for the confusion.

reply
pcael
4 hours ago
[-]
I think that does not hold, Kolmogorov complexity is measured relative to a pre-defined universal machine for everything. The machine is not counted in the description of π, for the same reason a book's length isn't measured by including the size of the reader. You fix one interpreter, then ask "how long is the shortest input that makes something?" The interpreter is a constant — the same constant for π, for the random file, for every string in the post
reply
grammarxcore
2 hours ago
[-]
Sure, but spigot is a library, not the interpreter. The author’s program is short because it hides that.
reply
andytratt
3 hours ago
[-]
there are many instances of claude in here, so not sure what that disclaimer was about.
reply
MoltenMan
10 minutes ago
[-]
Yep. I'm already not a huge fan of reading AI writing, and adding an obviously false disclaimer at the top is ridiculous.
reply
elendilm
1 hour ago
[-]
True. There is no shame in AI writing or human writing. Content is what matters. It appears the author is trying to fake genuinity in a fundamentally wrong way and yet doesn't even deliver on it.
reply
skybrian
2 hours ago
[-]
Maybe the author has been overly influenced by Claude's writing style?
reply
peterashford
1 hour ago
[-]
how do you know?
reply