After all, what Copernicus showed is that the mind bogglingly complicated motion of planets become a whole lot simpler if you change the coordinate system.
Ptolemaic model of epicycles were an adhoc form of Fourier analysis - decomposing periodic motions over circles over circles.
Back to frequencies, there is nothing obviously frequency like in real space Laplace transforms *. The real insight is that differentiation and integration operations become simple if the coordinates used are exponential functions because exponential functions remain (scaled) exponential when passed through such operations.
For digital signals what helps is Walsh-Hadamard basis. They are not like frequencies. They are not at all like the square wave analogue of sinusoidal waves. People call them sequency space as a well justified pun.
My suspicion is that we are in Ptolemaic state as far as GPT like models are concerned. We will eventually understand them better once we figure out what's the better coordinate system to think about their dynamics in.
* There is a connection though, through the exponential form of complex numbers, or more prosaically, when multiplying rotation matrices the angles combine additively. So angles and logarithms have a certain unity, or character.
The next big jumps were to collections of functions not parameterized by subsets of R^n. Wavelets use a tree shapes parameter space.
There’s a whole, interesting area of overcomplete basis sets that I have been meaning to look into where you give up your basis functions being orthogonal and all those nice properties in exchange for having multiple options for adapting better to different signal characteristics.
I don’t think these transforms are going to be relevant to understanding neural nets, though. They are, by their nature, doing something with nonlinear structures in high dimensions which are not smoothly extended across their domain, which is the opposite problem all our current approaches to functional analysis deal with.
For GPT like models, I see sentences as trajectories in the embedded space. These trajectories look quite complicated and no obvious from their geometrical stand point. My hope is that if we get the coordinate system right, we may see something more intelligible going on.
This is just a hope, a mental bias. I do not have any solid argument for why it should be as I describe.
That idea was pushed to its limit by the Koopman operator theory. The argument sounds quite good at first, but unfortunately it can’t really work for all cases in its current formulation [1].
We know that under benign conditions and infinite dimensional basis must exist but finding it from finite samples is very non-trivial, we don't know how to do it in the general case.
Here's an example of directly leveraging a transform to optimize the training process. ( https://arxiv.org/abs/2410.21265 )
And here are two examples that apply geometry to neural nets more generally. ( https://arxiv.org/abs/2506.13018 ) ( https://arxiv.org/abs/2309.16512 )
The Fourier and Wavelet transforms are different as they are self-adjoint operators (=> form an orthogonal basis) on the space of functions (and not on a finite dimensional vector space of weights that parametrize a net) that simplify some usually hard operators such as derivatives and integrals, by reducing them to multiplications and divisions or to a sparse algebra.
So in a certain sense these methods are looking at projections, which are unhelpful when thinking about NN weights since they are all mixed with each other in a very non-linear way.
It's disconcerting at times, the scope of finite and infinite dimensional linear algebra, especially when done on a convenient basis.
[1] https://www.cis.rit.edu/class/simg716/Gauss_History_FFT.pdf
In contrast, Gauss disliked teaching and also tended to hoard those good ideas until he could go through all the details and publish them in the way he wanted. Which is a little silly, as after a while he was already widely considered the best mathematician in the world and had no need to prove anything to anyone - why not share those half-finished good ideas like Fast Fourier Transforms and let others work on them! One of the best mathematicians who ever lived, but definitely not my favorite role model for how to work.
Of course, irascible brilliance and eccentricity has an honorable place in mathematics too - I don't want to exclude anyone. (Think Grigori Perelman and any number of other examples!)
They, Newton included, would often feel that their work was not good enough, that it was not completed and perfected yet and therefore would be ammunition for conflict and ridicule.
Gauss did not publicize his work on complex numbers because he thought he would attacked for it. To us that may seem weird, but there is no dearth of examples of people who were attacked for their mostly correct ideas.
Deadly or life changing attacks notwithstanding, I can certainly sympathize. There's not in figuring things out, but the process of communicating that can be full of tediousness and drama that one maybe tempted to do without.
There's joy in figuring things out, but the process of communicating what has been so figured can be tedious and full of drama -- the kind of drama that one maybe tempted to do without.
As if phd students need more imposter syndrom to deal with. Ona serious side, I wonder what conditions allow such minds to grow. I guess a big part is genetics, but I am curious if the "epi" is relevant and how much.
Not sure if true, but allegedy he insisted his son not go into maths, as he would simply end up in his father's shadow as he deemed it utterly Impossible to surpass his brilliance in maths :'D
Definitely true but also bad parenting. Gauss was somewhat of a freak of nature when it came to math. Him and Euler are two of the most unreasonably productive mathematicians of all time.
Nepotism existed since time immemorial but for a mathematical genius, what was the nepotistic deliverable for the child? A sinecure placement at university?
Implicit in the "correctness" of this motive is the idea that unless you're #1 in your field, you are nothing (depression implies strong feelings of worthlessness).
I don't know if you think that's a great lesson to teach your kids as a parent, but I don't.
Doubly so when the rationale is “I’m so fucking awesome”
Triply so when it’s something you’re passionate about, presumably inherently.
Quadruply so when it’s your child. Its tough as a kid hearing your parents come up with elongated excuses why you can’t dream and work towards a future.
When you let people find their own way, you might even learn something from it (ex. 70 yo Gauss learns he didn’t need to tie his mental state to his work because his son doesn’t suddenly become depressed from not matching dads output)
Re: second half, sounds about right, confused at relevancy though (is the idea the child would only do it to pursue nepotistic spoils and an additional reason is the spoils aren’t even good?)
So, a nepotistic delivery was beneficial for his family, and advising his son to seek excellence outside the shadow cast by Gauss himself wasn't stamping on dreams (in my view) it was seeking the happiest outcome.
Without overdoing it, the suicide rate for rich kids with famous parents isn't nothing. There are positive examples, Stella McCartney comes to mind. She isn't wings.
There's a spread of farmers, railroad and telegraph directors, high level practical infomation management skills in the children.
I had a couple charts that showed a trend line of the last n days until someone in OPs noticed that three charts were fully half of our daily burn rate for Grafana. Oops. So I started showing a -7 days line instead, which helped me but confused everyone else.
You'd probably want to use a tool like calculating the cepstrum rather than fourier transform. Cepstral methods are commonly used in mechanical analysis to detect periodic impacts like where a gear tooth gets damaged.
On a more mundane note: my wife and I always argue whose method of loading the dishwasher is better: she goes slow and meticulously while I do it fast. It occurred to me we were optimizing for frequency and time domains, respectively, ie I was minimizing time so spent while she was minimizing number of washes :-)
Ears are essentially 2 "pixels" of sound sensing; and for that limitation they are ABSOLUTELY AMAZING at pointing out the sound source.
That lower bound is the uncertainty principle, and that lower bound is hit by normal distributions.
It’s also the kind of thinking that can throw a wet blanket on the “beauty” of e.g. Eulers identity (not being critical, I genuinely appreciate the replies I got)
I'm probably just slow, but I'm not following. Do you mean because you went fast, you had to run another cycle to clean everything properly?
If you haven't already, you should watch the Technology Connections series on dishwashers.
And if loading dishwasher is on top of your marital issues you're probably in very happy marriage.
The constant small degree of conflict and strife is key to happiness, people can't be permanently happy, they just find ways to sabotage when they do
Not the first time I've heard this on HN. I remember a user commenting once that it was one of the few perspective shifts in his life that completely turned things upside down professionally.
https://github.com/giladoved/webcam-heart-rate-monitor
https://medium.com/dev-genius/remote-heart-rate-detection-us...
The Reddit comments on that second one have examples of people doing it with low quality webcams: https://www.reddit.com/r/programming/comments/llnv93/remote_...
It's honestly amazing that this is doable.
Non-paywalled version of the second link https://archive.is/NeBzJ
https://news.mit.edu/2014/algorithm-recovers-speech-from-vib...
> The unreasonable effectiveness of The Unreasonable Effectiveness title?
I agree this is getting old after 75 years. Not least because it seems slightly manipulative to disguise a declarative claim ("The Fourier transform is unreasonably effective."), which could be false, as a noun phrase ("The unreasonable effectiveness of the Fourier transform"), which doesn't look like a thing that can be wrong.
FTs are actually very reasonable, in the sense that they are a easy to reason about conceptually and in practice.
There's another title referenced in that link which is equally asinine: "Eugene Wigner's original discussion, "The Unreasonable Effectiveness of Mathematics in the Natural Sciences". "
Like, wtf?
Mathematics is the language of science, science would not compound or be explainable, communicable, or model-able in code without mathematics.
It's actually both plainly obvious for mathematics then to be extremely effective (which it is) and also be evidently reasonable as to why, ergo it is not unreasonably effective.
Also the slides are just FTs 101 the same material as in any basic course.
So, biology and medicine are not sciences? Or are only sciences to the extent they can be mathematically described?
The scientific method and models are much more than math. Equating the reality with the math has let to myriad misconceptions, like vanishing cats.
And silly is good for a title -- descriptive and enticing -- to serve the purpose of eliciting the attention without which the content would be pointless.
Which means, the language of some fields can’t be math.
However, I don’t think the original presenter was asserting those fields aren’t science, that’s an unreasonable interpretation. More so , ideally they would be use math as it is a language that would help prevent the silly argument “so, Y is not X?, or is Y only X provided Y is in the subset of X that excludes Z? “
(Even in Engineering, we hit this cognitive limit, and all sorts of silliness emerges about why things are or are not formalised)
If that is what you are saying I suggest that you actually go back and read it. Or at least the Wiki article:
https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...
By means of contrast: I think it's clear that mathematics is, for example, not unreasonably effective in psychology. It's necessary and useful and effective at doing what it does, but not surprisingly so. Yet in the natural sciences it often has been. This is not a statement about mathematics but about the world.
(As Wittgenstein put it some decades earlier: "So too the fact that it can be described by Newtonian mechanics asserts nothing about the world; but this asserts something, namely, that it can be described in that particular way in which as a matter of fact it is described. The fact, too, that it can be described more simply by one system of mechanics than by another says something about the world.")
> Wigner's first example is the law of gravitation formulated by Isaac Newton. Originally used to model freely falling bodies on the surface of the Earth, this law was extended based on what Wigner terms "very scanty observations"[3] to describe the motion of the planets, where it "has proved accurate beyond all reasonable expectations."
So despite 'very scant observations' they yielded a very effective model. Okay fine. But deciding they should be 'unreasonably' so is just a pithy turn of phrase.
That mathematics can model science so well, is reductive and reduces to the core philosophy of mathematics question of whether it is invented or discovered. https://royalinstitutephilosophy.org/article/mathematics-dis...
Something can be effective, and can be unreasonably so if it's somehow unexpected, but I basically disagree that FTs or mathematics in general are unreasonably so since we have so much prior information to expect that these techniques actually are effective, almost obviously so.
And no, this is unrelated to whether math is invented or discovered. If anything this is related to the extreme success of reductionism in physics.
As a general point of reflection: If an influential article by a smart person seems silly to you, it's good practice to entertain the question if you missed something, and to ask what others are seeing in it that you're missing.
It's not easy to separate cause and effect from direct and strong correlations that we experience.
The job of a scientist is not to give up on a hunch with a flippant "correlation is not causation" but pursue such hunches to prove it this way or that (that is, prove it or disprove it). It's human to lean a certain way about what could be true.
ok but it's not the FTs that are unreasonable, it's the effectiveness
I think we all understand at this point that "unreasonable effectiveness" just means "surprisingly useful in ways we might not have immediately considered"
Metaphorical language compels them to microrebuttals.
Why "The \"Unreasonable Effectiveness\" Title Considered Harmful" Matters
The Unreasonable Effectiveness of "\"Why \\\"The \\\\\\\"Unreasonable Effectiveness\\\\\\\" Title Considered Harmful\\\" Matters\" Considered Harmful"
Ironically a very relevant and accurate title.
Of course some differences exist (e.g. basis vectors are fixed in FFT, unlike PCA).
In my view the Fourier Transform is still useful in the real world. For example you can use it to analytically derive the spectrum of a given window.
But I think the parent is hinting at wavelet basis.
I guess if you want to use different modulations you treat the complex number corresponding to the subcarrier as an IQ point in quadrature. So you take the same symbols, but read them off in the frequency domain instead of the time domain.
And I guess this works out quite equivalently to normally modulating these symbols at properly offset frequencies (just by the superposition principle)
As to the listed patent, it moves uncomfortably close to being a patent on mathematics, which isn't permitted. But I wouldn't be surprised to see many outstanding patents that have this hidden property.
* not a legal definition, IANAL.
Yes, that's true. In that example, you're not patenting mathematics, you're patenting a specific application, which can be patented. In my reading I see that mathematics per se is an abstract intellectual concept, thus not patentable (reference: https://ghbintellect.com/can-you-patent-a-formula/).
There is plenty of case law in modern times where the distinction between an abstract mathematical idea, and an application of that idea, were the issues under discussion.
An obligatory XKCD reference: https://xkcd.com/435/
And IANAL also.