Munich 1991: The Roots of the Current AI Boom
56 points
by tosh
2 days ago
| 5 comments
| people.idsia.ch
| HN
jcattle
2 hours ago
[-]
There's this crowd on HN which is very vocal against academia. From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc.

And while it is very true that often the research coming out of Academia is useless, what is always neglected are the roots of the research done in private labs.

When Jürgen Schmidhuber and team published their work on Neural Nets back in 1991 it was also useless. Unless you had a supercomputer and very, very deep pockets you were not going to do anything with what came out of their lab.

But still, 30 years later here we are, standing on top of the shoulders of this useless research.

reply
yorwba
1 hour ago
[-]
Like half of what Schmidhuber is always complaining about is that (except for LSTMs) people aren't standing on the shoulders of his research very much. They try to solve some of the same problems people have always wanted to solve, try some of the same approaches people always tend to try, and then tinker until it works. At no point do they consult Schmidhuber's decade-old papers where he tried something kind of similar but didn't get very impressive results, and hence they also do not think to cite him. Then he comes out of the woodwork to assert priority.
reply
suddenlybananas
49 minutes ago
[-]
You can be influenced downstream by papers you haven't personally read.
reply
bonzini
12 minutes ago
[-]
Shane Legg was in Schmidhuber's lab at IDSIA before being one of the founders of DeepMind, so he probably read the papers personally and knows what influenced him or not...
reply
ACCount37
19 minutes ago
[-]
Where is "this crowd" that you are talking about?

The closest to that that I've seen is that traditional academia approaches are too far removed from practical applications for highly applied fields like software engineering, or too slow for fast-moving fields like modern day ML (thus, all the preprints).

reply
tcp_handshaker
8 minutes ago
[-]
I think most of criticism of academia is about the rampant fraud and unreproducible results, due to the way the incentives are structured.
reply
practal
30 minutes ago
[-]
TU Munich and Nipkow, Makarius et.al. are also at the center of the influential Isabelle theorem prover. TU Munich is cool :-)
reply
MeteorMarc
18 minutes ago
[-]
Also see Schmidhuber's take on the Hinton + Hopfield Nobel prize: https://people.idsia.ch/~juergen/physics-nobel-2024-plagiari...
reply
h8hawk
8 minutes ago
[-]
It's sad that he is the only one speaking out about Hinton. This whole Hinton glorification seems like it's being pushed by an agenda. I'm not sure if he would receive this much attention if he held a different view (closer to LeCun or Ng), rather than these Effective Altruism takes on current AI.
reply
emmelaich
1 hour ago
[-]
reply
jacknews
2 hours ago
[-]
Surely the roots, if we skip over the early preceptron work', are in backpropagation and Hinton, and the work going on at Edinburgh and elsewhere in the 80s.

Indeed I remember buying a set of three conference-papers-as-books around that time, titled Artificial Neural Networks .. proceedings of the whatever the conference was.

No doubt Schmidhuber made important contributions, but I see him pop up claiming to be the 'root' of it all every couple of years.

reply
h8hawk
2 hours ago
[-]
Hinton did not invent backpropagation.

related paragraph from Wikipedia:

Modern backpropagation was first published by Seppo Linnainmaa as "reverse mode of automatic differentiation" (1970)[26] for discrete connected networks of nested differentiable functions.[27][28][29]

In 1982, Paul Werbos applied backpropagation to MLPs in the way that has become standard.

reply
ogrisel
26 minutes ago
[-]
Paul Werbos did not apply backprop to MLPs as cleanly described in Hinton's paper, but rather to some kind of autoregressive non-linear parametrized functions with a much more specific application scope.

Both papers are direct applications of the chain rule applied to estimate the gradient of a multivariate function.

reply
hyttioaoa
1 hour ago
[-]
That's what bugs me about him. So much work has gone into today's models that calling his contributions "the root" isn't really warranted. He's always complaining that Hinton, LeCun, and Bengio get more credit than they deserve, and now he's over-claiming himself.
reply
BoredPositron
48 minutes ago
[-]
Both can be right.
reply
emil-lp
1 hour ago
[-]
Surely the roots go back to Turing, Gödel, Hilbert, Frege, Leibniz, Aristoteles.
reply