> This paper addresses the challenge by asking: how can we trade off more compute for less data?
There is evidence that training RNN models that compute several steps with same input and coefficients (but different state) lead to better performance. It was shown in a followup to [1] that performed ablation study.
[1] https://arxiv.org/abs/1611.06188
They fixed number of time steps instead of varying it, and got better results.
Unfortunately, I forgot the title of that ablation paper.
The fixed point nature of DEQs means that they inherently have a concept of self assessment how close they are to the solution. If they are at the solution, they will simply stop changing it. If not, they will keep performing calculations.
Edit: from the source [1], this quote pretty much sums it all up: "Our 2022 paper predicted that high-quality text data would be fully used by 2024, whereas our new results indicate that might not happen until 2028."
[1] https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-...
Easier said than done.
Robotics tends to be even more data-constrained than NLP. The real world only runs at 1x speed, and if your robot breaks something it costs real money. Simulators are simplistic compared to reality and take a lot of manual effort to build.
You will always need to make efficient use of the data you have.
There is also the problem that on-device learning is not yet practical.
However, due to how diffusion models are trained, they never see their own predictions as input, so they cannot learn to store information across steps. This is the complete opposite for reasoning models.
It should be trivial to make an encoder that has some memory of at least part of the prompt (say the tailing part) and do a diffusion step there too.