The Matrix: Infinite-Horizon World Generation with Real-Time Interaction
210 points
12 days ago
| 15 comments
| thematrix1999.github.io
| HN
alkonaut
12 days ago
[-]
What does "world" mean here? How does the spatiality fit into some latent space? Or what constitutes the "world"? If the answer is, there is none, the world is just frames of video and any consistency quickly blurs out after a few seconds. That's not a world generation, that's just generation of video frames following frames. Not that it isn't cool, but it has almost zero usability for generating a "world" simulation. The key to a realistic world is that you can reliably navigate it. Visit and revisit places. If you modify anything, those modifications are persisted. If you leave a room and re-enter it hours later, the base expectation is that the same objects are in that room.

Wouldn't a working approach be to just create a really low resolution 3D world in the traditional "3D game world" sense to get the spatial consistency. Then this crude map with attributes is fed into frame generation to create the resulting world? It wouldn't be infinite, but on the other hand no one has a need for an infinite world either. A spherical world solves the border issue pretty handily. As I understood it, there was some element of that in the new FS2024 (discussed yesterday on HN).

reply
jerf
12 days ago
[-]
"Wouldn't a working approach be to just create a really low resolution 3D world in the traditional "3D game world" sense to get the spatial consistency. Then this crude map with attributes is fed into frame generation to create the resulting world?"

That's Minecraft. It does work. And when you turn around, then turn back to where you were facing, you don't feel like you're in a fever dream, because the landscape hasn't completely shifted.

The tech is neat and I'm sure worth publishing. The rhetoric accompanying it is terribly overblown. Of course that's as likely to be their University PR department as the authors.

reply
astrobe_
11 days ago
[-]
> Wouldn't a working approach be to just create a really low resolution 3D world in the traditional "3D game world" sense to get the spatial consistency. Then this crude map with attributes is fed into frame generation to create the resulting world?

If you go "ultra-low" resolution, that's basically Minecraft or Luanti (formerly Minetest). There some mods for Luanti that let one generate a less "squary" world by adding slopes, but that's still heavily polygonal.

> It wouldn't be infinite, but on the other hand no one has a need for an infinite world either.

Minecraft has an infinite world (with glitches when you go very far from origin due to floating point errors). Luanti's world is finite (around 64000 m^3 because 16 bits coordinates; players move at 2-4m/s usually), some people are working to make it infinite. In my opinion, you are right that nobody needs an infinite world; the argument for an infinite world is that a finite world can be a problem on popular servers but in my eyes servers where this is a problem don't use enough the available space (in particular the huge vertical space that can be used to create multiple worlds) or shoot themselves in the foot by providing players with ways to move fast (mainly transports); increasing the player's speed actually shrinks the world.

I think there is a conundrum from the players and gameplay perspective. On one hand, you want a lot of space, but on the other hand you want to be close to other players (for commerce, play together, etc.). It's not enough to have an infinite world, you also have to have the gameplay that goes with it.

Another thing to consider are the interaction of mobiles (NPCs, enemies) with the terrain. Because nobody likes an infinitely empty world either. I haven't played an AAA game recently, but a decade ago even with precomputed terrain, some mobiles would look silly sometimes with steep terrains.

reply
scotty79
12 days ago
[-]
It's basically a dream. Maybe that's what dreams are? World prediction engine running with no sensor input and memory in read only mode?
reply
withinboredom
12 days ago
[-]
When you go to sleep, your brain stem disconnects from your body and your brain enters a feedback loop. The sensors are very much connected, just to whatever. Hence why you can grow dragon wings in your dreams and feel them. Memories can be made as well. I used to be really into lucid dreaming and time compression. My longest dream was nine years compressed into a 12-hour sleep period.
reply
Helonomoto
12 days ago
[-]
Where did you get the 'brain stem disconnects from your body'? Because thats not how it works in the brain.

We have the part which controls your muscles and we have another part which simulates the movement. Not executing on it has nothing to do with the brain stem disconnecting from the 'body'.

Its the same mechanism as you thinking about a movement but not doing it.

reply
lxgr
12 days ago
[-]
I believe it's a much more fundamental difference than just the distinction between ideating and acting.

Many people occasionally experience the transition between "conencted" and "disconnected" states as a sudden jerk or loud noise just at the moment of falling asleep.

Sleep paralysis is another "failure mode" of this mechanism that reveals what's going on. (I'm not sure if there is a reverse to it, i.e. whether sleepwalking could be explained as a drastic fail-open of the same mechanism).

reply
Helonomoto
11 days ago
[-]
I read a book, just a few weeks ago regarding this specific topic and my explanaition is directly from that book. We do have specific brain areas for this.

This sudden jerk you can read about it here: https://en.wikipedia.org/wiki/Hypnic_jerk and its not scienctificly clear why and how it works, the best assumption is that its a reflex.

You are not disconnecting your brain from anything.

reply
withinboredom
11 days ago
[-]
Your brain disconnects: https://article.imrpress.com/bri/Landmark/articles/pdf/Landm... is probably the best paper I know of that describes the actual process going on in detail.
reply
jebarker
12 days ago
[-]
> My longest dream was nine years compressed into a 12-hour sleep period.

Does the brain change in response to that sleep period? Or is there no change because there's no new information input?

reply
withinboredom
11 days ago
[-]
I dunno, I wasn't hooked up to any kind of measurement device to measure the changes in my brain. From a subjective point of view, I miss that place. I've been writing a book about my adventure there, off-and-on for years now. Maybe someday, I will finish it. If I could return, I'd do it in a heartbeat at any cost.

I _feel_ older than I am because there are a couple extra decades in my brain than in real life. Most of my time compression experiments were only a few months or weeks. That one long one changed me forever, and I've never done it on purpose since then.

I still have time compressed dreams from time to time, and when I wake up, two or three weeks have subjectively passed, but only a night has passed in the real world. There's a period of time, no more than 10-30 minutes, while the brain tries to reconcile two different and overlapping pasts. It can be a bit disorienting. My wife knows when I have these dreams because when I wake up, she says I look around surprised or confused to be there. The absolute worst is when you lay down to go to bed in the dream and wake up in the real world. Those will mess you up.

So, maybe my brain did change. Who knows? Maybe someone should study it.

reply
captainvalor
11 days ago
[-]
This is maybe the coolest thing I've ever read on HN.
reply
theshaper
10 days ago
[-]
"Awoken By A Lamp" vibes, definitely:

https://www.reddit.com/r/Glitch_in_the_Matrix/comments/30t9k...

Thanks for share.

reply
chamomeal
11 days ago
[-]
What like inception? Like you experienced 9 years and then woke up and went to work??
reply
withinboredom
11 days ago
[-]
In this particular case, I was in the hospital after a major motorcycle accident. So, I didn't go anywhere. To be honest, after a couple of months in the dream, I had determined I had died and found the afterlife. I had never thought it was a dream.

The trick to time compression in dreams is two things:

- being able to generate false memories

- being able to skip the passage of time

It requires acknowledgment that there is no proof you existed 5 minutes ago, only your memories of existing (and surviving the existential crisis that may cause) matching up with the current perceived reality. So, to have a time-compressed dream means to simply 'skip ahead' for a period of time and have access to the memories in-between. This last part is the part that needed the most practice for me. I was able to skip ahead, but it took years before I'd be able to create false memories with coherency. These days, it isn't uncommon to have a dream with an entire lifetime of memories that aren't mine. Luckily, these are forgotten within seconds of waking up, making it easy enough to determine which of my dreamt experiences are fake and which are real.

On a normal night, you only have a couple of hours to dream (more or less depending on sleep deprivation and need for deeper sleep). So it works kinda like a movie that covers a greater period of time, skipping ahead to the interesting parts. Then the access to the memories in-between the interesting part to make decisions and sense out of what you are experiencing.

reply
roboboffin
12 days ago
[-]
Not an expert, but I think procedurally generated terrain is generally fractal in nature, and is reproducible in that sense from a seed that is used in the generation. It is therefore recursive, as fractals are recursive.

A traditional neural network is a universal function approximator, however it is not recursive in nature, unless it is some sort of RNN. The transformer architecture, which this seems fairly similar to this one, is also not recursive in nature; although I believe, limited recursion can come about through CoT.

Therefore, I don't believe this could match the reproducibility, in an infinite sense, of a traditional procedural generator.

reply
jrvieira
12 days ago
[-]
"Fractals are not self-similar"

https://m.youtube.com/watch?v=gB9n2gHsHN4

If you want to learn more about the fascinating world of fractals.

reply
falcor84
12 days ago
[-]
The parent comment didn't mention self-similarity; did you maybe intend to reply to another comment?
reply
astrobe_
11 days ago
[-]
Luanti [1] (Minetest) has a fractal map generator ("mapgen"). You can test it for yourself. It's funny at first but becomes eventually boring.

Its other mapgens massively some kind of Perlin noise in various ways, so that you can have "realistic" landscapes (e.g. the Carpethian mapgen) or landscapes with impossible mountains and floating rocks sometimes (e.g. the V7 mapgen) that are good for fantasy/sci-fi worlds.

Noise is a pretty efficient way to fake complexity, and it's not a coincidence [2].

[1] https://www.luanti.org/ [2] https://en.wikipedia.org/wiki/Kolmogorov_complexity

reply
jerf
11 days ago
[-]
Procedurally generated terrain is whatever it wants to be. It isn't necessary for it to be fractal in any particular sense. Or even arguably all that useful, at least at the present time.
reply
QuantumG
12 days ago
[-]
What you really want is a story telling engine informing the world model and the humans need to generate power.
reply
ilaksh
12 days ago
[-]
I'm into VR and mixed reality, and I think this is headed to making the Holodeck real in an immersive way. That's the concept of the Matrix and what they are demoing, just in 2d.

I am guessing the main thing holding this stuff back in terms of fidelity and consistency or generalization is just compute. But the new techniques they have here have just dramatically lowered the compute costs and increased the generalization.

Maybe just something like the giant Cerebras SRAM chips will get to the next 10 X in scale that smooths this out and pushes it closer to Star Trek. Or maybe some new paradigm like memristors.

But I'm looking forward to within just a few years being able to put on some fairly comfortable mixed reality glasses and just asking for whatever or whoever I want to appear in my home (for example) according to my whim.

Or, train it on a lot of how-to videos such as cooking. It just materializes an example of someone showing you exactly what you need to do right in your kitchen.

Here's another crazy idea: train on videos and interactions with productivity applications rather than games. In the future, for small businesses, we skip having the AI generate source code and just describe how the application works. The data and program state are just stored in a giant context window, and the application functionality changes the instant you make a request.

reply
scotty79
12 days ago
[-]
I think we need devices that can produce gaussian splat videos to become as common as 2d cameras.
reply
chamomeal
11 days ago
[-]
I think fever dream AI stuff like this would be nightmarishly terrifying in VR lol.

Still, would be super cool to try

reply
wavemode
12 days ago
[-]
I wish researchers would spend more time on using generative models to create level geometry, rather than trying to generate video from scratch. It would be both cheaper and more effective for stable gameplay.
reply
jebarker
12 days ago
[-]
There's so many researchers working in AI now that we can afford to explore all avenues. This probably isn't going to lead to a fully generative AI game engine, but I bet there'll be useful learnings along the way.
reply
kridsdale1
11 days ago
[-]
I think Ubisoft are leaders in that space. I saw algorithmic generation of New York style geometry for some behind the scenes video for The Division 2.

I wouldn’t be surprised if epic has similar.

reply
pedalpete
11 days ago
[-]
This is the future I was trying to pitch in 2018 when we had built Ayvri and had every paraglider in the world, the world's largest ultramarathons, drone operators, and lots of other users of our real-world 3D environment.

Though we were using map tiles at the time, we were developing a model that took photos and a GPS track to add information that better matched environmental conditions (cloud, better lighting, etc).

People still ask me to open-source or give them our source code, but the code was acquired, so that isn't possible. But I do regularly say that if I were to rebuild Ayvri today, I'd do it as an interactive video rather than loading tiles.

reply
akutlay
12 days ago
[-]
Why would you want to generate all the pixels using this model instead of generating all the art, physics, and objects in the world using a game engine? The engine does so much of the physics and keeps everything stable for very cheap.
reply
FeepingCreature
12 days ago
[-]
Generating coherent objects in an infinite world is probably harder.
reply
01HNNWZ0MV43FF
11 days ago
[-]
Because you wouldn't have to wait for a human to write the engine and write the game
reply
drusepth
11 days ago
[-]
We generate 2d art, 3d models, some other in-world data, and soon animations at our game studio. The huge problems of generating pure pixels without a persistent game state behind it is illustrated well by the recent MinecraftAI craze (where it's a cool demo, but absolutely unplayable for anything other than novelty), and exacerbated on the dev side by missing out on tons of "free" stuff you get from existing engines that already do almost every kind of physics ops for you.
reply
darepublic
11 days ago
[-]
I didn't fully grok what this was about from the website. Though just last night I was talking to a friend about that quote from the Matrix that Morpheus tells Neo, so some nice synchronicity there. The sense I got from this is that they are developing a triple AAA type virtual world that can get generated on the fly based on text prompts? When the authors say frame level control do they mean that at any point, the next frame can be manipulated, either to be completely new or to influence the current story or context that is being played out?
reply
jerpint
12 days ago
[-]
I’m really excited for where this is going. From the demo videos, it seems to be a step up from Oasis, which itself came out only 2 weeks ago. I expect to see a lot of innovative use cases in this field
reply
grodes
12 days ago
[-]
unreadable website
reply
shmerl
12 days ago
[-]
> Click to play

Clicking - nothing works.

reply
tsaoyu
12 days ago
[-]
Click to play [...the video]
reply
airbreather
11 days ago
[-]
No source, no playable demo, just promises of.

Could be total vaporware for all we know.

It is an ad, a statement of achievement in case someone else states it first, or what?

Seems like it would be better on Youtube, it really doesn't offer much of use right now.

reply
m3kw9
11 days ago
[-]
Definitely used Cyberpunk2077 footage to train
reply
petermcneeley
11 days ago
[-]
"Welcome to the Matrix" with matrix-like rain seems like an invitation for Warner Bros to sue you into oblivion.
reply
efitz
12 days ago
[-]
Prediction: in 20 years, I’m going to be reading about some dude who wrote a program to drive the car continuously until it ran into some surreal edge condition, and finally hit it. There will be a subculture of “matrix glitchers” who spend much of their time doing these kinds of experiments.
reply
soulofmischief
12 days ago
[-]
People have been doing that with Minecraft for over a decade. In the old days, once you got far away enough, the terrain generation would go haywire. Lots of videos from that time period of people exploring the "edge of the world".

Personally, these were the kind of glitches which made games feel magical and "real" to me as a kid. Being able to analyze a system by breaking it made it seem so much more tangible, like an actual place I had an NTSC-sized porthole into.

reply
billisonline
12 days ago
[-]
Cf. the "Minus World" in Super Mario Bros. for the NES.

https://en.wikipedia.org/wiki/Minus_World

reply
soulofmischief
12 days ago
[-]
Ha! I remember being either 5 or 6 when my uncle showed me Minus World and it blowing my mind. That might have actually been my first exposure to "backrooms" glitches like that. What an amazing glitch. It even worked on my combo Super Mario Bros / Duck Hunt cartridge

MissingNo. is another good example. I have fond memories spending untold hours in my favorite game engines trying to break free. The Jak and Daxter series were some of my favorite to break, due to the uniqueness and flexibility of the engine and the weird ways that the chunk loading system could be broken.

reply
axus
11 days ago
[-]
Ahh, "Mountain King" on the Atari 2600 was the game for me finding a cool bug. If you bounced just right, you'd soar over the mountain into the glitches far above. Games didn't crash, they just worked with what they had.
reply
soulofmischief
11 days ago
[-]
I didn't have Mountain King for my 2600 so I looked it up. What a neat glitch. Platformer glitches are fun, I really enjoyed breaking the early Sonic games for things like the Hyper Sonic glitch, or some of the map glitches.

I think this is one thing about Super Mario Bros. 3 that felt so magical to me. With the addition of the hidden whistles and intentional "glitches" like crouching for an extended time on a white platform, running behind map elements, etc. you felt like some kind of plane walker just bending time and space to your will. Fantastic implementation of a level skip mechanism for veteran players. It gave an already incredibly expansive game quite a lot of extra replay value, just like Minus World.

reply
sunnybeetroot
11 days ago
[-]
Thank you for the reminder of MissingNo! Takes me back to when I was a child and received a Gameboy Color without any games. I spent months just watching the start up animation on repeat before I got Pokémon yellow.
reply
soulofmischief
11 days ago
[-]
That is one of the saddest thing I've ever heard. Did your parents just not know it needed games, or was it a budget thing?

I was extremely poor growing up but I did get lucky and get a Gameboy Color for Christmas with a copy of Pokémon Gold at age 5, right before my guardians went insane and forbade any non-Christian media such as "Pocket Demons" or any fantasy content. That game expanded my mind so much, introduced me to a lot of things I'd never encountered before. It seemed so mysterious and huge, especially with the entire extra Kanto campaign. Still one of the greatest and most complete games ever made.

reply
pugworthy
12 days ago
[-]
Far Lands or Bust is a great YouTube channel for this. He’s been walking in one direction in a Minecraft world since 2011.
reply
bobim
12 days ago
[-]
"there's no bugs in there!" a chick with a star on the cheek probably.
reply
ilaksh
12 days ago
[-]
That community already exists because the current version of these types of AI game engines are constantly running into a surreal edge condition since they don't track things consistently when they go off frame.
reply
abricq
12 days ago
[-]
This is surely really cool. Just a bit sad that, as phrased by the authors, the "First Real-Time" virtual world created for the demo is a fat & fast SUV driving on virgin lands.
reply
pmayrgundter
12 days ago
[-]
Sigh, agreed!
reply
spuz
12 days ago
[-]
Why is that sad?
reply
ferguu_
11 days ago
[-]
When you think of "The Matrix" - An infinitely generated and extremely complex world, I don't think most people would picture this, and especially not in a car like that. The car choice makes sense when you consider there aren't any roads at all! Of course, it is a very new technology, but then that does bring into question the paper author's frankly ambitious title choice.

Interestingly, this website does remind me of something I played a while back - check it out! [https://slowroads.io/]. It has roads!

reply
spuz
11 days ago
[-]
What would you expect it to look like based on the title? (I don't understand the title and have no expectations)
reply
ribcage
12 days ago
[-]
Someone should ban "AI" articles on Hacker News.
reply
steego
12 days ago
[-]
I don't know if you're actually new here, or you've been reading HN for years, but your account is only 11 months old.

We talk about a lot of things here, but when we do talk about AI, we tend to prefer talk about things with code or papers.

This particular project has a paper. They're expecting to publish their code soon.

Here's the paper:

https://thematrix1999.github.io/article/the_matrix.pdf

You don't have to read it, but you may want to consider it if you want to learn something and/or contribute something meaningful to the conversation.

reply
ferguu_
12 days ago
[-]
But then there would be no articles!
reply
Cthulhu_
12 days ago
[-]
Why? It's a mostly democratic news aggregator site without much editorial overview, if you don't like it, downvote / don't upvote it, or write a client that filters topics you don't like out based on some keywords. You didn't need to open this page and comment on it if you don't like it, nobody's making you read things you don't like.
reply
olddustytrail
11 days ago
[-]
Yes, and that someone should be you. Be the change you want to be.
reply
brcmthrowaway
11 days ago
[-]
Check lobsters
reply