Question for those making Splats...how do you get such large environments? I've been playing around with them a bit and I'm finding I'm running out of memory with surprisingly little built even on an RTX6000. Any tips or ideas would be awesome!
Plays decently smooth on my M4 Max. It's probably still a long way from being a production-ready replacement for meshed environments, but I could imagine a hybrid mode where certain elements like grass and shrubbery are drawn with gaussians, perhaps with support for basic procedural animation. Great work with the playable demo!
Endless fields of grass and other things where you can make copies of a single base thing and just argue in some parameters like position, color, type etc are cheap to render. Making them sway or react to a body also isn't a problem.
For me, the biggest issue this solves is the blank canvas paralysis problem. Artists are visual thinkers and need a little nudge in the right (art) direction. This is a great way to fill that blank sheet of paper with something that they can take and run with.
Editing Gaussian Splats is still a pain in the ass in the artist's perspective. Even if you can create a good-enough first try using scanned data or generative AI, you just end up with a rough draft that you cannot polish in any way. Existing mesh-based tools allow you to edit the geometry relatively easily, since they are in a higher level discrete representation rather than just a point cloud data structure.
It seems to me that there's some overheated rhetoric that reminds me of the tech-specific spin on the Appeal to Novelty fallacy [1], where people think a new tech is going to uniformly improve on an old tech, that if it isn't an improvement on every front it is somehow a "failure", and therefore if we like the new tech and we are on Team New Tech that we must defend how the new tech is an improvement on every aspect.
Gaussian splats are definitely interesting and do something things older tech is not very good at, but at the same time, it's definitely going to end up being a tool in the tool chest and not completely murderating mesh-based tech or something because they have a lot of other weaknesses, like editability. Or dynamic animation.
What I think some people may not realize is, that's not particularly uncommon. There's a really, really long line of graphical techs that do something particularly well but their weaknesses have kept them in a limited use. It's not a problem for Gaussian splats to become a tool in the toolchest; they aren't a "failure" if we're still using meshes for a lot of things in 10 years.
Mesh-type techs are the "default" for some good reasons.
There’s some movement in this area to be able to surface quantize the splats but you are right, right now it’s simply just visual language and isn’t useful in the pipeline.
Extracting a surface mesh is possible, but the result is going to be really ugly (like the high-poly meshes from generative AI that are useless to artists)!
Mesh processing is a very difficult research domain in computer graphics that has been iterated for several decades, and we still don't have a good automated solution for retopology (Partly because the problem is hard to define in a mathematical way, but also since it's not a problem you can just solve with AI by throwing data and compute at it)
I used voxelization of the splats in the past so I appreciate the notes on the difficulty but this is sort of what PlayCanvas is doing here. Taking the splats, making voxels, meshing.
It's a novel approach and worked well in BIM a few years ago, though not anything real-time.
Yeah, but this seems to be just a 3D GS video (captured from several different camera angles), similar to how an ordinary 2D video is just a series of still frames. For 3D games this would be unsuitable since animations often have to be generated on the fly based on game physics. Even for pre-baked animations the memory cost of loading each frame individually would be too inefficient. For polygon meshes you have just a single static mesh that is deformed over time.
> Dreams managed to animate splats on the PS4. Admittedly, not quite the same type of splats, but there is probably a middle ground here where it can be made to work
I'm pretty sure Dreams only allowed animations as translations and rotations, not something that approximates soft skeletal animations. And even translations and rotations would be problematic since 3D GS scenes rely on baked lighting which would then result in objects no longer fitting the scene.
Dreams managed to animate splats on the PS4. Admittedly, not quite the same type of splats, but there is probably a middle ground here where it can be made to work
dynamic objects are still largely unsolved problem, I just tried to approach it in this demo. also this particular place doesn't have reflective surfaces, but technology supports it - check for example this splat https://superspl.at/scene/ff1d0393 or this one https://superspl.at/scene/6c822f84
How practical would it be to include LIDAR in the initial real-world environmental scan to get (or at least seed/constrain with real data samples) an even better collision mesh?
I'm looking forward to seeing what will happen when gaussian splatting can be combined with DLSS 5. Gaussian splatting has a lot of potential in video games yet to be realised.
This for some reason reminded me of the "Killerspiele" debate [1] we had in Germany after a dramatic school shooting. The shooter had previously built a map of the school in Counter-Strike. With this it's not a long stretch from there to having a realistic map of a school... Which would have given him a better rating than the one he got for his map: "I'd like to see the school that actually has lighting like this." [2]
Hopefully this tech will never used for something like this.
Question for those making Splats...how do you get such large environments? I've been playing around with them a bit and I'm finding I'm running out of memory with surprisingly little built even on an RTX6000. Any tips or ideas would be awesome!
Plays decently smooth on my M4 Max. It's probably still a long way from being a production-ready replacement for meshed environments, but I could imagine a hybrid mode where certain elements like grass and shrubbery are drawn with gaussians, perhaps with support for basic procedural animation. Great work with the playable demo!
Endless fields of grass and other things where you can make copies of a single base thing and just argue in some parameters like position, color, type etc are cheap to render. Making them sway or react to a body also isn't a problem.
For me, the biggest issue this solves is the blank canvas paralysis problem. Artists are visual thinkers and need a little nudge in the right (art) direction. This is a great way to fill that blank sheet of paper with something that they can take and run with.
Editing Gaussian Splats is still a pain in the ass in the artist's perspective. Even if you can create a good-enough first try using scanned data or generative AI, you just end up with a rough draft that you cannot polish in any way. Existing mesh-based tools allow you to edit the geometry relatively easily, since they are in a higher level discrete representation rather than just a point cloud data structure.
It seems to me that there's some overheated rhetoric that reminds me of the tech-specific spin on the Appeal to Novelty fallacy [1], where people think a new tech is going to uniformly improve on an old tech, that if it isn't an improvement on every front it is somehow a "failure", and therefore if we like the new tech and we are on Team New Tech that we must defend how the new tech is an improvement on every aspect.
Gaussian splats are definitely interesting and do something things older tech is not very good at, but at the same time, it's definitely going to end up being a tool in the tool chest and not completely murderating mesh-based tech or something because they have a lot of other weaknesses, like editability. Or dynamic animation.
What I think some people may not realize is, that's not particularly uncommon. There's a really, really long line of graphical techs that do something particularly well but their weaknesses have kept them in a limited use. It's not a problem for Gaussian splats to become a tool in the toolchest; they aren't a "failure" if we're still using meshes for a lot of things in 10 years.
Mesh-type techs are the "default" for some good reasons.
[1]: https://en.wikipedia.org/wiki/Appeal_to_novelty
There’s some movement in this area to be able to surface quantize the splats but you are right, right now it’s simply just visual language and isn’t useful in the pipeline.
Extracting a surface mesh is possible, but the result is going to be really ugly (like the high-poly meshes from generative AI that are useless to artists)!
Mesh processing is a very difficult research domain in computer graphics that has been iterated for several decades, and we still don't have a good automated solution for retopology (Partly because the problem is hard to define in a mathematical way, but also since it's not a problem you can just solve with AI by throwing data and compute at it)
I used voxelization of the splats in the past so I appreciate the notes on the difficulty but this is sort of what PlayCanvas is doing here. Taking the splats, making voxels, meshing.
It's a novel approach and worked well in BIM a few years ago, though not anything real-time.
https://github.com/ziplab/VolSplat
> but I could imagine a hybrid mode where certain elements like grass and shrubbery are drawn with gaussians
Those tend to move in the wind. Animations don't work well with splats. Or with any data structure except polygon meshes.
Edit:
Responding in the parent because HN says I'm "posting too fast" and should "slow down".
> Have you seen https://www.4dv.ai/
Yeah, but this seems to be just a 3D GS video (captured from several different camera angles), similar to how an ordinary 2D video is just a series of still frames. For 3D games this would be unsuitable since animations often have to be generated on the fly based on game physics. Even for pre-baked animations the memory cost of loading each frame individually would be too inefficient. For polygon meshes you have just a single static mesh that is deformed over time.
> Dreams managed to animate splats on the PS4. Admittedly, not quite the same type of splats, but there is probably a middle ground here where it can be made to work
I'm pretty sure Dreams only allowed animations as translations and rotations, not something that approximates soft skeletal animations. And even translations and rotations would be problematic since 3D GS scenes rely on baked lighting which would then result in objects no longer fitting the scene.
Have you seen https://www.4dv.ai/
> Animations don't work well with splats
Dreams managed to animate splats on the PS4. Admittedly, not quite the same type of splats, but there is probably a middle ground here where it can be made to work
Dreams uses SDFs though?
Thanks! Yeah hybrid is a way forward, dynamic stuff is not easy
I’m trying to understand from the video why this is better, it looks like a normal high resolution textures with precooked shadow maps.
It has no dynamic lighting or effects, which makes the video look like a high quality game from 2006.
dynamic objects are still largely unsolved problem, I just tried to approach it in this demo. also this particular place doesn't have reflective surfaces, but technology supports it - check for example this splat https://superspl.at/scene/ff1d0393 or this one https://superspl.at/scene/6c822f84
This is better, but I think that a demo with more reflections and radiosity would be much more impressive
Sort of unfortunate that one ends up putting normal meshed characters that clash with the photorealistic splat environment
Probably for the best as, well, they are being pumped with lead.
idealy it should be 4DGS but we are far away from it - real actors scan etc.. but somebody will do it later i am sure
Strikingly similar to the never completed Unrecord from 3 years ago: https://www.youtube.com/watch?v=IK76q13Aqt0
there another game, Bodycam, released on Steam - https://store.steampowered.com/app/2406770/Bodycam/ - but it's Unreal engine 5. This demo runs on my old 2020 iPhone, in browser
This tech reminds me of that Source Code(2011) movie for some reason.
How practical would it be to include LIDAR in the initial real-world environmental scan to get (or at least seed/constrain with real data samples) an even better collision mesh?
I'm looking forward to seeing what will happen when gaussian splatting can be combined with DLSS 5. Gaussian splatting has a lot of potential in video games yet to be realised.
This for some reason reminded me of the "Killerspiele" debate [1] we had in Germany after a dramatic school shooting. The shooter had previously built a map of the school in Counter-Strike. With this it's not a long stretch from there to having a realistic map of a school... Which would have given him a better rating than the one he got for his map: "I'd like to see the school that actually has lighting like this." [2] Hopefully this tech will never used for something like this.
[1] https://de.wikipedia.org/wiki/Killerspiel [2] https://www.spiegel.de/netzwelt/web/schuelerhobby-mapping-me...