It will be interesting to see the first AAA game that uses these methods instead of rendering a 3D world. Even if made from CGI worlds, it would be a very interesting approach and with somewhat predictable performances.
Reminds me of Ecstatica [1], a 1994 game that had intense visuals with a very odd/different rendering engine made of 3D ellipsoids; in a way really crude splats in gouraud shading.
This is "rendering a 3D world". It's basically the exact same techniques that traditional rendering uses, just with a different primitive that's not triangles. Everything else pretty much carries over.
If you mean the technique of splatting specifically, Dreams for PS4 [1] is prior art.
If you mean pre-rendering, there's Myst and games like the original FF7 for PS1.
The contributions of 3DGS lie in how fast you can make them in modern GPU hardware (tiling + sorting with threads), and how to make the pipeline differentiable so that you can fit the Gaussian splats with photogrammetry data. Similar to the history of deep learning, it became technically feasible once the GPU hardware was powerful enough.
Many years ago there was a game called Casebook[1], a small little detective game where you investigated rooms for clues. But unlike similar FMV games where you jumped from point to point, it had photorealistic environments that could be smoothly walked around in, much like later lightfield or gaussian splatting experiments.
Can someone point to a resource/tutorial for learning point splatting (the 90s rendering technique)? Gaussian Splatting has completely over taken the search results, and the original technique is now near impossible to find.
I love this site design. It uses the entire width of the monitor rather than a slender column of pixels down the middle with large blocks of unused space on either side, with a font for my old man eyes.
Really nice idea for 3DGS rendering - though the main problem is the noise (an unfortunate issue for all Monte-Carlo based methods).
I think future papers would probably continue improving on this method and focus on how to sample the points more efficiently while being unbiased (similar to how ray-tracing solved their performance issues). Or maybe... we can just add a deep-learning based denoiser and call it a day!
It's not new - that was sort of my point with my other comment.
At least if it's progressive (so refines and resolves over time), this has been done with pointclouds in the VFX industry in GPU shaders for years in terms of stochastically drawing different points so eventually the whole point set gets rasterised to a fidelity threshold.
ookay, thanks for the clarification! So, the interesting part here seems to be the 3DGS-specific opacity correction and GPU workload mapping. Am I wrong?
Cannot find anything related to Monte Carlo methods in the source code. I thought Spark implemented a conventional 3DGS pipeline with LoD optimizations (And it seems they do the sorting on the CPU using Rust/WebAssembly because of WebGL limitations)
Their point splatting method is orthogonal to level-of-detail rendering (they reference a few papers which try to do this), so both point splatting and LoD could be combined in the future for an even greater performance gain during rendering. They already implement occlusion and frustum culling.
Point splatting does introduce a lot of noise though, and their denoiser introduces ghosting, but they say a more sophisticated denoiser would give considerably better quality.
Sorting the gaussians is the compute heavy part in gaussian splatting. So, Im guessing this will give only marginal improvement in terms rendering speed.
Really?! What OSs can handle that many native threads?
Also, this seems quite similar to stochastic progressive drawing of pointclouds for realtime that has been done for > 15 years in the VFX industry with GPU shaders in a tiled/bucketed fashion, unless this isn't progressive maybe? (The fact it's been accepted for Siggraph likely indicates it's slightly different).
It will be interesting to see the first AAA game that uses these methods instead of rendering a 3D world. Even if made from CGI worlds, it would be a very interesting approach and with somewhat predictable performances.
Reminds me of Ecstatica [1], a 1994 game that had intense visuals with a very odd/different rendering engine made of 3D ellipsoids; in a way really crude splats in gouraud shading.
[1] https://ecstatica.fandom.com/wiki/Ecstatica
There was this FPS demo recently https://playcanv.as/p/qxGSuzYq/
People have also converted some small sections of Unreal 5 demos into splats https://superspl.at/scene/692c4f91
Or perhaps use a real world scan - it was suggested this one would make an ideal setting for zombies https://superspl.at/scene/6359774f
This is "rendering a 3D world". It's basically the exact same techniques that traditional rendering uses, just with a different primitive that's not triangles. Everything else pretty much carries over.
If you mean the technique of splatting specifically, Dreams for PS4 [1] is prior art.
If you mean pre-rendering, there's Myst and games like the original FF7 for PS1.
[1] https://en.wikipedia.org/wiki/Dreams_(video_game)
Note that the first published work of rendering Gaussian Volumes was in this 1991 paper (https://articles.tomasparks.name/publications/Westover1991.p...) - so 3DGS is really a rehash of an old method from the 90s!
The contributions of 3DGS lie in how fast you can make them in modern GPU hardware (tiling + sorting with threads), and how to make the pipeline differentiable so that you can fit the Gaussian splats with photogrammetry data. Similar to the history of deep learning, it became technically feasible once the GPU hardware was powerful enough.
Many years ago there was a game called Casebook[1], a small little detective game where you investigated rooms for clues. But unlike similar FMV games where you jumped from point to point, it had photorealistic environments that could be smoothly walked around in, much like later lightfield or gaussian splatting experiments.
[1] https://www.youtube.com/watch?v=o-VAaC5BgVE
Can someone point to a resource/tutorial for learning point splatting (the 90s rendering technique)? Gaussian Splatting has completely over taken the search results, and the original technique is now near impossible to find.
Westover’s thesis https://www.cs.unc.edu/techreports/91-029.pdf
It's going to be even more impossible to find now because the present paper introduces "Gaussian point splatting".
Could this be a new direction for Google Streetview perhaps?
I love this site design. It uses the entire width of the monitor rather than a slender column of pixels down the middle with large blocks of unused space on either side, with a font for my old man eyes.
<3
> It uses the entire width of the monitor rather than a slender column of pixels down the middle with large blocks of unused space on either side
Umm on my machine it has 560px margin on both sides with the content being only 474px sliver in the middle?
Imo they need to pad it just a bit. My scrollbar overlaps.
Maybe use Tampermonkey?
Really nice idea for 3DGS rendering - though the main problem is the noise (an unfortunate issue for all Monte-Carlo based methods).
I think future papers would probably continue improving on this method and focus on how to sample the points more efficiently while being unbiased (similar to how ray-tracing solved their performance issues). Or maybe... we can just add a deep-learning based denoiser and call it a day!
This feels like Monte Carlo rendering applied to rasterization. I'm wondering if it's a brand-new or a well established methodology
It's not new - that was sort of my point with my other comment.
At least if it's progressive (so refines and resolves over time), this has been done with pointclouds in the VFX industry in GPU shaders for years in terms of stochastically drawing different points so eventually the whole point set gets rasterised to a fidelity threshold.
ookay, thanks for the clarification! So, the interesting part here seems to be the 3DGS-specific opacity correction and GPU workload mapping. Am I wrong?
Possibly yeah.
Or the per-pixel coord atomic I guess?
Right, that part seems to be based on Schütz et al. 2021 https://arxiv.org/abs/2104.07526
Monte Carlo in 3dgs is established enough that Spark [1] has been doing it for a while in the browser.
https://github.com/sparkjsdev/spark
Cannot find anything related to Monte Carlo methods in the source code. I thought Spark implemented a conventional 3DGS pipeline with LoD optimizations (And it seems they do the sorting on the CPU using Rust/WebAssembly because of WebGL limitations)
Their point splatting method is orthogonal to level-of-detail rendering (they reference a few papers which try to do this), so both point splatting and LoD could be combined in the future for an even greater performance gain during rendering. They already implement occlusion and frustum culling.
Point splatting does introduce a lot of noise though, and their denoiser introduces ghosting, but they say a more sophisticated denoiser would give considerably better quality.
Sorting the gaussians is the compute heavy part in gaussian splatting. So, Im guessing this will give only marginal improvement in terms rendering speed.
I'm not sure it does a sort. Each group of threads only handles a select number of gaussians
Yea, I think avoiding sorting is kinda the whole point here
> millions of threads
Really?! What OSs can handle that many native threads?
Also, this seems quite similar to stochastic progressive drawing of pointclouds for realtime that has been done for > 15 years in the VFX industry with GPU shaders in a tiled/bucketed fashion, unless this isn't progressive maybe? (The fact it's been accepted for Siggraph likely indicates it's slightly different).
I believe they mean GPU threads. Plenty of cuda files in their repository.
Fair enough, but that's then only absolutely max 1024 threads per SM, which wouldn't get anywhere near 1 million, given 5090 only has 192 SMs...
Future proofing I guess...
You can launch much more logical threads than the available physical threads. The GPU scheduler will automatically dispatch the work to the SMs.
I guess they never say that they execute at the same time technically haha