3D Gaussian Splatting for Real-Time Radiance Field Rendering

“3D Gaussian Splatting for Real-Time Radiance Field Rendering”, by Bernhard Kerbl and Georgios Kopanas (equal contribution) and George Drettakis of Inria and Université Côte d’Azur, with Thomas Leimkühler of the Max-Planck-Institut für Informatik, was published in ACM Transactions on Graphics 42(4) and presented at SIGGRAPH 2023, where it was one of five Best Papers. A preprint followed on arXiv on 8 August 2023. The paper showed that a radiance field could be stored as an explicit set of anisotropic 3D Gaussians and rendered by GPU rasterization instead of ray marching a network, giving real-time rendering of unbounded real scenes at a quality the authors found comparable to the best NeRF variant of the time. This article covers the paper as published; the model, with its equations, training details and variants, is described in 3D Gaussian splatting. Details below come from the authors’ version of the TOG paper; arXiv v1 has the same text, equations and tables.

Problem

In 2023, neural radiance fields (NeRF paper) set the quality standard for novel view synthesis of captured scenes, and the authors framed the state of the art as a trade-off. The highest quality on unbounded scenes came from Mip-NeRF 360, which they report needs up to 48 hours of training. Faster methods, Instant-NGP with its hash grid and Plenoxels with its sparse voxel grid, trained in minutes but at lower quality and short of real-time rendering at high resolution. According to the abstract, no method rendered unbounded, complete scenes at 1080p in real time. The authors located the cost in the continuous representations themselves: volume rendering must place many samples along every ray, which is expensive and noisy, and the structured grids used for acceleration both limit image quality and still struggle to represent empty space (§1, §2.2).

The related-work section (§2) covers three lines of prior work. Image-based rendering built on structure from motion and multi-view stereo re-projects and blends the input photos, but cannot recover from geometry that MVS misses or invents. Neural rendering and radiance fields reduce those artifacts but pay for ray marching. Point-based rendering, from fixed-size points to splats of elliptical discs and surfels, is fast; recent neural point renderers decoded learned point features with a CNN at real-time rates but still needed MVS geometry, and the CNN could cause temporal flicker. The authors argue that point-based alpha blending and NeRF-style volume rendering share one image formation model, so the choice between them is a choice of rendering algorithm, not of the image model (§2.3). Pulsar’s sphere rasterizer inspired their tile-based renderer, and earlier 3D Gaussian models of human bodies and faces inspired the primitive, though those handled single objects with little depth complexity.

Contribution

The introduction lists three contributions:

• Anisotropic 3D Gaussians as a high-quality, unstructured representation of radiance fields.
• An optimization of the Gaussians’ properties interleaved with adaptive density control, which adds and removes Gaussians to produce high-quality representations of captured scenes.
• A fast, differentiable, visibility-aware renderer for the GPU that supports anisotropic splatting and fast backpropagation.

Two further points run through the paper. The method starts from the sparse points that SfM calibration produces anyway, where earlier point-based renderers needed MVS, and on the synthetic Blender scenes it works even from random points. The conclusion also states that, contrary to widely held opinion, a continuous representation is not strictly necessary for fast, high-quality radiance-field training (§8).

Method

Each Gaussian has a position, a covariance stored as a scale vector and a rotation quaternion so that it stays valid under gradient descent, an opacity, and view-dependent colour as spherical-harmonic coefficients (§4–§5). The Gaussians are initialized from the SfM points and optimized with Adam on a mix of L1 and D-SSIM losses, while every 100 iterations Gaussians in under-reconstructed regions are cloned, large ones in over-reconstructed regions are split, and nearly transparent ones are removed (§5, Appendix B). Rendering projects each Gaussian to a 2D splat with the EWA approximation of Zwicker et al., sorts all splats per 16 × 16 pixel tile with one GPU radix sort, and blends them front to back; the backward pass places no limit on how many splats per pixel receive gradients (§6). Gradients for all parameters are derived by hand (Appendix A); the rasterizer is written as custom CUDA kernels and the rest in PyTorch, and frame rates were measured in a viewer built on SIBR (§7.1, Appendices A–C).

Results

The evaluation covers 13 real scenes, all nine scenes of the Mip-NeRF 360 dataset, Truck and Train from Tanks and Temples, and Playroom and DrJohnson from Deep Blending, plus the eight synthetic Blender scenes, with one hyperparameter setting throughout. Every eighth photo of the real scenes is held out for testing, following Mip-NeRF 360. All timings are on an NVIDIA A6000 except Mip-NeRF 360’s, which the authors trained on a four-GPU A100 node for 12 hours and count as 48 hours on one GPU, noting that A100s are faster than A6000s (§7.2, footnote 2).

Table 1 compares two settings of 3DGS, 7,000 and 30,000 iterations, with Plenoxels, Instant-NGP in a Base configuration run for 35,000 iterations and a larger Big configuration, and Mip-NeRF 360, reporting SSIM, PSNR, LPIPS, training time, frames per second and memory per dataset. All competitor numbers come from the authors’ runs of the original code, except Mip-NeRF 360 on its own dataset, which is copied from its paper (their own run averaged PSNR 27.58, Appendix D). On the Mip-NeRF 360 scenes, the 7K setting reached PSNR 25.60 in 6 min 25 s at 160 frames per second, against 25.30 for Instant-NGP Base in 5 min 37 s at 11.7 frames per second, 25.59 for Instant-NGP Big in 7 min 30 s at 9.43, and 23.08 for Plenoxels in 25 min 49 s at 6.79. On Deep Blending the gap was larger: 27.78 for the 7K setting after 4 min 35 s, against 24.96 for Instant-NGP Big, the better of its two settings there. Fully trained models rendered at 154 frames per second on Tanks and Temples and 137 on Deep Blending. The price is storage: the 30K models took 411 MB on Tanks and Temples and 676 MB on Deep Blending, against 13 MB and 48 MB for the two Instant-NGP configurations and 2.1–2.7 GB for Plenoxels. The comparisons with Mip-NeRF 360 are given in 3D Gaussian splatting and 3D representations.

Table 2 reports PSNR on the synthetic scenes, starting from 100,000 random Gaussians that the optimization prunes to about 6,000–10,000 before growing to about 200,000–500,000 (§7.2), with competitors’ numbers taken from their papers. The average, 33.32, was 0.02 dB above Point-NeRF’s 33.30, and 3DGS had the highest score on three of the eight scenes (Chair, Drums, Hotdog). The trained synthetic scenes rendered at 180–300 frames per second. A separate compactness experiment against the point-based models of Zhang et al. (2022) matched their PSNR within 2–4 minutes and surpassed it with about a quarter of their points, 3.8 MB against 9 MB, using two spherical-harmonic degrees.

Table 3 ablates six choices on Truck, Garden and Bicycle at 5,000 and 30,000 iterations, using input images the authors downsampled themselves to reduce JPEG artifacts, so (in our reading) its values are not directly comparable with Table 1. Every ablation lowered the average PSNR at 30,000 iterations below the full model’s 26.05, least disabling cloning (25.91) and most limiting gradients to the ten front-most splats per pixel (19.19). At 30,000 iterations, random initialization cost between 4.6 dB (Bicycle) and 6.8 dB (Truck); the authors observe that it degrades mainly the background and leaves floaters in poorly covered areas (§7.3).

Impact

The paper was one of five Best Papers in the SIGGRAPH 2023 Technical Papers Awards, announced on the ACM SIGGRAPH Blog on 6 July 2023. The authors released their code and data with the paper. The official GitHub repository’s first commits are dated 4 July 2023, and as of October 2026 its licence, held by Inria and the Max Planck Institute for Informatics, permits non-commercial research and evaluation use only; the 3D Gaussian splatting article gives the pinned commit and independent implementations. The paper has been widely cited.

Limitations

The authors list the following limitations (§7.4, §8):

• Poorly observed regions produce artifacts, as they do for Mip-NeRF 360 (Figs. 11–12), and the anisotropic Gaussians can form elongated or “splotchy” artifacts in such regions.
• Popping occurs occasionally when the optimization creates large Gaussians, often where appearance is view-dependent. The authors blame the guard band that rejects Gaussians in the rasterizer and the simple visibility algorithm, which can suddenly swap blending order; they suggest antialiasing and regularization, neither of which the method applies.
• Very large scenes, such as urban datasets, may need a lower position learning rate, according to early experiments.
• Memory is significantly higher than for NeRF-based methods: peak training memory can exceed 20 GB for large scenes in the unoptimized prototype, and rendering needs the full model, several hundred megabytes for large scenes, plus 30–500 MB for the rasterizer. They point to point-cloud compression as a possible remedy.
• Implementation: about 80% of training time is spent in Python code; a full CUDA port could be faster.
• Geometry: whether the Gaussians can yield meshes is left open.

Discrepancies in the paper

Four printed details are inconsistent with the rest of the paper; all appear in both the authors’ TOG version and arXiv v1. In Equation (2), the transmittance product over $j$ is printed with $\alpha_i$ instead of $\alpha_j$. Equation (4) prints the Gaussian with $x$ in place of $x - \mu$, although the text says it is centred at $\mu$. The introduction gives 1–5 million Gaussians “for all scenes tested” (§1), while §7.2 gives about 200,000–500,000 for the synthetic scenes. And §7.2 states a training time of 35–45 minutes, while Table 1 lists 41 min 33 s, 26 min 54 s and 36 min 2 s for the three real datasets, so Tanks and Temples falls below the stated range.

What Came After

Later work took up the paper’s open problems; 3D Gaussian splatting describes these variants with their results. Mip-Splatting (Yu et al., CVPR 2024) took up the aliasing that appears when a model is rendered at a different sampling rate than it was trained on, with a 3D smoothing filter and a 2D Mip filter. Surface-oriented variants such as 2D Gaussian splatting (Huang et al., SIGGRAPH 2024), which flattens the primitives into discs, take up the paper’s open question about meshes. Other lines of work compressed the models, the memory limitation the authors had flagged, extended them to dynamic scenes, predicted Gaussians from images in a single forward pass, and used them as maps for SLAM.