Next-Best-View 3D Gaussian Splatting with an Autonomous Mobile Robot

Overview

A ROS 2 + PyTorch system that closes the loop between robotic exploration and 3D reconstruction: a Hello Robot Stretch 3 autonomously decides where to look next, physically navigates there, and feeds each new observation back into a live 3D Gaussian Splatting model, repeating until the reconstruction stops meaningfully improving. Viewpoint selection is driven by a Shannon mutual-information scoring function over the Gaussian scene representation, explicitly choosing the views that resolve the most uncertainty.

System Architecture

System Architecture

Hardware

Intel RealSense D435i

The onboard RGB-D camera mounted on the Stretch 3’s head, providing the color images and depth maps used both for live capture and for seeding each new region of the Gaussian point cloud.

Hello Robot Stretch 3

The mobile manipulator platform. Its head pan/tilt joints aim the D435 at a requested 6-DOF pose, and its mobile base (driven via Nav2) carries it to the standoff position for each next-best-view candidate before the head IK takes over for final aiming.

Stretch 3

Results & Validation

To validate that Shannon mutual-information scoring is actually driving the exploration, I ran two identical 11-round pipelines(one that uses the scoring mechanism to determine the next capture pose and another that randomly samples from the candidate poses to determine the next capture pose), holding everything else fixed (same initial images, same holdout poses, same training budget). First, the model’s own top-candidate uncertainty score fell 82.7% over 11 rounds under scored selection, but rose 75.8% under random selection, confirming that without targeted selection, the scene’s worst-covered regions are never visited. Second, real image quality follows suit: on 3 fixed held-out views never trained on by either run, scored selection held a consistent ~0.3 dB PSNR edge across most rounds (mean 17.90 dB vs. 17.65 dB), and at the newly-captured pose itself, an informed round-0 pick resolved more than twice the reconstruction error of a random one (+10.40 dB vs. +4.52 dB), a gap that narrows by round 10 (+5.99 dB vs. +5.59 dB) as overall scene coverage saturates for both strategies.

PSNR values for scored versus random NBV selection

Uncertainty values for scored versus random NBV selection

Rendering the same camera pose from the model checkpoint immediately before and after that pose was captured and trained on shows the reconstruction resolving new geometry.

Scored selection:

Random baseline:

The consistent spike in Peak Signal to Noise Ratio(PSNR) across multiple rounds shows the uncertain geometry in the scene being resolved.

Media

3DGS Renderings

The first image below is a rendering of the top scoring candidate pose rendered from the initial model. The second image was rendered from the next model checkpoint, after an image at that pose was captured by the robot and the model trained with that image added to the dataset. The third image is the actual RGB photo captured by the onboard D435i camera. Plainly the additional capture was able to resolve much of the uncertainty that the initial model failed to do.

Rendered pose - initial model

Rendered pose - updated model

Actual RGB image

Video demonstrations

Top Scoring Pose After Model Retraining

Initial versus Optimized Model

Citations

This project’s Shannon-MI viewpoint scoring is a plain-PyTorch reimplementation based on the GauSS-MI approach.

Y. Xie, Y. Cai, Y. Zhang, L. Yang, and J. Pan, “GauSS-MI: Gaussian Splatting Shannon Mutual Information for Active 3D Reconstruction,” arXiv preprint arXiv:2504.21067, 2025.