Spectacular AI: Deblurring Gaussians

I have said it time and time again, but nothing kills a radiance field faster than blurry images. Or so it's been so far. There have been a couple efforts popping up to make blurry images usable and disprove the often cited film maxim, garbage in, garbage out.

Unlike the high-end equipment used by professional cinematographers, most people today capture radiance fields with what's readily available—namely, their smartphones. Traditionally, creating high-quality 3D scene reconstructions and novel view syntheses required steady, high-quality photographs—a tall order for handheld devices prone to motion blur, rolling shutter effects, and pose noise.

Most cameras operate with rolling shutters, meaning their sensors capture data in a sequential roll across the frame. Although the delay between each segment is just microseconds, it can still significantly impact accuracy.

These phenomena, stemming from the camera's and objects' relative motion and the camera sensors' line-by-line scanning, can degrade the quality of reconstructed scenes. "Gaussian Splatting on the Move" presents a breakthrough by leveraging camera motion for high-quality reconstruction from handheld video data.

Their method models the physical image formation process by taking into account the camera shutter’s exposure time and velocity estimated using visual-inertial odometry (VIO) during image capture. Modelling these effects during 3DGS training can recover sharpened versions of input images, and can also be used to simulate motion blur and rolling shutter effects in the 3D Gaussian Splatting (3DGS) framework as a post-processing effect.

Essentially, they're using the camera's own movement to help make the final image much clearer. The proposed differentiable rendering pipeline efficiently incorporates these effects, demonstrating superior performance over traditional methods in both synthetic and real-world scenarios.

The paper comes from Spectacular AI, who provides an SDK for low-level embedded spatial AI, easy tools, scripts, and code for processing data captured with iPhones, Android devices, Kinect devices, and other RGB-D sensors used in robotics and industry.

The research was also conducted in part by Matias Turkulainen who is also one of the authors of Nerfstudio's Splatfacto, so it shouldn't be a surprise that they have implemented their method as an extension to the Nerfstudio and gsplat, providing a practical tool for advancing 3DGS in naturalistic settings.

Spectacular AI's compatibility with Nerfstudio simplifies adoption. Designed with the consumer in mind, the method was trained on a GPU with 16 GB of VRAM. It operates efficiently, completing tasks in about 20 minutes—marginally longer than Splatfacto under normal conditions. The code is available for immediate use with Nerfstudio's Splatfacto method. In order to capture usable data, you need to use Spectacular AI's free app on either iOS or Android.

With the amount of capture based startups that have emerged over the course of the last year, I imagine that this problem is high up on their lists. By raising the floor of consumer capture, a larger band of people will see success from what might have started as unusable footage. Further, if there are ways to restore and capture more details, I think we can all agree that it makes sense to pursue it.