Dense Dynamic Scene Reconstruction and Camera Pose Estimation from Multi-View Videos
INFORMATION
Authors: Shuo Sun, Unal Artan, Malcolm Mielle, Achim J. Lilienthal, Martin Magnusson
Status: arXiv pre-print
We address the challenging problem of dense dynamic scene reconstruction and camera pose estimation from multiple freely moving cameras — a setting that arises naturally when multiple observers capture a shared event.
Prior approaches either handle only single-camera input or require rigidly mounted, pre-calibrated camera rigs, limiting their practical applicability.
Method
We propose a two-stage optimization framework that decouples the task into robust camera tracking and dense depth refinement.
- Multi-camera tracking: we extend single-camera visual SLAM to the multi-camera setting by constructing a spatiotemporal connection graph that exploits both intra-camera temporal continuity and inter-camera spatial overlap, enabling consistent scale and robust tracking. A wide-baseline initialization strategy using feed-forward reconstruction models ensures robustness under limited overlap.
- Dense refinement: we refine depth and camera poses by optimizing dense inter- and intra-camera consistency using wide-baseline optical flow.
We also introduce MultiCamRobolab, a new real-world dataset with ground-truth poses from a motion capture system.
Results
Our method significantly outperforms state-of-the-art feed-forward models on both synthetic and real-world benchmarks, while requiring less memory.
What’s next
- Multi-camera dynamic reconstruction is a natural fit for inspection robots with several sensors — combining it with thermal modalities (see SEAR) is an open direction.
- The spatiotemporal connection graph assumes overlapping fields of view; handling fully disjoint viewpoints remains open.