Research Assistant at UPenn CIS

Philadelphia
Excited to share our new ๐—ผ๐—ฝ๐—ฒ๐—ป-๐˜€๐—ผ๐˜‚๐—ฟ๐—ฐ๐—ฒ work, ๐—›๐—”๐—ฅ๐— ๐—ข๐—ก๐—ฌ, a framework for reconstructing compositional 3D scenes from single indoor images. Much like a 3D artist, our VLM agent builds the scene in a ๐—ต๐—ถ๐—ฒ๐—ฟ๐—ฎ๐—ฟ๐—ฐ๐—ต๐—ถ๐—ฐ๐—ฎ๐—น ๐—ผ๐—ฟ๐—ฑ๐—ฒ๐—ฟ: first the room structure and camera, then wall-mounted objects, free-standing furniture, and finally dependent decorations. After each stage, it compares its render against the input image, reflects on what went wrong, and iteratively corrects the scene to prevent errors from accumulating. Even with older VLM backbones, our framework remains ๐—ฐ๐—ผ๐—บ๐—ฝ๐—ฒ๐˜๐—ถ๐˜๐—ถ๐˜ƒ๐—ฒ ๐˜„๐—ถ๐˜๐—ต ๐—š๐—ฃ๐—ง-๐Ÿฒ ๐—”๐˜€๐˜๐—ฟ๐—ฎ ๐—ถ๐—ป ๐˜ƒ๐—ถ๐˜€๐˜‚๐—ฎ๐—น ๐—พ๐˜‚๐—ฎ๐—น๐—ถ๐˜๐˜†, ๐˜„๐—ต๐—ถ๐—น๐—ฒ ๐—ฎ๐—ฐ๐—ต๐—ถ๐—ฒ๐˜ƒ๐—ถ๐—ป๐—ด ๐—ฏ๐—ฒ๐˜๐˜๐—ฒ๐—ฟ ๐—ด๐—ฒ๐—ผ๐—บ๐—ฒ๐˜๐—ฟ๐—ถ๐—ฐ ๐—ฎ๐—น๐—ถ๐—ด๐—ป๐—บ๐—ฒ๐—ป๐˜ for single-image 3D scene reconstruction. We also release ๐—›๐—”๐—ฅ๐— ๐—ข๐—ก๐—ฌ๐Ÿฏ๐Ÿฌ๐Ÿฌ, a benchmark for single-image 3D scene reconstruction with 300 scenes across three difficulty tiers: easy, medium, and complicated. ๐Ÿ”— Project: cwchenwang.github.io/harmony ๐Ÿ“Š Leaderboard: cwchenwang.github.io/harmonyโ€ฆ ๐Ÿค— Data: huggingface.co/datasets/Shufโ€ฆ ๐Ÿ“„ Paper: arxiv.org/abs/2609.26793
6
42
307
30,348