NVIDIA Research Scientist - Generative Graphics

Zurich
Filter
Exclude
Time range
-
Minimum likes
Peter Kocsis retweeted
Congrats to @LukasHollein for his successful PhD defense! Lukas’s thesis is on π†πžπ§πžπ«πšπ­π’π§π  πŸ‘πƒ 𝐖𝐨𝐫π₯𝐝𝐬 𝐟𝐫𝐨𝐦 πŸπƒ π†πžπ§πžπ«πšπ­π’π―πž 𝐌𝐨𝐝𝐞π₯𝐬. His most impactful work so far, Text2Room (ICCV’23 Oral), was the first method to generate complete room-scale 3D scenes with textured geometry from text alone. The key idea is to lift pre-trained 2D text-to-image models into 3D: iteratively generate new views, estimate depth, and fuse everything into a seamless textured mesh, with a carefully designed viewpoint selection to complete the scene. This inspired many others on text-to-3D scene generation and led to a long line of follow-up works, including Ctrl-Room, LucidDreamer, WonderJourney, WorldGen, and many more. Today, the Text2Room paradigm of leveraging 2D priors has become the primary approach for both image and video diffusion models to generate 3D scenes. However, Lukas has done much more cutting-edge research. ViewDiff turns text-to-image models into 3D-consistent multi-view generators, WorldExplorer generates fully navigable 3D scenes from video diffusion models, and 3DGS-LM speeds up Gaussian Splatting optimization with Levenberg-Marquardt, and most recently World Reconstruction From Inconsistent Views. In summary, his PhD has helped define how we generate 3D worlds with generative AI. Lukas also keeps pushing the boundaries of 3D world generation. He’s now continuing his research at Apple and I'm sure we can expect more great research very soon! Check his great research: lukashoel.github.io Congrats Dr. HΓΆllein, you will be deeply missed in the lab!
6
10
150
8,094
Happy to share that I’ve successfully defended my PhD! πŸŽ“ Excited to continue this research at NVIDIA and explore what comes next in generative graphics!
Congrats to @Peter4AI for his successful PhD defense! Peter’s thesis is on π†πžπ§πžπ«πšπ­π’π―πž 𝐏𝐫𝐒𝐨𝐫𝐬 𝐟𝐨𝐫 𝐌𝐚𝐭𝐞𝐫𝐒𝐚π₯ π„π¬π­π’π¦πšπ­π’π¨π§ 𝐚𝐧𝐝 𝐋𝐒𝐠𝐑𝐭𝐒𝐧𝐠 𝐂𝐨𝐧𝐭𝐫𝐨π₯. His arguably most impactful work, Intrinsic Image Diffusion, was among the first to bring diffusion models to material estimation. The key idea is that intrinsic decomposition is inherently ambiguous, so rather than regressing a single answer, it is formulated probabilistically and samples plausible decompositions, leveraging the strong image priors of diffusion models. This opened up a new line of diffusion-based inverse rendering work, including IntrinsicAnything, Marigold-IID, DiffusionRenderer, and many more. Today, generative priors are the standard approach for material estimation and relightable reconstruction. However, Peter has done much more cutting-edge research: LightIt brought explicit lighting control to diffusion models, IntrinsiX enabled direct PBR generation from text, and Intrinsic Image Fusion lifts these priors to high-quality multi-view 3D material reconstruction. In summary, his PhD has shaped how we think about materials and lighting in the age of generative AI. I’m also particularly proud that Peter is continuing this line of research as a Research Scientist at NVIDIA, e.g., his new work on Texture Space Material Diffusion and NeuMatEx. Also thanks to Ruediger Westermann and George Drettakis for being on the committee. Check out Peter's fantastic work: peter-kocsis.github.io Congrats Dr. Kocsis, it was an honor to have you in the lab!
4
1
31
878
Replying to @MattNiessner
Thanks Matthias for the kind words, your mentorship, and all the support along the way! Grateful for the years in the lab, the collaborators, and everyone who made this journey possible. Research never stops - Excited to keep exploring what’s next in graphics! πŸŽ“
1
4
189
πŸ“’ Texture Space Material Diffusion We generate relightable materials from photos or text using video diffusion priors directly in texture space, scaling to 8K textures. Great project with Jacob Munkberg (@yoslber) and Jon Hasselgren! nvlabs.github.io/texdiffusio…
3
11
99
6,302
πŸ“’NeuMatEx - Material Reconstruction beyond PBR We extract neural materials from multi-view images using a Large Material Reconstruction Model combined with differentiable rendering, Great project from @kim_youwang! nvlabs.github.io/neumatex/
1
7
45
2,967
Excited to share that I’ve joined NVIDIA as a Research Scientist in Generative Graphics! πŸš€ Looking forward to working on generative rendering and the next steps in graphics with an incredible team. Grateful to everyone who helped along the way! #NVIDIAlife
24
1,450
We present Intrinsic Image Fusion in Poster Session 4 (106) on Saturday at #CVPR πŸ₯³ Come by to learn more about room-scale material estimation! PS: Code is released πŸš€
πŸ“’ Intrinsic Image Fusion for Multi-View 3D Material Reconstruction πŸ“’ We combine generative material priors with inverse path tracing: 1) define a parametric texture space 2) fuse monocular predictions across views into consistent textures 3) optimize low-dimensional parameters for physically-grounded reconstructions. The results are relightable PBR textures for 3D scenes: check out the result on a real-world 3D scan from the ScanNet++ dataset! 🌍peter-kocsis.github.io/Intri… πŸŽ₯piped.video/-Vs3tR1Xl7k Great work by @Peter4AI @LukasHollein!
6
40
7,113
Peter Kocsis retweeted
πŸ“’ 3D world models from video diffusion suffer from inconsistent frames -> blurry output. Our fix: instead of naΓ―ve 3D reconstruction, we non-rigidly align each frame into a globally-consistent 3DGS representation. ->sharp visuals on top of any VDM! lukashoel.github.io/video_to…
6
81
519
53,519
Peter Kocsis retweeted
πŸ“’πŸ“’πŸ“’Data release: high-res, multi-view, OLAT face recordings πŸ“’πŸ“’πŸ“’ We captured individuals in our custom light stage with 16 high-end, global shutter cameras (72 fps) and 40 LED modules, totaling 2.8M precisely calibrated frames. We us the data for BecomingLit (#NeurIPS2025): intrinsically decomposed Gaussian avatars, enabling photorealistic and real-time relighting via hybrid neural shading. Code & Data: jonathsch.github.io/becoming… Great work by @jnthnschmdt, @SGiebenhain
5
40
236
19,553
πŸ“’Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single ImageπŸ“’ We directly regress neural parametric head models (NPHMs) from a single image β€” fast, stable, and significantly more expressive than classical 3DMMs such as FLAME. Face tracking & 3D reconstruction are often limited by the representational capacity of PCA-based face models. By lifting NPHMs to a first-class reconstruction primitive, we enable more accurate geometry, richer expressions, and finer animation control. Pix2NPHM obtains fast and reliable NPHM reconstructions on real-world data. Inference-time optimization against surface normals and canonical point maps can further increase fidelity. Key to successful and generalized training of our ViT-based network are: (1) large-scale registration of existing 3D head datasets, and (2) self-supervised training on vast in-the-wild 2D video datasets using pseudo ground-truth surface normals. Finally, we show that geometry-aware pretraining on pixel-aligned reconstruction tasks significantly outperforms generic visual pretraining (e.g., DINO-style features) in terms of generalization. 🌍simongiebenhain.github.io/Pi… πŸŽ₯piped.video/MgpEJC5p1Ts Great work by @SGiebenhain, @TobiasKirschst1, @liamschoneveld, Davide Davoli, Zhe Chen
15
79
536
38,037
πŸ“’πŸ“’πŸ“’πŒπžπ¬π‘π‘π’π©π©π₯𝐞: Structured Autoregressive Generation of Artist-Meshes High-fidelity, topologically complete 3D assets that expand naturally like a ripple on a surface! 🌊 Existing AR models often rely on sliding-window inference over truncated segments. However, this limitation breaks long-range geometric dependencies, causing holes and fragmentation. Instead, MeshRipple uses frontier-aware BFS and sparse-attention global memory to ensure coherent growth with an unbounded receptive field. -> Highly detailed-mesh generations -> Artist-like meshing quality -> Works on room-scale environments 🌍maymhappy.github.io/MeshRipp… πŸŽ₯piped.video/AHvaLslzXQU Great work by Junkai Lin, Hang Long, Huipeng Guo, Jielei Zhang, Jiayi Yang, Tianle Guo, Yang Yang, Jianwen Li, Wenxiao Zhang, Wei Yang
5
100
649
34,173
For the occluded regions we don't have any predictions, so now they don't receive gradients, but one option would be to backprop through multiple bounces.
49
πŸ“’ Intrinsic Image Fusion for Multi-View 3D Material Reconstruction We reconstruct clean and sharp relightable textures using inverse path tracing and monocular priors. Check out our project page for more results: peter-kocsis.github.io/Intri…
πŸ“’ Intrinsic Image Fusion for Multi-View 3D Material Reconstruction πŸ“’ We combine generative material priors with inverse path tracing: 1) define a parametric texture space 2) fuse monocular predictions across views into consistent textures 3) optimize low-dimensional parameters for physically-grounded reconstructions. The results are relightable PBR textures for 3D scenes: check out the result on a real-world 3D scan from the ScanNet++ dataset! 🌍peter-kocsis.github.io/Intri… πŸŽ₯piped.video/-Vs3tR1Xl7k Great work by @Peter4AI @LukasHollein!
11
714
Releasing Echo today is incredibly exciting for me β€” because it is a critical step for generative AI, enabling the creation of virtual worlds. Echo is our first world model at SpAItial AI. It turns text or images into explorable 3D environments β€” spaces you can move through, inspect, and build on. Seeing this work in real time still feels a bit surreal. My fascination with this goes back a long way: video games, virtual environments, and the idea of capturing the real world in 3D. As a researcher, I spent years working on 3D reconstruction, neural rendering, and scene understanding β€” all driven by the same question: how do we teach machines to understand the world? One thing became clear over time: the biggest bottleneck isn’t compute or rendering β€” it’s 3D worlds themselves. High-quality, consistent environments are expensive to create by hand and don’t scale to the experiences we want to build. In particular, I believe that the ability to generate virtual worlds is ultimately key towards understanding the real world. That’s why we founded SpAItial AI. We’re building spatial world models that combine geometric understanding with creative generation β€” models that can generate, edit, and eventually reason about 3D environments. Echo is just the beginning. For me, this feels like the moment when decades of research finally meet the imagination that got many of us into graphics, games, 3D understanding in the first place.🌍 spaitial.ai/
πŸš€ Announcing Echo β€” our new frontier model for 3D world generation. Echo turns a simple text prompt or image into a fully explorable, 3D-consistent world. Instead of disconnected views, the result is a single, coherent spatial representation you can move through freely. This is part of a bigger shift in AI: from generating pixels and tokens to generating spaces. Echo predicts a geometry-grounded 3D scene at metric scale, meaning every novel view, depth map, and interaction comes from the same underlying world β€” not independent hallucinations. Once generated, the world is interactive in real time. You control the camera, explore from any angle, and render instantly β€” even on low-end hardware, directly in the browser. High-quality 3D world exploration is no longer gated by expensive equipment. Under the hood, Echo infers a physically grounded 3D representation and converts it into a renderable format. For our web demo, we use 3D Gaussian Splatting (3DGS) for fast, GPU-friendly rendering β€” but the representation itself is flexible and can be easily adapted. Why this matters: consistent 3D worlds unlock real workflows β€” digital twins, 3D design, game environments, robotics simulation, and more. From a single photo or a line of text, Echo builds worlds that are reliable, editable, and spatially faithful. Echo also enables scene editing and restyling. Change materials, remove or add objects, explore design variations β€” all while preserving global 3D consistency. Editing no longer breaks the world. This is only the beginning. Echo is the foundation for future world models with dynamics, physical reasoning, and richer interaction β€” environments that don’t just look right, but behave right. Explore the generated worlds on our website and sign up for the closed beta. The era of spatial intelligence starts here. 🌍 #Echo #WorldModels #SpatialAI #3DFoundationModels Check it out: spaitial.ai/
24
100
627
71,361
Peter Kocsis retweeted
Radiance Meshes for Volumetric Reconstruction πŸŽ‰ We made a thing! A Radiance Field that is composited out of a triangle meshes that renders accurately the underlying volumetric field, with no popping AND faster than approximate methods like Gaussian Splatting.
5
37
306
26,525
πŸ“’ π™„π™£π™©π™§π™žπ™£π™¨π™žπ™“ @NeurIPSConf If you are excited about PBR materials, drop by at Wednesday from 4:30pm to 7:30pm (poster 4308), or feel free to dm me. PS: We have just released the training code, so you can also train your own models now! peter-kocsis.github.io/Intri…
πŸ“’ IntrinsiX: High-Quality PBR Generation using Image Priors πŸ“’ From text input, we generate renderable PBR maps! Next to editable image generation, our predictions can be distilled into room-scale scenes using SDS for large-scale PBR texture generation. We first train separate LoRA modules for the intrinsic properties of albedo, rough/metal, normal. Then, we introduce cross-intrinsic attention using a rerendering loss with importance-weighted light sampling to enable coherent PBR generation. Our method outperforms text -> image -> PBR methods both in generalization and quality, since directly generating PBR maps does not suffer from the inherent ambiguity of intrinsic image decomposition. In addition, our design choice facilitates SDS-based PBR texture distillation. 🌍 peter-kocsis.github.io/Intri… πŸŽ₯ piped.video/b0wVA44R93Y Great work by @Peter4AI, @LukasHollein
2
9
1,263
Peter Kocsis retweeted
FLUX.2 is here - our most capable image generation & editing model to date. Multi-reference. 4MP. Production-ready. Open weights. Into the new.
161
458
2,925
485,260
Congrats @yawarnihal! Amazing work, really well-deserved!
Congrats to @yawarnihal for winning the @MdsiTum best paper award for his amazing πŒπžπ¬π‘π†ππ“ workπŸŽ‰ MeshGPT autoregressively generates compact, artist-style triangle meshes by tokenizing faces into a learned discrete vocabulary (VQ-style codebook) and training a decoder-only transformer to predict those face tokens β€” because discrete tokenization + attention lets GPT-style models learn long-range geometric & topological patterns and produce coherent, high-fidelity 3D assets. MeshGPT's use cases go far beyond traditional content creation applications in computer graphics. For instance, the method was developed in collaboration with @Audi to help rapid prototyping of car designs, where explicit and precise mesh design are essential. In the research community, there have already been many follow ups such as MeshAnything, MeshXL, Meshtron, and many more - finally, we can use AI to generate high-fidelity 3D content :) Project: nihalsid.github.io/mesh-gpt/ Video: piped.video/UV90O1_69_o
1
3
703
Peter Kocsis retweeted
πŸ“’ProcGen3D: Learning Neural Procedural Graphs for Image-to-3D Reconstruction @xinyi092298 learns neural procedural graphs to generate high-fidelity 3D - MCTS-guided sampling maintains consistency with the input image, even from real images! Check it out: xzhang-t.github.io/project/P…
8
53
238
14,717