PhD Student at Carnegie Mellon University | Computer Vision and Language | Generative AI
Pittsburgh
- Tweets 100
- Following 436
- Followers 768
- Likes 294
Pinned Tweet
Before AI can generate professional videos, it needs to see like a professional.
We spent a year with 100+ content creators teaching AI to describe video like a filmmaker would.
Introducing CHAI: Critique-based Human-AI Oversight for Building a Precise Video Language [CVPR'26 Highlight, Top 3%].
Try prompting a video generator for a dolly zoom, dutch angle, point of view, or camera roll. Most fall back to the same bland defaults: a push-in, a level shot, a third-person view. Why? These techniques require a language of cinema that current models rarely speak.
We built that language:
1️⃣ Precise specification: 5-aspect structured captions co-designed with professional cinematographers covering subject, scene, motion, spatial, and camera dynamics
2️⃣ Scalable oversight: LLMs draft captions, humans critique what's wrong and how to fix it
3️⃣ Post-training recipes: Qwen3-VL-8B surpasses Gemini-3.1 and GPT-5
4️⃣ Video generation: fine-tuned Wan follows 400-word cinematic prompts with precise control
Here's how each works 🧵
Work led by CMU and Harvard with @chancharikm, @du_yilun, and @RamananDeva.
📄 Paper: huggingface.co/papers/2604.2…
🌐 Site: linzhiqiu.github.io/papers/c…
25
61
370
35,785
Zhiqiu Lin retweeted
Concept bottleneck models promise interpretable vision: predict named attributes first, classify from them. Where do the attributes come from? Usually an LLM prompted with the class name. No image is ever consulted. That turns out to be the weak link 🧵
ALT For "strawberry" on ImageNet-Sketch, an LLM lists red and ripe because it never sees an image. Scoring attributes against the sketch images returns what actually holds: colorless, hand-drawn.
1
5
11
542
Zhiqiu Lin retweeted
[1/6] Can one model fix rendering artifacts from any 3D representation? And what does it even mean to "fix" a rendering?
At #ECCV2026, we present FixAnything (fix-anything.github.io): a single generalist video model that refines 3DGS/mesh/sparse point-cloud renderings into photorealistic, 3D-consistent videos.
7
45
263
16,776
Congratulations, Chancharik! Working with you over the past few years has been incredibly inspiring. MIT is lucky to have you, and I’m excited to see everything you accomplish in this next chapter!
🎓I’m excited to share that I recently graduated with my Master’s in Machine Learning from Carnegie Mellon University’s School of Computer Science (@CarnegieMellon, @SCSatCMU).
This fall, I will be joining MIT (@MIT) as a PhD student in Electrical Engineering and Computer Science, advised by Paul Pu Liang (@pliang279) and Antonio Torralba, as part of the MIT Media Lab (@medialab) and CSAIL (@MIT_CSAIL).
I’ve been incredibly fortunate to learn from and work with amazing advisors throughout the years. I want to thank Deva Ramanan (@deva_ramanan) for his guidance and support during my time at CMU as well as Trevor Darrell (@TrevorDarrell) and Narges Norouzi for shaping my growth during my undergraduate years at UC Berkeley (@UCBerkeley).
No academic journey is complete without the vital support of mentors who challenge, encourage, and guide us along the way. I’m especially grateful to Roei Herzig (@roeiherzig), Zhiqiu Lin (@ZhiqiuLin), Rudy Corona (@_rodolfocorona_), Abrar Anwar (@_abraranwar), Jesse Thomason (@_jessethomason_), and Jean Oh (@jeanoh) for mentorship, collaboration, and for influencing the way I think about research. Finally, thank you to my family for their constant love and encouragement. A special thank you to my mom, whose unwavering support has been with me through every step of my academic journey.
I feel very lucky for the opportunities and people that brought me here. I'm excited for the next chapter and to build on my past work in sample-efficient adaptation, multimodal AI, robotics, and mechanistic interpretability while continuing to learn from the incredible communities at MIT.
Looking forward to all the things left to learn!
1
9
1,557
Zhiqiu Lin retweeted
🎓I’m excited to share that I recently graduated with my Master’s in Machine Learning from Carnegie Mellon University’s School of Computer Science (@CarnegieMellon, @SCSatCMU).
This fall, I will be joining MIT (@MIT) as a PhD student in Electrical Engineering and Computer Science, advised by Paul Pu Liang (@pliang279) and Antonio Torralba, as part of the MIT Media Lab (@medialab) and CSAIL (@MIT_CSAIL).
I’ve been incredibly fortunate to learn from and work with amazing advisors throughout the years. I want to thank Deva Ramanan (@deva_ramanan) for his guidance and support during my time at CMU as well as Trevor Darrell (@TrevorDarrell) and Narges Norouzi for shaping my growth during my undergraduate years at UC Berkeley (@UCBerkeley).
No academic journey is complete without the vital support of mentors who challenge, encourage, and guide us along the way. I’m especially grateful to Roei Herzig (@roeiherzig), Zhiqiu Lin (@ZhiqiuLin), Rudy Corona (@_rodolfocorona_), Abrar Anwar (@_abraranwar), Jesse Thomason (@_jessethomason_), and Jean Oh (@jeanoh) for mentorship, collaboration, and for influencing the way I think about research. Finally, thank you to my family for their constant love and encouragement. A special thank you to my mom, whose unwavering support has been with me through every step of my academic journey.
I feel very lucky for the opportunities and people that brought me here. I'm excited for the next chapter and to build on my past work in sample-efficient adaptation, multimodal AI, robotics, and mechanistic interpretability while continuing to learn from the incredible communities at MIT.
Looking forward to all the things left to learn!
12
4
81
10,261
Two years ago we released VQAScore with a simple idea: ask a VLM "does this image show {prompt}?" and use P(Yes) as the score.
It has since become the go-to evaluation metric and reward model for visual generation, replacing CLIPScore across the field. Adopted by Google DeepMind (Imagen 3 & 4), NVIDIA, ByteDance, and other frontier labs. Our open-source model has 2M+ downloads on HuggingFace.
Today: a major upgrade. VQAScore now supports text-to-video evaluation using 20+ SOTA VLMs including GPT, Gemini, and Qwen, capturing generation accuracy for prompts like "a shallow depth of field shot rack focusing from a foreground crumpet to a dog entering the background, catching it at the focus shift."
As VLMs get stronger, VQAScore gets stronger. For free.
📄 Paper: arxiv.org/abs/2404.01291
💻 Code: github.com/linzhiqiu/t2v_met…
Thanks to @chancharikm for driving this upgrade, and @gneubig and @RamananDeva.
1
19
62
59,512
Zhiqiu Lin retweeted
What if AI could caption videos with the precision of a professional filmmaker?
Researchers from Carnegie Mellon University and Harvard University introduce CHAI just for that.
They built a structured video language using hundreds of visual primitives defined with filmmakers. Then, trained experts critique and refine AI-generated captions, boosting accuracy and efficiency.
The resulting model beats Gemini 3.1 Pro and even fine-tunes video generators like Wan to follow 400-word prompts, unlocking fine-grained control over camera motion, angle, and framing.
Building a Precise Video Language with Human-AI Oversight
Paper: arxiv.org/abs/2604.21718
Code: github.com/chancharikmitra/C…
Our report: mp.weixin.qq.com/s/SCk63tSur…
📬 #PapersAccepted by Jiqizhixin
1
10
1,593
Zhiqiu Lin retweeted
Before AI can generate professional videos, it needs to see like a professional.
We spent a year with 100+ content creators teaching AI to describe video like a filmmaker would.
Introducing CHAI: Critique-based Human-AI Oversight for Building a Precise Video Language [CVPR'26 Highlight, Top 3%].
Try prompting a video generator for a dolly zoom, dutch angle, point of view, or camera roll. Most fall back to the same bland defaults: a push-in, a level shot, a third-person view. Why? These techniques require a language of cinema that current models rarely speak.
We built that language:
1️⃣ Precise specification: 5-aspect structured captions co-designed with professional cinematographers covering subject, scene, motion, spatial, and camera dynamics
2️⃣ Scalable oversight: LLMs draft captions, humans critique what's wrong and how to fix it
3️⃣ Post-training recipes: Qwen3-VL-8B surpasses Gemini-3.1 and GPT-5
4️⃣ Video generation: fine-tuned Wan follows 400-word cinematic prompts with precise control
Here's how each works 🧵
Work led by CMU and Harvard with @chancharikm, @du_yilun, and @RamananDeva.
📄 Paper: huggingface.co/papers/2604.2…
🌐 Site: linzhiqiu.github.io/papers/c…
25
61
370
35,785
Building a Precise Video Language with Human-AI Oversight
paper: huggingface.co/papers/2604.2…
2
12
52
11,067
Before AI can generate professional videos, it needs to see like a professional.
We spent a year with 100+ content creators teaching AI to describe video like a filmmaker would.
Introducing CHAI: Critique-based Human-AI Oversight for Building a Precise Video Language [CVPR'26 Highlight, Top 3%].
Try prompting a video generator for a dolly zoom, dutch angle, point of view, or camera roll. Most fall back to the same bland defaults: a push-in, a level shot, a third-person view. Why? These techniques require a language of cinema that current models rarely speak.
We built that language:
1️⃣ Precise specification: 5-aspect structured captions co-designed with professional cinematographers covering subject, scene, motion, spatial, and camera dynamics
2️⃣ Scalable oversight: LLMs draft captions, humans critique what's wrong and how to fix it
3️⃣ Post-training recipes: Qwen3-VL-8B surpasses Gemini-3.1 and GPT-5
4️⃣ Video generation: fine-tuned Wan follows 400-word cinematic prompts with precise control
Here's how each works 🧵
Work led by CMU and Harvard with @chancharikm, @du_yilun, and @RamananDeva.
📄 Paper: huggingface.co/papers/2604.2…
🌐 Site: linzhiqiu.github.io/papers/c…
25
61
370
35,785
🎬 Better video understanding → better video generation.
We re-caption large-scale professional videos (films, commercials, music videos, games) and fine-tune Wan to follow detailed cinematic prompts of up to 400 words.
The result: precise control over techniques current generators struggle with — dolly zoom, rack focus, speed ramps, Dutch angles, isometric game perspectives, underwater-to-surface transitions, revealing shots, and more.
1
3
8
794
Why trained professionals? We first tried crowdsourced workers. They still confuse "dolly-in" with "zoom-in," call full shots "close-ups," and describe fisheye distortion as "circular buildings."
We also evaluated 8 video-text datasets (2016–2025) and found recurring issues from lack of specification and oversight.
To support reliable video-language data curation, we openly release our full recipe: specification, training materials, annotation platform, quality control pipeline, data, code, and models.
🌐 linzhiqiu.github.io/papers/c…
CHAI (CVPR'26 Highlight) is part of a broader effort alongside CameraBench (NeurIPS'25 Spotlight) and Moodio (coming soon). Stay tuned!
2
14
727
Zhiqiu Lin retweeted
[1/7] Video diffusion has come a long way, generating more & more realistic videos.
Can we revisit sparse-view novel view synthesis through these video priors?
Meet FrameCrafter: a permutation-invariant multi-view model built on video diffusion 🧵
🌐 frame-crafter.github.io
2
33
156
11,872
Zhiqiu Lin retweeted
🎉Despite massive pretraining, VLAs need to adapt to specific physical contexts. We introduce Robotic Steering, a novel finetuning method using mechanistic interpretability to surpass standard FT:
🎁 22× fewer parameters
🎁 +53% on unseen tasks
🎁 Interpretable
Thread below👇
10
45
282
46,715
🎉CameraBench has been accepted as a Spotlight (3%) @ NeurIPS 2025. Huge congrats to all collaborators at CMU, MIT-IBM, UMass, Harvard, and Adobe.
CameraBench is a large-scale effort that pushes video-language models to reason about the language of camera motion just like professional cinematographers.
🌍 Our open-source dataset, models, and code are also gaining strong interest and adoption from frontier labs such as DeepMind and Kling to advance video generation research.
📄Paper: arxiv.org/abs/2504.15376
🌐 Website: linzhiqiu.github.io/papers/c…
📷 Can AI understand camera motion like a cinematographer?
Meet CameraBench: a large-scale, expert-annotated dataset for understanding camera motion geometry (e.g., trajectories) and semantics (e.g., scene contexts) in any video – films, games, drone shots, vlogs, etc. Links below!
We contribute a taxonomy of motion primitives, co-designed over months with professional cinematographers, and apply rigorous quality control to label and caption all aspects of camera motion.
CameraBench shows that even the best SfMs and VLMs struggle with real-world, dynamic videos. Yet, a generative VLM post-trained on our high-quality data matches SOTA SfM (MegaSAM) in geometric understanding and outperforms SOTA VLMs (Gemini-2.5 / GPT-4o) in semantic understanding, e.g., describing how the camera moves.
📄 Paper: huggingface.co/papers/2504.1…
🌐 Website: linzhiqiu.github.io/papers/c…
Work led by CMU, MIT-IBM, UMass, Adobe, Harvard, Emerson with @censiyuan1, @chancharikm, @JayKarhade, @du_yilun, @gan_chuang, and @RamananDeva.
7
25
160
26,222
Zhiqiu Lin retweeted
📷 Can AI understand camera motion like a cinematographer?
Meet CameraBench: a large-scale, expert-annotated dataset for understanding camera motion geometry (e.g., trajectories) and semantics (e.g., scene contexts) in any video – films, games, drone shots, vlogs, etc. Links below!
We contribute a taxonomy of motion primitives, co-designed over months with professional cinematographers, and apply rigorous quality control to label and caption all aspects of camera motion.
CameraBench shows that even the best SfMs and VLMs struggle with real-world, dynamic videos. Yet, a generative VLM post-trained on our high-quality data matches SOTA SfM (MegaSAM) in geometric understanding and outperforms SOTA VLMs (Gemini-2.5 / GPT-4o) in semantic understanding, e.g., describing how the camera moves.
📄 Paper: huggingface.co/papers/2504.1…
🌐 Website: linzhiqiu.github.io/papers/c…
Work led by CMU, MIT-IBM, UMass, Adobe, Harvard, Emerson with @censiyuan1, @chancharikm, @JayKarhade, @du_yilun, @gan_chuang, and @RamananDeva.
10
46
214
56,207
Zhiqiu Lin retweeted
Meet MapAnything – a transformer that directly regresses factored metric 3D scene geometry (from images, calibration, poses, or depth) in an end-to-end way. No pipelines, no extra stages. Just 3D geometry & cameras, straight from any type of input, delivering new state-of-the-art results 🚀
One universal model enables SoTA for:
🔥 Mono Depth Estimation
🔥 Multi-View SfM
🔥 Multi-View Stereo
🔥 Depth Completion
🔥 Registration
… and many more possibilities! – plus everything is metric 🎯
We release code for data processing, training, benchmarking & ablations – everything Apache 2.0!
Details & Links 👇
30
130
734
123,690
Zhiqiu Lin retweeted
Excited to share recent work with @kaihuac5 and @RamananDeva where we learn to do novel view synthesis for dynamic scenes in a self-supervised manner, only from 2D videos!
webpage: cog-nvs.github.io
arxiv: arxiv.org/abs/2507.12646
code (soon): github.com/Kaihua-Chen/cog-n…
4
33
114
24,349