International
The Morning Science

What science found today

‹
 

Watch · Machine learning and mathematics

Fei-Fei Li’s $8B World Model

The Brief

  • The video traces Fei-Fei Li's work on world models, from the ImageNet effort through NeRF and newer generative approaches.
  • It covers the robotics data problem and how camera-controlled video generation (CameraCtrl, built on Stable Video Diffusion) relates to world modeling.
  • It discusses training methods such as bidirectional diffusion, teacher forcing, and diffusion forcing, and the question of an 'ImageNet of robotics'.
  • The description lists sections and references but gives no detailed claims beyond these topics.

Why watch · It offers a walkthrough of how researchers are trying to build AI models that simulate 3D environments, which matters for robotics and video generation.

  • World models
  • Fei-Fei Li
  • NeRF
  • CameraCtrl

Summarised by Claude from the video’s title and description – no transcript was available, so check the video for the details.

Show the full video description ›

Apply to join Hudson River Trading: https://www.hudsonrivertrading.com/welchlabs New Welch Labs Book! https://www.welchlabs.com/store/welch-labs-illustrated-guide-to-ai-volume-2 Support on Patreon: https://www.patreon.com/c/welchlabs SECTIONS 0:00 - Intro 1:53 - The ImageNet Slog 3:48 - The Robotics Data Problem 5:23 - NeRF 6:43 - The Competition Arrives 9:42 - Gaussian Splats w/ Marble 10:58 - New View Prediction 13:38 - CameraCTRL & A Huge Problem 15:18 - [Ad] World Modeling at HRT 17:15 - Bidirectional Diffusion 21:38 - Teacher Forcing 25:14 - Diffusion Forcing 28:56 - Modality Forcing & Atlas 31:03 - Training From Scratch 32:50 - The ImageNet of Robotics? 37:50 - Fei-Fei’s Dream 39:40 - The Welch Labs Book! REFERENCES Bloomberg AMD acquisition clip: https://www.youtube.com/watch?v=uD62Yn8j_Pg Li, Fei-Fei. *The worlds I see: Curiosity, exploration, and discovery at the dawn of AI*. Flatiron books: a moment of lift book, 2023. Early Robotics Video from Fei-Fei’s lab: https://www.youtube.com/watch?v=THq7I7C5rkk Behavior Project: https://behavior.stanford.edu/ NeRF Apartment Flythrough: https://www.youtube.com/watch?v=DS90g8WFvJk OpenAI Sora Launch: https://openai.com/index/video-generation-models-as-world-simulators/ Genie 1: Bruce, Jake, et al. "Genie: Generative interactive environments." *Forty-first international conference on machine learning*. 2024. He, Hao, et al. "Cameractrl: Enabling camera control for text-to-video generation." *arXiv preprint arXiv:2404.02101* (2024). CameraCtrl is based on Stable Video Diffusion: Blattmann, Andreas, et al. "Stable video diffusion: Scaling latent video diffusion models to large datasets." *arXiv preprint arXiv:2311.15127* (2023). THANKS TO PATRONS Juan Benet, Yan Babitski, AJ Englehardt, Alvin Khaled, Eduardo Barraza, Hitoshi Yamauchi, Jaewon Jung, Mrgoodlight, Shinichi Hayashi, Sid Sarasvati, Dominic Beaumont, Shannon Prater, Ubiquity Ventures, Matias Forti, Brian Henry, Tim Palade, Petar Vecutin, Nicolas baumann, Jason Singh, Robert Riley, vornska, Barry Silverman, Jake Ehrlich, Mitch Jacobs, Lauren Steely, Jeff Eastman, Rodolfo Ibarra, Clark Barrus, Rob Napier, Andrew White, Richard B Johnston, abhiteja mandava, Burt Humburg, Kevin Mitchell, Daniel Sanchez, Ferdie Wang, Tripp Hill, Richard Harbaugh Jr, Prasad Raje, Kalle Aaltonen, Midori Switch Hound, Zach Wilson, Chris Seltzer, Ven Popov, Hunter Nelson, Amit Bueno, Scott Olsen, Johan Rimez, Shehryar Saroya, Tyler Christensen, Beckett Madden-Woods, Darre