International
The Morning Science

What science found today

‹
 

Watch · Machine learning and mathematics

The most cited paper of the century is a brilliant hack

Show the full video description ›

Thank you to Jane Street for sponsoring this video: https://www.janestreet.com/join-jane-street/internships/?utm_source=welchlabs26&utm_medium=youtube&utm_campaign=welch-labs Welch Labs Book: https://www.welchlabs.com/resources/ai-book-ezrzm-msrmc Patreon: https://www.patreon.com/c/welchlabs SECTIONS 0:00 - Intro 1:04 - When Deep Learning Stopped Working 3:17 - Tracing Data Forward 8:18 - The Degradation Problem 9:11 - Probing the Model 13:20 - Loss Landscapes 15:49 - Shattered Gradients 16:39 - Residual Networks 21:22 - Jane Street is a Fascinating Place to Work 23:45 - ResNets Force a New Understanding 25:16 - The Residual Stream 27:52 - Vision Transformers Need Registers 32:18 - AI & Scientific Progress 34:27 - Support Welch Labs on Patreon! NOTES Kaiming He declined to be interviewed for this video, Xiangyu Zhang, and Shaoqing Ren did not respond to our interview requests, and Jian Sun unfortunately passed away in 2022. Our reporting of the development of ResNet is based on close reading of the team’s papers and recorded talks. The original ResNet paper used slightly different skip connections (Pre ReLU) than presented here, the ResNet updated their approach in 2016, see “Identity mappings in deep residual networks.” The loss landscape approach taken here is from Li, Hao, et al. "Visualizing the loss landscape of neural nets." REFERENCES Hutson, Matthew, and Richard Van Noorden. "The most-cited papers of the twenty-first century." *Nature* 640 (2025): 589. Veit, Andreas, Michael J. Wilber, and Serge Belongie. "Residual networks behave like ensembles of relatively shallow networks." *Advances in neural information processing systems* 29 (2016). He, Kaiming, et al. "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification." *Proceedings of the IEEE international conference on computer vision*. 2015. He, Kaiming, et al. "Deep residual learning for image recognition." *Proceedings of the IEEE conference on computer vision and pattern recognition*. 2016. (One more citation lol) He, Kaiming, et al. "Identity mappings in deep residual networks." *European conference on computer vision*. Cham: Springer International Publishing, 2016. Balduzzi, David, et al. "The shattered gradients problem: If resnets are the answer, then what is the question?." *International conference on machine learning*. PMLR, 2017. Li, Hao, et al. "Visualizing the loss landscape of neural nets." *Advances in neural information processing systems*