International
The Morning Science

What science found today

‹
 

Watch · Mathematics, visualised

How to compress text to 1 bit per character | Compression is Intelligence Part 3

The Brief

  • The video explains arithmetic codes and how they relate to the series title 'Compression is Intelligence'.
  • It covers Shannon's estimate of text entropy and how text can be mapped to real numbers for compression.
  • It discusses reframing pretraining in terms of compression and shows arithmetic coding in action.
  • It ends with reflections on the idea that compression and intelligence are linked.

Why watch · It connects information-theoretic compression with prediction and machine learning, which is useful for readers interested in how language models relate to data compression.

  • Arithmetic coding
  • Shannon entropy
  • Data compression
  • Language models

Summarised by Claude from the video’s title and description – no transcript was available, so check the video for the details.

Show the full video description ›

Arithmetic codes and how they elucidate this trilogy's title Channel career fair: https://3b1b.com/talent New store items! https://3b1b.com/store Early views for supporters: https://3b1b.com/support Other resources you may enjoy. On Asymmetric numeral systems: https://arxiv.org/abs/0902.0271 https://demonstrations.wolfram.com/DataCompressionUsingAsymmetricNumeralSystems/ https://arxiv.org/pdf/2201.01741v2 https://arxiv.org/pdf/2510.13867 On Arithmetic Codes: https://dl.acm.org/doi/10.1145/214762.214771 Ilya Sutskever, An Observation on Generalization https://www.youtube.com/watch?v=AKMuA_TVz3A&t=24s A bit on prequential coding: https://yidingjiang.github.io/blog/post/curriculum/ A few other good papers/posts on the relationship between compression, prediction, and ML: https://deepmind.google/research/publications/39768/ https://www.hutter1.net/ai/uaibook.htm https://link.springer.com/chapter/10.1007/3-540-62858-4_69 https://www.sciencedirect.com/science/article/abs/pii/0005109878900055 Timestamps: 0:00 - The absurdity of Shannon’s estimate 6:27 - From text to real numbers 16:57 - Is this Shannon’s limit? 20:26 - Reframing pretraining 22:41 - Arithmetic codes in action 27:29 - Reflections on “Compression is Intelligence” 36:30 - Three ways to support Secret Enscreen Vlog: https://youtu.be/hoX3IiKXT-M ------------------ These animations are largely made using a custom Python library, manim. See the FAQ comments here: https://3b1b.co/faq#manim Music by Vincent Rubinetti. https://vincerubinetti.bandcamp.com/album/the-music-of-3blue1brown https://open.spotify.com/album/1dVyjwS8FBqXhRunaG5W5u ------------------ 3blue1brown is a channel about animating math, in all senses of the word animate. If you're reading the bottom of a video description, I'm guessing you're more interested than the average viewer in lessons here. It would mean a lot to me if you chose to stay up to date on new ones, either by subscribing here on YouTube or otherwise following on whichever platform below you check most regularly. Mailing list: https://3blue1brown.substack.com Twitter: https://twitter.com/3blue1brown Bluesky: https://bsky.app/profile/3blue1brown.com Instagram: https://www.instagram.com/3blue1brown Reddit: https://www.reddit.com/r/3blue1brown Facebook: https://www.facebook.com/3blue1brown Patreon: https://patreon.com/3blue1brown Website: https://www.3blue1brown.com