This $8 Microcontroller Is Running a Language Model Locally!

This is a 28.9 million parameter language model generating text right now, one word at a time on a chip that costs about $8. There's no Wi-Fi. Nothing is being sent to a server. Everything is happening inside an ESP32S3, a microcontroller with less RAM than a computer from the '90s. This is insane. How is this actually possible? What's

the magic behind it? and how can we build something similar? Well, those are all good questions that we're going to look at in today's video. It's going to be a lot of fun. So, let's dive into it. So, the ESP32S3 is a chip that gives you 512 kilob of SRAMM. And that is the amount of fast

memory this chip can compute with. Normally, if you would attempt to run an LLM inside of it, the whole model would have to fit right in there. The last language model anyone got running on a chip like this topped out at 260,000 parameters. So, this one holds about 110 times more. And the person who pulled it

off is a Ukrainian developer, Slava S. So, what's the magic trick? How did they do it? How did they fit a 28.9 million parameter model on an eight chip? Well, the workaround comes from an idea Google used in Gemma. It's called per layer embeddings. Most of a language model's parameters do not compute anything. They

sit in an embedding table that just gets read from. If most of your parameters are only ever looked up, they don't need fast memory. So you can get away with leaving that table sitting in a slow cheap flash memory and only pull the rows the current token needs and the small part that actually computes and

thinks about the next token the attention head and feed forward stays in SRAMM. Okay, that's the compute part. But the question still remains, how do we fit such a big model onto such a tiny chip? So, the 25 million row table lives in flash memory, and that's the biggest chunk of the model's whole 28.9 million