FreeToken: How a 24GB GPU Can Serve 700B AI Models (VRAM as a Cache)

We have spent years thinking about local LLM inference in one simple equation. Model size must fit inside GPU memory. But that assumption is starting to break. A new class of inference systems, such as Free Token, is asking a much more interesting question. What if GPU memory is just a cache for the model, rather than the model's home?

The problem with local AI is not that modern GPUs are too slow. It is that they are too small. Your computer GB of VRAM, but 128 GB of system RAM, a fast CPU, and high-speed PCIe. Yet, traditional inference architectures treat GPU VRAM as the entire universe. If the model does not fit in VRAM,

execution halts immediately. Here's where modern model architectures change the math. Many frontier models are mixture of experts. While they contain hundreds of billions of total parameters, each forward pass routes every token to only two to four active experts. A 300 billion parameter model only performs 20 to 30 billion parameters of compute per

token. Why force all inactive experts to live in precious VRAM? This is the core insight of Free Token. Instead of treating GPU memory as the permanent home for the entire model, treat VRAM as a high-speed L1 cache. The full set of cold experts lives across large, inexpensive system RAM. Hot experts are dynamically cached in VRAM.

When the router requests an expert, on a cache hit, it runs instantly. On a miss, the runtime fetches it across PCIe. This fundamentally redefines how we view local computer hardware. Instead of GPU as the sole computer and RAM as passive storage, Free Token treats GPU compute, CPU compute, VRAM, system RAM, and PCIE