YouTube Instagram TikTok How it works FAQ

The fastest way to turn talking videos into clean, searchable text. Free, private and no account required.

About

Built by Rishav Raj with one goal: free transcripts for everyone.

rishav@yttotranscript.com

Transcribe

YouTube to text
YouTube with timestamps
Instagram Reels
TikTok videos

Learn

How it works
FAQ
YouTube transcripts guide

More

Chrome extension
Contact
Privacy policy
Terms of service

© 2026 YTtoTranscript. Made for creators & researchers.

Not affiliated with YouTube, Instagram or TikTok. Respect creators’ copyright.

How Discord Stores TRILLIONS of Messages — Transcript · YTtoTranscript

Home/Transcripts/YouTube

YouTube Transcript

How Discord Stores TRILLIONS of Messages

Coding with Lewis · 2,346 words · 12 min read · EN-ORIG

Watch on YouTube

Share X LinkedIn WhatsApp

Below is the complete, readable transcript of How Discord Stores TRILLIONS of Messages by Coding with Lewis on YouTube. Read the full text, copy any part you need, or generate a transcript for any video with our free tool.

imagine storing the trillions of messages ever sent on Discord while more are being sent by the millions every hour it is critical that the system remains operational without any downtime Discord faced the situation and was outgrowing their previous database so they needed to find a solution this video is based off of the Discord

engineering blog article written by Bo Ingram senior software engineer at Discord so make sure you check it out I put the link in the description below hi my name is Lewis and if you're new here I'm a professional software developer on a mission to inspire developers and Tech enthusiasts but before we start let's

roll back a little bit funny enough I made a video on this previously called House Discord stores billions of messages which is kind of annoying like they couldn't have worked on my schedule anyway if you don't want to watch that video here's a little backstory in 2017 the mongodb database was causing significant scaling issues

for Discord as its popularity grew After experiencing significant scaling issues with mongodb in 2017. Discord switched to a Cassandra database now despite some challenges with migration the team was ultimately able to successfully transition to Cassandra and get their system up and running since then Discord has continued to grow exponentially with over 140 million users in 2021 which

means that they had to continue to make updates and improvements to their database infrastructure to keep up with the increasing demand of their users but 2017 was six years ago in 2017 10 million users were using Discord and in 2021 there's over a hundred and forty million people using Discord so 14 times

Transcribe another video

Paste any YouTube, Instagram or TikTok link to get a free transcript.

Free · No sign-up · Unlimited

Add to Chrome — transcribe in one click

the users Something's Gotta change with this massive growth came a massive problem how do you store trillions of existing messages initially they started with just 12 Cassandra nodes that stored billions of messages between them but fast forward to last year it grew to 177 nodes with trillions of messages on them and managing this was becoming a

nightmare the system was going through insane slowdowns and the Discord engineering team was having a hard time maintaining this at all times it was clear that they needed a new solution and they needed it fast okay so let's look at the current issues each message is partitioned by the channel they are sent in as well as a static time window

that they call a bucket since it's partitioned this way Cassandra will store all the messages for a given Channel and bucket together and replicate it across three nodes this design caused major issues for the two types of users that use Discord one a small server that sends a few messages versus a giant server like mid-journey

where messages are sent like every millisecond and for Cassandra to read a message it takes more time and effort than it does to write something when you send a message to Discord it's saved in a log and in memory that eventually writes to disk when you are doing something else but when you want to get

a message it has to look in multiple places no's partitions disk files Etc to get the message that you want but when too many people do this it causes the partitions to slow down and cause latency across the whole database cluster kind of like a road closing and then the cars are forced to merge into

your lane it just slows everybody down so not only did this result in slowdowns the software and the databases but it also slowed down engineering time to maintain this database to optimize your reads for Cassandra Discord would have to use compaction which is a maintenance task that will make it easier for Cassandra to organize information on the

disk to make it faster to find however with the issues already mentioned doing this task was put off and caused it to be way more expensive down the line and then just caused a domino effect of issues after that also another classic enemy showed his face the Java garbage collector which constantly had to be

tuned when things would break Java garbage collection is still causing issues to people I thought we were done with this this clearly wasn't cutting it so they did what any programmer would want to do when they see something new and that is to change architectures so funny enough Discord actually had their eyes set on another database years

ago called Cilla DB and yes that's actually how it's pronounced here come here with me cylindb is a high performance nosql database yup I say no SQL get over it that is designed to handle the most demanding workloads it is a Cassandra compatible database that is written in C plus and it offers many

awesome features that set it apart from other databases one of the best features of facilityb is the removal of the garbage collector since it isn't written in Java so this makes it a great choice for organizations that want to avoid the performance overhead that comes with the garbage collector from java socilla DB

is designed to handle a large amount of requests simultaneously and it can scale up easily where their users are frequently accessing the database or not it is a great choice for organizations that need a high performance scalable and reliable nosql database so in summary Cilla DB was arguably perfect for what Discord needed in this current

moment so after some experimentation and toying around the Discord Engineers made the Bold choice of migrating all of the databases to scilla DB they were successful and they had moved every database over to Cilla except for one and that database being Cassandra messages trillions of messages 200 nodes this was going to be a serious

undertaking also just because you get the latest and greatest technology doesn't mean it's always the best sometimes the issues are with your own setup and infrastructure one of the most important components of any web application is this database if you had a load balancer that was pointing at different versions of Discord already

there's a huge risk of one of those applications being able to store information on one database but not being replicated when all is finished speaking of which sometimes the part that we do see doesn't interact with a new API well at all so in some instances maybe you run into an error message

because the newest version of Discord still hasn't settled in your area so to think this happening with over 140 million active users and moving over a database with discord's primary function messages ah sounds stressful huh first a great optimization was finding a way to handle the traffic that was being led to their overloaded

partitions so in between their API monolith and their database they wrote something called data services now when dealing with something that should be milliseconds fast you need a programming language that is fast and secure it's 2023 so of course they chose rust I mean can you blame them though Russ is awesome it offers an amazing developer

experience while also providing some of the advantages you would typically see in low-level programming languages such as C or C plus plus rust has been ranked the most loved programming language to work with for the past four years straight on the stack Overflow developer survey precisely for this reason and rust seamless integration with the Tokyo

ecosystem for managing asynchronous requests makes it a Top Choice for developers working on high performance applications which is probably why big companies like AWS Facebook and Dropbox are using this daily in their own applications the combination of rust in Tokyo provides a reliable scalable solution for handling large amounts of data and requests without experiencing

significant slowdowns or downtime but let's look at these data services each data service contains a grpc endpoint per database query and doesn't contain any business logic now grpc is a newer way to transmit data across different services that you might have for example if I wanted my python app to talk to my

Java app in the server grpc can be used the benefits is that it's more performant as it uses HTTP 2.0 to transmit the data and it's safe as it uses something called protocol buffers to help validate Data before it's even send performant and safe seems like the word of the day here the biggest feature

this data service did was called coalescing if multiple users try to read the same message at the same time rather than querying x amount of people trying to read it it would pull it once and then distribute this amongst the other users and this solved many problems like when someone pinged at everybody or of a

new announcement was made and people swarm to a server remember the less work a database has to do the better to optimize it even more they also provided a way to Route the request to go to a data service based off of the channel ID so all requests for the same channel go

to the same data service so that can pull together the requests and take advantage of the first app mentioned above okay cool but they're still on Cassandra when do we abandon ship okay so move trillions of messages while 140 million people use your application at the same time but do this without any

downtime that's like migrating the population of Russia to a brand new house without them even noticing this is going to be tough step one boot up a new solo DB cluster step two migrate newer data using a cut over time and then migrate historical data behind it awesome let's start after they got

this process started they're now writing new data to their old Cassandra database and their shiny new Sila DB database and after booting up their Sila DB migration tool they got an estimated time of three months three months is a very long time by that time they'll probably want to migrate to another database so doing

what every programmer does in the year 2023 they decided to completely rewrite the migration tool in Rust to make it go 10 times faster I mean maybe even 100 times faster so here's how it works it reads the range of data from the database and then using a smaller database like SQL Lite it uses a

checkpoint where they are at in the migration then it dumps it all into Cilla DB the new estimation put it all the way down to nine days instead of three months this means that they can do their whole migration and just one click swoop with this approach they were able to migrate 3.2 million messages per

second after the nine days it was all migrated successfully well apart from A small collection of tombstones that you can find more about on the previous video and just like that silly DB is the primary database of Discord hosting trillions of messages goodbye Cassandra so after a year of this migration things are still going extremely well

beforehand there are 177 Cassandra nodes but now is set down to 72 still a DB nodes latency to fetch messages also improves going from 40 to 125 milliseconds down to 15 milliseconds inserting a message took five milliseconds to 70 milliseconds and is now down to a consistent five milliseconds so all was looking great

during the World Cup finals Discord Engineers would look at the message performance graphs and were easily able to see the height behind the whole match based on the metrics so looking at the metrics you can see right here where Lionel Messi has a penalty at what impape ties it back up hope has said his

name right it's funny to think that when millions of people are stressing about a World Cup game these Discord Engineers are looking at these metrics and stressing even hard making sure Discord doesn't go down being able to handle the communication in such a popular event is truly a flex and is a perfect way to

Showcase that their new system just worked the migration clearly was a success it's funny to think of some takeaways that we can take from this story as only a few software Engineers have truly gone through this Mega scale project as of this video Discord is one of the world's most popular applications so having the app go down for the time

it took to migrate would make history if it happened what I really liked about this article is that the Discord Engineers just knew that if they moved over to a shiny new object that wasn't going to solve all of their problems developers have shiny object syndrome which distracts developers from solving the problems that they run into because

they think that the shiny object is the answer discourse transition to Cilla was for sure an improvement but the data services in between the database and the API was a massive Improvement on top of the database migration that they were talking about now although it sounds like I'm going to contradict myself here

don't be too comfortable with the tech stack that you have used just because something fits in somewhere doesn't mean that it belongs there discourse switch to rust helps us understand that getting into the gritty helps us determine what truly is the best tool for the job is thanks for watching this video I have a

playlist dedicated for more videos like this showing how big Tech builds the tools that you love and if you like more stories like this make sure you subscribe to my newsletter the better Dev where I Inspire developers to become better devs who should I cover next peace out coders