Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 8 - LLM Evaluation

Hello, everyone, and welcome to lecture 8 of CME 295. So today's topic will be LLM evaluation. And I think this class is probably one of the most important classes of this quarter because the idea is if we don't how to measure the performance of our LLM, we don't really know what to improve.

And so this class will focus on how we can quantify how the LLM performs in a bunch of different cases. So with that said, we are going to start the class as usual by recapping what we saw last week. So if you remember, last week, we saw how our LLM could interact with systems that

are outside of the LLM itself. So we saw one core technique that is called RAG that allows our LLM to fetch information from external knowledge bases. And so here RAG stands for Retrieval-Augmented Generation. And we saw how we could improve the retrieval system. So we saw that it was composed of two main steps.

So one was candidate retrieval, which is typically something that is done with a bi-encoder setup. So Sentence-BERT was a good example of how people would design such a model. And so this first step is typically there to filter down the potential relevant candidates for a given incoming query. And then we saw that there was a second step, which

was reranking, and that one was a bit more involved and involved cross-encoders, which were more sophisticated. And we also saw some ways to quantify how well our retrieval system performed. And then we also saw something that was called tool calling, which is the ability for a model to which tool to call with which argument.

So if you remember, if we give our LLM the knowledge of the tools that are available to it, it can figure out which arguments it needs to input to the function as a function of the input query, and then run that function and then output the result in natural language to the user.