Four production agent projects in the last two years. Four different frameworks. LangGraph for a stateful pipeline with human review gates at critical decisions. CrewAI for a research workflow that broke down naturally into role-based delegation. Pydantic AI for typed tool calls behind a thin API. OpenAI Agents SDK for one that was going to live inside the OpenAI runtime anyway. Every single project opened the same way: two weeks of reading docs and building toy demos before I could commit.
The reference I kept wishing existed on every one of those decisions was a single side-by-side table that stated control style, state model, what the framework is actually shaped for, license, and a rough liveness signal. I never found a good one. So I built it, at compare-lab.xyz/ai-agent-frameworks. Fifteen frameworks at launch, one row each, taglines written the way I would describe the tool to a friend making the selection.
Framework selection is a decision, not a leaderboard
The obvious first instinct when comparing agent frameworks is to look for a benchmark. There are several, and they measure real things. Tool-call success on synthetic tasks. Success rate on multi-turn planning problems. Cost per completed task. If picking a framework were like picking a database engine on TPC-C numbers, a leaderboard would be the right answer.
It is not the same shape of decision. When I picked LangGraph for a stateful pipeline, the deciding factor was not throughput, it was whether the framework's state model matched the shape of the problem. When I picked CrewAI for the research workflow, the deciding factor was whether decomposing the task into named roles produced something I could reason about. When I picked Pydantic AI for the typed-tool API, the deciding factor was that the abstraction was thin enough to disappear behind my own interface. None of these decisions has a benchmark number that answers them.
Three things a benchmark cannot see
The public agent benchmarks measure tool-call success on small canned tasks and miss the three things that actually decide a real project.
The first is state-model fit. If your task requires explicit state that survives across steps and branches conditionally on intermediate results, you need a framework whose primitives are nodes and edges, like LangGraph or Semantic Kernel. If your task is naturally decomposable into independent roles that collaborate, CrewAI or multi-agent AutoGen patterns fit better. If your task is a typed function loop with strict schemas, Pydantic AI is closer than either. Picking the wrong state model means fighting the framework on every commit rather than building with it.
The second is abstraction escape. Every framework has an abstraction layer designed for the common case. Production agents hit edge cases constantly. What matters is whether you can break out of the abstraction cleanly when you need to, without rewriting the whole pipeline. Some frameworks let you drop down to a raw model call and hand the result back up. Some make that painful. The difference is invisible on a benchmark and dominates the third month of a real project.
The third is failure recovery. What does the framework actually do when a tool call fails halfway through a long-running run? Can you retry from the last successful checkpoint, or do you replay the whole thing. Can you log the partial state and restart at step three, or is the run gone. These questions are invisible in toy demos and critical the first time production wakes you at 3 AM.
What a good directory row promises
A directory row does something different from a leaderboard. It does not try to declare a winner. It tries to compress the framework into the smallest set of facts that lets a reader eliminate wrong choices fast.
The row I settled on has five columns worth reading side by side. Control style, which is one of graph, role crew, typed tool, or conversational. State model, which describes how the framework holds and passes state between steps. License, which decides whether the framework is even eligible for a given project. What the framework is shaped for, which is one sentence written the way I would describe the tool to a friend, not the way the project page describes it. And a rough liveness signal, which right now is a GitHub star count. Star counts are a bad proxy for quality and a decent proxy for whether the project is still maintained at all.
Fifteen frameworks at launch: LangGraph, CrewAI, AutoGen, Pydantic AI, OpenAI Agents SDK, Mastra, LangChain Agents, LlamaIndex Agents, Semantic Kernel, Haystack Agents, Smolagents, Atomic Agents, Phidata, DSPy, AG2. Each row has a tagline I would actually write to a friend making the selection decision, not the one from the marketing site.
What the directory is not
It is not a benchmark. It is not a ranking. Every framework on the list has a use case where it is the right pick. The directory is meant to shave the two weeks of doc-reading down to an afternoon of comparing rows, not to tell anyone which framework to use.
It is also not the last word on any row. The data file is open. If you have shipped on one of these in production and a row gets a detail wrong, a correction lands in a one-line PR. If you have shipped on a framework not yet listed, the row template is documented and adding one is small.
Where this sits
Right now the directory is a static site with fifteen rows, generated from a YAML data file. The next moves are wiring in a machine-readable feed so the rows can be pulled into other comparison tools, and letting each row link out to a longer note when I have hands-on experience worth writing up. The point was never a definitive framework leaderboard. It was a working reference for the two weeks that open every agent project, so those two weeks can be an afternoon.
The directory is at compare-lab.xyz/ai-agent-frameworks. The related post on how I keep production agents honest against the model itself: Claude Code Hallucination Discipline.