Contents
What Makes a Model Good for Agents
Not every large language model is a good agent model. A model can score well on general benchmarks like MMLU or HumanEval and still fail at the specific capabilities agents need. Before ranking individual models, you need to understand what "good for agents" actually means in concrete, measurable terms.
The single most important metric is function calling accuracy. When an agent decides to use a tool, the model must generate a valid JSON tool call with the correct function name, correct parameter names, and correctly typed values. The Berkeley Function Calling Leaderboard (BFCL) measures exactly this. It tests models on simple calls (one function, obvious parameters), parallel calls (multiple functions in one turn), multiple calls (sequential tool use across turns), and relevance detection (knowing when NOT to call a tool). BFCL now reports several scores: single-turn accuracy (the classic one-shot tool call), multi-turn accuracy, and an overall score that also covers web search and memory tasks. Quote the right one, because they differ enormously for the same model. A model scoring below 70% on the single-turn score will frustrate you with malformed tool calls, hallucinated parameter names, and wrong argument types. Aim for 80%+ for production agents.
Instruction following is the second criterion. Agent systems rely on system prompts that define behavior: "You are a customer service agent. Always greet the user. Never discuss competitors. If the user asks about refunds, call the lookup_order tool first." A model that drifts from these instructions - ignoring constraints, forgetting persona rules mid-conversation, or hallucinating tools that do not exist - will produce unreliable agents. MT-Bench scores measure multi-turn instruction following, and IFEval measures strict format compliance. Both matter for agents.
Context window size determines how much information your agent can work with in a single session. Agent conversations grow fast: the system prompt (500-2,000 tokens), conversation history (grows with every turn), tool definitions (200-500 tokens per tool), and tool results (often 500-5,000 tokens each). A 10-tool agent with 15 turns of conversation easily consumes 30,000-50,000 tokens of context. Models with 8K context windows hit the wall quickly. For production agents, 32K context is the minimum. 128K is comfortable. Anything less than 32K means you will be implementing conversation summarization and tool result truncation from day one.
Inference speed matters more for agents than for chatbots. A chatbot makes one LLM call per user message. An agent using the ReAct pattern might make 3-8 LLM calls per user message (initial reasoning, tool selection, processing tool results, possibly calling more tools, generating the final response). If each call takes 3 seconds, a single user interaction takes 9-24 seconds. At 1 second per call, the same interaction takes 3-8 seconds. Speed compounds across the tool-calling loop. Measure tokens per second at your target quantization level on your actual hardware - synthetic benchmarks on A100 clusters are not useful if you are deploying on consumer GPUs.
License determines what you can legally do with the model. For agent deployments, the key question is: can you run this commercially without restrictions? Apache 2.0 and MIT licenses are fully permissive. Meta's Llama license allows commercial use but has a monthly active user threshold (700 million users) above which you need a separate license. Some models use non-commercial research licenses that prohibit production deployment entirely. Always check the license before investing engineering time in a model. If you are building agents for a self-hosted deployment, license terms directly affect your go-to-market.
One factor that is often overlooked: tooling ecosystem support. A model that works flawlessly in isolation but is not supported by Ollama, vLLM, LangChain, or your preferred agent framework will cost you weeks of integration work. The models ranked in this post all have first-class support in the major serving and agent frameworks, because practical deployability matters as much as benchmark scores.
#1 Llama 3.3 70B - The Default Choice
Best for: General-purpose agent deployments where you want the safest, most battle-tested option with the widest ecosystem support.
Llama 3.3 70B Instruct is the model to start with. On the current Berkeley Function Calling Leaderboard it scores 88.0% on single-turn function calling (the non-live AST category), with 76.1% on simple calls and 90.0% on parallel calls. Be aware of the other half of the picture: its multi-turn score is 21.5% and its overall BFCL V4 score, which adds web search and memory tasks, is 31.9%. One-shot tool calls are its strength; long autonomous chains are where you add structure around it. It has a 128K token context window - more than enough for complex multi-tool agent sessions. And it runs on hardware that is actually accessible: two RTX 3090 GPUs (48 GB total VRAM) at Q4 quantization.
The numbers in context: Llama 3.3 70B's single-turn score of 88.0% means roughly 1 in 8 tool calls will have some issue - a wrong parameter type, a missing optional argument, or a slightly malformed JSON structure. In practice, most of these failures are recoverable (the agent framework catches the error and retries), so the effective success rate for a well-built agent is higher than the raw score - measure it on your own tool schemas. Compare this to Llama 3.1 8B at 84.0% single-turn and 11.1% multi-turn, and you see where the extra parameters go: not into the single call, but into keeping a conversation of several calls on track.
VRAM requirements at different quantization levels: Q4_K_M (the sweet spot for most deployments) is a 43 GB download on Ollama, which fits on 2x RTX 3090 (24 GB each), 2x RTX 4090 (24 GB each), a single A6000 (48 GB), or a single A100 80GB. Q8 is 75 GB, requiring an A100 80GB or 2x A6000. FP16 (full precision) is 141 GB - two A100 80GBs or a multi-GPU setup on enterprise hardware. For most agent use cases, Q4_K_M provides virtually identical function calling accuracy to FP16 while using less than a third of the VRAM.
Get started with Ollama in one command:
ollama run llama3.3:70b-instruct-q4_K_M
For production serving with vLLM (higher throughput, continuous batching, OpenAI-compatible API):
python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3.3-70B-Instruct --quantization awq --tensor-parallel-size 2 --max-model-len 65536 --gpu-memory-utilization 0.90 --port 8000
On an A6000 (48 GB), Llama 3.3 70B at Q4 quantization generates 28-35 tokens/sec for single requests. With vLLM's continuous batching handling 8 concurrent requests, aggregate throughput reaches 120-160 tokens/sec. For the ReAct agent loop where each LLM call generates 50-200 tokens, that is 1.5-7 seconds per call at single-request latency - fast enough for interactive agents, though not instant.
Why #1 and not Qwen or DeepSeek? Three reasons, and a license note: Meta's Llama 3.3 Community License permits commercial use but requires a separate license from Meta above 700 million monthly active users. First, ecosystem support: every major agent framework (LangChain, LlamaIndex, AutoGen, CrewAI) has been tested extensively with Llama models. Llama-specific prompt templates and tool calling formats are documented and battle-tested. Second, community: when you hit an issue with Llama 3.3 on Ollama or vLLM, someone has probably already solved it on GitHub or the Ollama Discord. Third, predictability: Llama 3.3 70B has been deployed in thousands of production systems since its release, and its failure modes are well-understood. The other models on this list outperform Llama in specific scenarios, but none match its overall reliability-ecosystem-community combination. For a complete guide on setting up the infrastructure around any of these models, see our self-hosted LLM agent stack guide.
#2 Qwen 2.5 72B - Best for Multilingual Agents
Best for: Agent deployments that need to handle multiple languages, especially Chinese, Japanese, Korean, or European languages alongside English.
Qwen 2.5 72B Instruct from Alibaba Cloud is the multilingual pick in this size class. Its model card lists 72.7B parameters, support for more than 29 languages, and a 131,072 token context window with up to 8,192 generated tokens. Qwen 2.5 72B is not on the current BFCL board, so there is no like-for-like function calling score to quote against Llama; the newer Qwen3 models are listed, and they are in the comparison table below. If your agents do tool calling in Chinese, Japanese or Korean, test both models on your own prompts before committing.
The benchmark comparison tells a clear story. On English-only agent tasks, Llama and Qwen are interchangeable - pick whichever has better tooling support in your stack. On multilingual tasks, Qwen is the one built for the job. If your agents serve users in East Asian or European languages, or if your tools return results in non-English languages (local search APIs, regional databases, multilingual knowledge bases), Qwen 2.5 72B is the right choice. If your agents are English-only, Llama's larger ecosystem gives it the edge.
VRAM requirements are nearly identical to Llama 3.3 70B given the similar parameter count: the Q4_K_M build is a 47 GB download on Ollama, which is tight on 48 GB of VRAM once you add context. Note that Ollama's Qwen 2.5 tags default to a 32K context window, so raise it if your agent needs more. Performance is also comparable: 26-33 tokens/sec on an A6000 at Q4, with vLLM throughput scaling linearly with additional GPUs.
Check the license before you plan around it. Most of the Qwen 2.5 family, including the 7B below, is Apache 2.0, but the 72B weights are published under the separate Qwen license, as the model card shows. Read its terms for your use case rather than assuming Apache 2.0 applies to every size. If a fully permissive license is a hard requirement, the 7B is the Qwen 2.5 model that meets it.
Deploy with Ollama:
ollama run qwen2.5:72b-instruct-q4_K_M
Production vLLM configuration:
python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-72B-Instruct --quantization awq --tensor-parallel-size 2 --max-model-len 65536 --gpu-memory-utilization 0.90 --port 8000
One practical consideration: Qwen's tool calling format differs slightly from Llama's. Where Llama uses the standard OpenAI-style tool call format natively, Qwen uses its own format that agent frameworks translate. LangChain, LlamaIndex, and vLLM all handle this translation automatically, but if you are building a custom agent framework or calling the model directly, you will need to implement the Qwen tool call parser. The Qwen2.5 model card on HuggingFace documents the exact format.
When to choose Qwen over Llama: your agents serve multilingual users, or your tools return non-English content. When to stick with Llama: English-only deployments where ecosystem depth and community support matter most. Both models are excellent agent backbones - this is a "which is slightly better for your specific case" decision, not a "good vs. bad" decision. For guidance on choosing between self-hosted models like these and cloud APIs, see our cost comparison guide.
#3 DeepSeek V3 - Best Reasoning for Complex Chains
Best for: Agent workflows involving multi-step reasoning, complex tool chains, and tasks where answer quality matters more than latency.
DeepSeek V3 is architecturally different from every other model on this list. It uses a Mixture-of-Experts (MoE) architecture with 671 billion total parameters, but only activates approximately 37 billion parameters per token. This means you get the knowledge capacity of a 671B model with the inference cost of a ~40B model. DeepSeek V3 itself is not on the current BFCL board, but its successor DeepSeek-V3.2-Exp is, at 56.7% overall in prompt mode with thinking - well ahead of Llama 3.3 70B's 31.9% overall, because it holds up far better on multi-turn tasks (44.9% against 21.5%).
The reasoning advantage is where DeepSeek V3 justifies its spot at #3 despite the hardware it demands. On simple agent tasks (single tool call, straightforward parameters), all three models perform similarly. On complex chains - where the agent needs to reason about which of 15 tools to call, synthesize results from 3 previous tool calls, handle ambiguous user requests, or plan a multi-step sequence before executing - the DeepSeek family is the stronger choice, and the multi-turn gap on the public leaderboard points the same way. The cases to test on your own workload are multi-step order lookups, conditional refund logic, and situations where the agent should recognize it does not have enough information and ask a clarifying question.
VRAM is the tricky part. Despite only activating 37B parameters per forward pass, the full 671B model weights need to be in memory (or loadable from fast storage). At Q4 quantization, DeepSeek V3 needs approximately 350 GB VRAM - that is 5x A100 80GB or 8x A6000 48GB. At FP8, it needs around 670 GB. This is not consumer hardware territory. However, there are two practical deployment paths: first, community quantizations with aggressive quantization (Q2/Q3) bring the requirement down to ~200 GB, fitting on 3x A100 80GB with acceptable quality loss. Second, offloading strategies that keep inactive expert weights in CPU RAM or NVMe, loading them on demand - this trades latency for VRAM savings.
vLLM deployment for a multi-GPU setup:
python -m vllm.entrypoints.openai.api_server --model deepseek-ai/DeepSeek-V3 --quantization fp8 --tensor-parallel-size 8 --max-model-len 32768 --gpu-memory-utilization 0.92 --trust-remote-code --port 8000
Inference speed at FP8 on 8x A100 80GB: approximately 20-25 tokens/sec for single requests. The MoE routing adds a small overhead per token compared to dense models, but the active parameter count keeps per-token compute manageable. For agent workloads, where LLM calls generate 50-200 tokens each, expect 2-10 seconds per call. This is slower than Llama 3.3 70B on equivalent hardware, which is why DeepSeek V3 ranks #3 despite its reasoning advantage - for most agent use cases, the speed-quality tradeoff favors the faster model.
When to use DeepSeek V3 over Llama 3.3 70B: your agent handles complex, multi-step workflows where reasoning quality directly impacts business outcomes (financial analysis agents, legal research agents, complex technical support with branching decision trees). When to stick with Llama: your agent does straightforward tool routing (look up order, check status, send notification) where 88% single-turn accuracy is more than sufficient and speed matters more than reasoning depth. If you are building agent infrastructure that supports multiple models, consider using Llama for simple agent tasks and routing complex tasks to DeepSeek V3 - a pattern we cover in our AI Agents for Operators course.
#4-6 Smaller Models for Constrained Hardware
Not every agent deployment has 48+ GB of VRAM available. If you are running agents on a single consumer GPU, an edge device, or need ultra-low latency, three smaller models deliver surprisingly capable agent performance.
#4 Llama 3.1 8B Instruct
Best for: Single-GPU agent deployments where Llama ecosystem compatibility matters.
Llama 3.1 8B Instruct scores 84.0% on single-turn BFCL (11.1% multi-turn, 25.8% overall) with a 128K context window. At Q4 quantization it is a 4.9 GB download - it runs on a single RTX 3060 12GB with room to spare. On an RTX 4090, it generates 45-55 tokens/sec, making each agent LLM call nearly instant at 1-2 seconds. Deploy with ollama run llama3.1:8b-instruct-q4_K_M.
The 84% single-turn score means roughly 1 in 6 tool calls will have issues, and the low multi-turn score means it loses the thread quickly on long chains. For agents with 3-5 simple tools (search, lookup, calculate), this is workable - framework-level retries catch most failures. For agents with 10+ tools or complex parameter schemas, the error rate compounds and becomes a problem. Use Llama 3.1 8B for focused, narrow agents with well-defined tool schemas and clear parameter descriptions.
#5 Qwen 2.5 7B Instruct
Best for: Multilingual agents on a single GPU, or when Apache 2.0 licensing is required for a small model.
Qwen 2.5 7B Instruct has 7.61B parameters and, unlike the 72B, an Apache 2.0 license. It is not on the current BFCL board, so test it against Llama 3.1 8B on your own tools; like its larger sibling it is the one to try first for multilingual tool calling. It is a 4.7 GB download at Q4 and runs at 48-58 tokens/sec on an RTX 4090. Deploy with ollama run qwen2.5:7b-instruct-q4_K_M. Apache 2.0 licensed.
#6 Phi-3.5 Mini (3.8B)
Best for: Edge deployments, extremely latency-sensitive agents, or when VRAM is severely limited.
Phi-3.5 Mini from Microsoft packs surprising capability into 3.8 billion parameters. It has a 128K context window, and at Q4 it is a 2.4 GB download. Its model card does not mention tool calling, so treat it as a router and classifier rather than a multi-tool agent. It runs at 70-90 tokens/sec on an RTX 4090. This is fast enough that the agent's LLM calls feel nearly instantaneous - sub-second response times for tool routing decisions. MIT licensed.
Deploy with Ollama:
ollama run phi3.5:3.8b-mini-instruct-q4_K_M
When small models win on more than just cost: there are legitimate scenarios where a 7-8B model outperforms a 70B model for agent tasks. The most common is latency-sensitive workflows where the agent makes many sequential tool calls. A 70B model at 30 tokens/sec taking 5 LLM calls means 15-25 seconds of total latency. An 8B model at 50 tokens/sec for the same 5 calls finishes in 5-10 seconds. If your users are waiting synchronously, that latency difference matters more than the accuracy difference. Another scenario: high-concurrency deployments where you need to serve 50+ simultaneous agent sessions on a single GPU. A 70B model at Q4 cannot fit alongside 50 concurrent KV caches, but an 8B model handles this comfortably.
The decision between small and large models is not binary. A practical architecture runs a small model for initial intent classification and simple tool routing (fast, cheap), and escalates to a 70B model only for complex reasoning steps. This hybrid approach - covered in detail in our AI Agent Stack Picker tool - gives you the latency of a small model for routine requests and the quality of a large model for the ones that need it.
Benchmark Comparison Table
Below is a side-by-side comparison of every model discussed in this post, using only figures you can check yourself: parameter counts, context windows and licenses from each model card, Q4_K_M download sizes from the Ollama library, and function calling scores from the current Berkeley Function Calling Leaderboard. Download size is a floor for VRAM, not the full requirement: budget extra for the context cache.
| Model | Params | Context | License | Q4_K_M download | BFCL single-turn | BFCL multi-turn | BFCL overall |
|---|---|---|---|---|---|---|---|
| Llama 3.3 70B | 70B | 128K | Llama 3.3 Community | 43 GB | 88.0% | 21.5% | 31.9% |
| Qwen 2.5 72B | 72.7B | 131K | Qwen license | 47 GB | Not listed | Not listed | Not listed |
| DeepSeek V3 | 671B MoE (37B active) | 128K | MIT code, DeepSeek model license (commercial use allowed) | Model card publishes FP8 weights only | Not listed | Not listed | Not listed |
| Llama 3.1 8B | 8B | 128K | Llama 3.1 Community | 4.9 GB | 84.0% | 11.1% | 25.8% |
| Qwen 2.5 7B | 7.61B | 131K | Apache 2.0 | 4.7 GB | Not listed | Not listed | Not listed |
| Phi-3.5 Mini | 3.8B | 128K | MIT | 2.4 GB | Not listed | Not listed | Not listed |
Checked November 2026 against the model cards for Llama 3.3 70B, Qwen 2.5 72B, DeepSeek V3, Qwen 2.5 7B and Phi-3.5 Mini, the Ollama library tag pages, and the BFCL leaderboard. "Not listed" means the model is not on the current board, not that it scored zero. BFCL single-turn is the non-live AST category; overall is the V4 score including multi-turn, web search and memory.
Newer open-weight models on the same leaderboard
The leaderboard has moved on since the 2.5-generation Qwen and the original DeepSeek V3, and the newer open-weight entries are worth knowing about before you commit hardware. Single-turn scores have converged in the high 80s. Multi-turn is where the models now separate.
| Model (BFCL mode) | License | Single-turn | Multi-turn | Overall |
|---|---|---|---|---|
| DeepSeek-V3.2-Exp (prompt, thinking) | MIT | 85.5% | 44.9% | 56.7% |
| Qwen3-32B (function calling) | Apache 2.0 | 88.8% | 47.9% | 48.7% |
| Qwen3-8B (function calling) | Apache 2.0 | 87.6% | 41.8% | 42.6% |
| Llama 3.3 70B (function calling) | Llama 3.3 Community | 88.0% | 21.5% | 31.9% |
| Gemma 3 27B (prompt) | Gemma terms of use | 87.2% | 10.8% | 29.5% |
| Phi-4 (prompt) | MIT | 69.6% | 3.9% | 28.8% |
| Llama 3.1 8B (prompt) | Llama 3.1 Community | 84.0% | 11.1% | 25.8% |
Scores as published on the BFCL leaderboard, checked November 2026. If multi-turn autonomy matters more to you than ecosystem maturity, Qwen3-8B on a single GPU is the entry on this board to evaluate first.
Reading the table: For most teams, start with the download size column. If you have 48+ GB across GPUs, Llama 3.3 70B or Qwen 2.5 72B will give you the best agent experience. If you have a single 12-24 GB GPU, Llama 3.1 8B or Qwen 2.5 7B are your options. DeepSeek V3 is for teams with enterprise GPU clusters who need maximum reasoning quality and can absorb the hardware cost.
The single-turn gap between Llama 3.3 70B (88.0%) and Llama 3.1 8B (84.0%) looks small in percentage terms but compounds. At 88%, an agent with 5 tool calls per interaction has a (~0.88^5) = 52.8% chance of all calls succeeding. At 84%, that drops to (~0.84^5) = 41.8%. With retry logic, the effective success rate is higher for both, but the reliability gap is real and compounds with agent complexity.
Notice that all six models support context windows of 128K or more on their model cards. This was not the case even a year ago, when many open-source models topped out at 8K or 32K. The convergence on 128K means context window size is no longer a differentiator between open-source models for agent use - it is table stakes. If you are evaluating a model not on this list, context window remains worth checking.
How to Pick the Right Model
Model selection for agents follows a decision tree driven by constraints, not preferences. Start with what you cannot change (hardware budget, language requirements) and let those constraints narrow the field. Here is the decision process.
Step 1: What is your available VRAM?
- Under 8 GB: Phi-3.5 Mini (3.8B) is your only practical option. It is a 2.4 GB download at Q4 and leaves room for KV cache.
- 8-24 GB (single GPU): Llama 3.1 8B or Qwen 2.5 7B. Both fit comfortably on a single RTX 3060/3070/3080/3090/4060/4070/4080/4090.
- 24-48 GB (single large GPU or 2x consumer GPUs): Llama 3.3 70B or Qwen 2.5 72B at Q4. This is the sweet spot for production-quality agents on accessible hardware.
- 200+ GB (enterprise multi-GPU): DeepSeek V3 becomes an option. Only choose this if reasoning quality on complex tasks justifies the hardware investment.
Step 2: What languages do your agents need?
- English only: Llama models at every size tier. Widest ecosystem, most community support, best-documented failure modes.
- Multilingual (especially CJK or European languages): Qwen models at every size tier. The Qwen 2.5 model cards list support for more than 29 languages; confirm tool calling quality in your languages with your own test set.
Step 3: What is your agent's complexity?
- Simple (1-5 tools, straightforward routing): A 7-8B model is sufficient. The speed advantage over a 70B model outweighs the accuracy gap for simple tool schemas.
- Moderate (5-15 tools, some conditional logic): A 70B model is the right tier. The single-turn accuracy difference between 84% and 88% compounds across multiple tool calls, and the multi-turn gap is wider still.
- Complex (15+ tools, multi-step reasoning chains, ambiguous inputs): Start with a 70B model. If you see reasoning failures on complex cases, evaluate DeepSeek V3.
The decision flowchart in plain text:
START
|
v
VRAM < 8 GB? --yes--> Phi-3.5 Mini (3.8B)
|no
v
VRAM < 24 GB? --yes--> Need multilingual? --yes--> Qwen 2.5 7B
|no --no---> Llama 3.1 8B
v
VRAM < 50 GB? --yes--> Need multilingual? --yes--> Qwen 2.5 72B
|no --no---> Llama 3.3 70B
v
VRAM 200+ GB and need max reasoning? --yes--> DeepSeek V3
--no---> Llama 3.3 70B
Common mistake: defaulting to the biggest model. A 70B model is overkill for an agent that routes 3 tools with simple parameters. A 7B model can do this faster with acceptable accuracy, and the saved VRAM allows higher concurrency. Size up only when you have evidence that your agent's failure cases are caused by model capability, not prompt engineering or tool schema issues. Most agent failures below 85% BFCL are fixable with better tool descriptions, more specific system prompts, and structured output schemas - improvements that benefit any model size.
The bigger mistake: picking a model before picking a job. Every branch of the tree above assumes you already know what the agent is for and roughly how hard the task is, and that assumption is wrong more often than the VRAM numbers are. If you cannot yet state the agent's job in one sentence that a non-engineer would recognise, no amount of benchmark reading will help - a 70B model doing an undefined task is just an expensive way to be uncertain. Pick the workflow first, ideally one where success is obvious within a week. Missed-call text-back is a good calibration build for exactly this reason: the task is small enough that a 7B model handles it, which tells you something useful about how much capability your real workload actually needs.
It is also worth pricing the alternative before you commit to the tree at all. Self-hosting an open model shifts your spend from tokens to time - model updates, quantisation configs, evals to re-run every time weights change - and that time is the expensive part for any team that is not already running GPUs. Our breakdown of what AI automation actually costs puts that ongoing work next to the hosted-API line, and if the honest answer is that nobody wants to own it, having the agent built and handed over with documentation is a reasonable place to land. Open weights are a means, not a scoreboard.
Getting started: If you have no prior preference, start with Llama 3.3 70B on Ollama. It takes one command to install and run. Build your agent, evaluate it on your actual use cases, and measure where it fails. If failures are speed-related, try Llama 3.1 8B. If failures are reasoning-related, try DeepSeek V3. If failures are language-related, try Qwen 2.5 72B. Let your data - not benchmark tables - drive the final decision. For step-by-step deployment instructions, our guide to running agents on your own server walks through the full infrastructure setup. And if you want a tool that automates this decision process based on your specific constraints, try our AI Agent Stack Picker.
Once you have picked a model, the next decision is the serving engine: our Ollama vs vLLM guide covers which one fits your concurrency and hardware.
FAQ
- Can open-source models match GPT-4 or Claude for agent tasks?
- On single-turn function calling, yes: Llama 3.3 70B scores 88.0% on the BFCL leaderboard, level with Claude Opus 4.5 at 88.6% in the same category. The gap is in everything after the first call. On the overall BFCL score, which adds multi-turn, web search and memory tasks, Llama 3.3 70B scores 31.9% against 77.5% for Claude Opus 4.5. For most production agents with 5-15 tools and clear schemas, open-source 70B models are production-ready.
- How much VRAM do I need to run a 70B model for agents?
- At Q4_K_M quantization (the recommended sweet spot), Llama 3.3 70B needs approximately 40 GB VRAM. This fits on 2x RTX 3090 (24 GB each), 2x RTX 4090, a single NVIDIA A6000 (48 GB), or a single A100 80GB. You also need headroom for the KV cache, which grows with context length - budget an extra 4-8 GB for 32K-64K context agent sessions.
- Is quantization safe for agent models? Does it hurt tool calling accuracy?
- Q4_K_M and Q5_K_M quantization are generally regarded as having a small impact on output quality compared to FP16, while Q3 and lower degrade more noticeably. For agent workloads, Q4_K_M is the usual tradeoff: 4-bit weights take roughly a quarter of the memory of 16-bit ones. Run your own tool-calling test set at the quantization you plan to deploy, since the effect varies by model.
- Should I use Ollama or vLLM to serve models for agents?
- Use Ollama for development, prototyping, and single-user deployments - it is dead simple to set up and works well for one request at a time. Use vLLM for production deployments with concurrent users - it provides continuous batching (critical for throughput), an OpenAI-compatible API, and tensor parallelism for multi-GPU setups. The model quality is identical; the difference is serving infrastructure.
- Can I fine-tune these models to improve agent performance?
- Yes, and it is often the highest-ROI improvement. Fine-tuning a 7B model on 1,000-5,000 examples of your specific tool schemas and expected agent behavior can close much of the gap with a 70B base model. LoRA fine-tuning on a single GPU takes 2-4 hours. Start with the base model, measure failures, collect the failure cases as training data, and fine-tune.
If this resonated
Module 02 is the long version of this post.
Eval set, refusal pattern, kill switch. Build it in a weekend.
