Skip to content
Posts en inglés. Usá el traductor del navegador para leerlos en tu idioma.

How to Estimate Production Costs for a RAG System

Yammbo
· 9 min read
Copyright: CC BY-NC-SA 4.0
ai production costs llm deployment cost vector database pricing inference costs rag cost breakdown
How to Estimate Production Costs for a RAG System

Deploying a Retrieval Augmented Generation (RAG) system in production involves several components, each contributing to the overall operational cost. Many teams initially overestimate one-time setup costs like document embedding and vector storage, while underestimating the recurring expenses associated with query processing. This tutorial provides a practical, step-by-step guide to accurately estimate the production costs for a RAG system, helping you understand the true financial drivers and make informed architectural decisions.

Step 1: Deconstructing the RAG Pipeline for Cost Analysis

Before diving into numbers, it's crucial to understand the lifecycle of data within a RAG system and identify where costs accrue. A typical RAG pipeline involves several stages, each with its own computational and storage requirements.

Document Ingestion and Preprocessing

This initial phase involves preparing your knowledge base. Documents are often cleaned, split into smaller, manageable 'chunks,' and then converted into numerical representations called embeddings. Chunking strategies (e.g., fixed size with overlap) directly impact the total number of tokens processed for embeddings.

Embedding Generation

Once chunked, each text segment is passed through an embedding model, which transforms it into a high-dimensional vector. This is typically a one-time cost for your entire corpus, though it might recur if your corpus frequently updates.

Vector Storage and Indexing

The generated vectors, along with references back to their original text chunks, are stored in a vector database or a traditional database augmented with vector capabilities (like PostgreSQL with pgvector). This storage incurs a recurring monthly cost.

Query Processing at Inference Time

This is where the variable costs come into play. When a user submits a query:

  1. Query Embedding: The user's query is also converted into an embedding using the same model as the document chunks.
  2. Retrieval: The query embedding is used to find the most semantically similar document chunks from your vector store.
  3. Reranking (Optional but Recommended): A smaller, more precise model often re-evaluates the top retrieved chunks to select the most relevant subset for the Large Language Model (LLM). This step significantly improves answer quality but adds to token processing.
  4. Answer Generation: Finally, an LLM takes the user's query and the selected relevant chunks to generate a coherent answer. This involves processing both input (query + chunks) and output (generated answer) tokens.

Understanding these stages helps pinpoint where your budget will actually be spent.

Step 2: Calculating One-Time Ingestion Costs

The initial cost of ingesting your entire document corpus into the RAG system is often a one-time expense. While it might seem substantial, it's typically a minor fraction of the overall production cost compared to ongoing inference.

Let's consider a corpus of 100,000 documents, each averaging 1,500 tokens. After chunking with a strategy like 512 tokens per chunk and a 64-token overlap, the total embedded tokens might expand to approximately 171.4 million tokens (a 1.14x expansion factor due to overlap).

Using an embedding model like Qwen3 Embedding 0.6B, which might be priced at $0.04 per million tokens, the calculation is straightforward:

171.4 million tokens * ($0.04 / 1 million tokens) = $6.86

This shows that embedding a substantial corpus can be surprisingly inexpensive. Even with optional steps like chunk enrichment (e.g., using a small LLM to summarize or extract keywords from chunks), the one-time cost remains relatively low. For instance, processing an additional 218 million input tokens and 20 million output tokens for enrichment might add around $18.92, bringing the total ingestion cost to under $20. These costs are fixed for your corpus size and do not scale with query traffic.

Verification Note:

To verify this step, identify your corpus size, average document token count, and chunking strategy. Use the pricing of your chosen embedding model to calculate the total cost. If your corpus doubles, this cost will double proportionally.

Step 3: Estimating Fixed Monthly Storage Costs

After generating embeddings, these vectors need to be stored in a database that supports efficient similarity search. While specialized vector databases exist, a common and cost-effective approach is to use a managed relational database like PostgreSQL, enhanced with an extension like pgvector.

The storage cost for your vector index is a fixed monthly expense, largely independent of query traffic. For a corpus of 100,000 documents resulting in roughly 335,000 chunks, storing these vectors and their associated metadata typically requires a modest amount of database storage.

For example, a managed PostgreSQL instance on a cloud provider might offer tiers like:

  • 4 GiB Single Node: Approximately $60 per month.
  • 4 GiB High Availability Pair: Approximately $120 per month (for redundancy).
  • 8 GiB High Availability Pair: Approximately $240 per month (for larger corpora).

The exact cost depends on your chosen provider, instance size, and whether you opt for high availability. This cost remains constant regardless of how many queries your RAG system processes, making it a predictable part of your budget. For most initial deployments, a 4 GiB instance is often sufficient for a 100,000-document corpus.

Verification Note:

Check your chosen cloud provider's managed database pricing for PostgreSQL or your preferred vector database. Select an instance size that accommodates your vector storage needs and note its monthly cost. This will be your base fixed cost.

Step 4: Analyzing Variable Query-Time Costs

This is where the majority of your RAG system's operational budget will be spent, as these costs scale directly with the number of queries your system handles. Query-time costs are primarily driven by token consumption for query embedding, reranking, and answer generation.

Query Embedding Costs

Each user query needs to be embedded. If a typical query is 32 tokens, and you process 1,000 queries per day (30,000 queries per month), the token count is relatively low. Using the same $0.04 per million tokens rate:

30,000 queries/month * 32 tokens/query = 0.96 million tokens/month0.96 million tokens/month * ($0.04 / 1 million tokens) = $0.0384 per month

Even at 100,000 queries per day, this cost only rises to about $3.84 per month, showing that query embedding itself is a negligible cost driver.

Reranking Costs

Reranking is a critical step for improving retrieval quality. It involves sending a subset of retrieved documents (e.g., 20 candidates) along with the query to a dedicated reranking model. This process consumes significant tokens.

For 20 retrieved candidates, this might translate to approximately 10,400 input tokens per query (query + 20 chunks) and 40 output tokens. Using a model like DeepSeek V4 Flash at rates of $0.068 per million input tokens and $0.168 per million output tokens:

For 1,000 queries/day (30,000 queries/month):Input tokens: 30,000 * 10,400 = 312 million tokensOutput tokens: 30,000 * 40 = 1.2 million tokensCost: (312M * $0.068/M) + (1.2M * $0.168/M) = $21.216 + $0.2016 = ~$21.42 per month

As traffic increases, this cost scales linearly. At 10,000 queries/day, it becomes around $214 per month; at 100,000 queries/day, it jumps to approximately $2,142 per month. Reranking quickly becomes a significant cost center.

Answer Generation Costs

The final and often most expensive step is generating the answer using a powerful LLM. This involves providing the query and the top relevant chunks (e.g., 5 chunks) as input to the model.

For 5 top chunks, this might be around 2,900 input tokens per query and 350 output tokens for the generated answer. Using a model like gpt-oss-120b at rates of $0.10 per million input tokens and $0.70 per million output tokens:

For 1,000 queries/day (30,000 queries/month):Input tokens: 30,000 * 2,900 = 87 million tokensOutput tokens: 30,000 * 350 = 10.5 million tokensCost: (87M * $0.10/M) + (10.5M * $0.70/M) = $8.70 + $7.35 = ~$16.05 per month

Similar to reranking, this cost scales directly with traffic. At 10,000 queries/day, it's about $160 per month; at 100,000 queries/day, it's approximately $1,605 per month. The choice of LLM here is critical, as more advanced or larger models often have higher per-token costs. For instance, using Llama 3.3 70B (hypothetically at $0.65 in / $0.65 out per million tokens) would significantly increase these generation costs.

Verification Note:

Define your expected query volume, average query length, number of chunks for reranking, and number of chunks for generation. Use the specific pricing for your chosen reranking and generation models to project monthly costs at various traffic levels.

Step 5: Bringing It All Together: Total Monthly Costs and Optimization

When you combine all the components, a clear picture of your RAG system's operational costs emerges. The one-time ingestion and fixed monthly storage costs are quickly dwarfed by the variable query-time expenses as traffic grows.

Here's an illustrative summary of monthly costs for a 100,000-document RAG system, assuming the model prices and token counts detailed above:

  • At 1,000 queries per day (30,000/month):
    • Query embeddings: ~$0.04
    • Reranking (DeepSeek V4 Flash): ~$21.50
    • Answer generation (gpt-oss-120b): ~$16.05
    • Managed PostgreSQL (4 GiB single node): ~$60.00
    • Total: ~$97.59 per month
  • At 10,000 queries per day (300,000/month):
    • Query embeddings: ~$0.38
    • Reranking (DeepSeek V4 Flash): ~$215.00
    • Answer generation (gpt-oss-120b): ~$161.00
    • Managed PostgreSQL (4 GiB HA pair): ~$120.00
    • Total: ~$496.38 per month
  • At 100,000 queries per day (3,000,000/month):
    • Query embeddings: ~$3.84
    • Reranking (DeepSeek V4 Flash): ~$2,150.00
    • Answer generation (gpt-oss-120b): ~$1,605.00
    • Managed PostgreSQL (8 GiB HA pair): ~$240.00
    • Total: ~$3,998.84 per month

These figures clearly illustrate that reranking and answer generation are the dominant cost factors, accounting for over 99% of the model spend at higher traffic levels. The database cost, while significant at low volumes, becomes a smaller percentage of the total bill as query volume increases.

Optimization Strategies:

To manage costs effectively, focus your optimization efforts on:

  • Model Selection: Choose reranking and generation models carefully. Smaller, more efficient models can drastically reduce per-token costs.
  • Number of Retrieved Chunks: Experiment with the number of chunks sent to the reranker and the final LLM. Reducing these numbers, even slightly, can lead to substantial savings.
  • Prompt Engineering: Optimize prompts to minimize input and output token counts for answer generation.
  • Caching: Implement caching for common queries or frequently accessed information to reduce the need for full RAG pipeline execution.

Verification Note:

Use the formulas and your specific assumptions to create your own cost projection spreadsheet. Regularly review actual usage and model pricing to keep your estimates accurate.

Accurately forecasting the costs of a RAG system is vital for successful deployment and scaling. By focusing on the variable costs associated with query processing, particularly reranking and answer generation, you can make strategic decisions that optimize your budget without compromising performance. For more insights into building robust digital infrastructure, explore the resources available at Yammbo.