When an enterprise RAG system underdelivers, the instinct is to blame the AI model. In practice, the model is rarely the problem. The real gap sits upstream, in the discipline of embedding generation for RAG, the unglamorous engineering work of preparing data, choosing a model, and running the pipeline that turns raw company knowledge into something an AI can actually search. Get this stage wrong, and you get a chatbot that confidently returns the wrong answer. Get it right, and you get a system that finds the correct answer, fast, at a cost your finance team can defend.
This guide walks through the full framework for producing reliable embeddings that power RAG at scale, from data preparation through cost control and ongoing monitoring, in plain language a technical leader and a CEO can both act on.
The numbers behind the urgency. Vector databases, the infrastructure layer that stores every embedding your pipeline produces, grew 377 percent year over year according to Databricks’ State of Data and AI report, based on activity across more than 10,000 global customers including over 300 Fortune 500 companies. That is the fastest growth of any AI related technology category Databricks tracks. The market backing that growth is expanding just as fast. MarketsandMarkets projects the global retrieval augmented generation market to climb from 1.94 billion dollars in 2025 to 9.86 billion dollars by 2030, a 38.4 percent annual growth rate. And Gartner’s 2025 AI survey found that RAG has already become the dominant architecture in production enterprise AI, used in 63 percent of enterprise AI deployments. Put simply, the RAG embeddings pipeline sitting quietly underneath your AI roadmap is no longer a side project. It is the fastest growing, most heavily invested layer of enterprise AI infrastructure, and the embedding quality you build there compounds directly into the quality of every product built on top of it.
Why this matters more than most teams realize. Vector embeddings are the layer that determines whether your large language model application is grounded in your actual business reality or guessing based on generic training data. Every AI assistant, every internal search tool, and every customer facing recommendation feature you plan to ship this year depends on the embedding quality sitting underneath it. A pipeline with weak embedding quality does not fail loudly. It fails quietly, in the form of a customer support bot that sounds confident while pointing a customer to the wrong policy, or a sales enablement tool that misses the one case study that would have closed the deal. Treating embedding generation for RAG as a core engineering discipline, not an afterthought bolted onto the language model, is the difference between a pilot that stalls and a production RAG system the whole company relies on.
The Embedding Generation Workflow
Producing RAG embeddings for enterprise applications is not primarily a modeling problem. It is a discipline problem, about what happens before the model call and what happens after it. Four things determine whether your pipeline produces reliable embeddings or expensive noise.
Data preparation comes first. Clean your source content before anything touches an embedding model. Strip boilerplate such as page headers, footers, navigation menus, and repeated legal disclaimers, since all of it pollutes the meaning of the text you are trying to capture. Deduplicate aggressively. A support article that exists in three near identical versions, or a product description copied across five regional pages, wastes embedding spend and causes your retriever to return repetitive, low value results instead of genuinely useful ones. Then chunk the cleaned content into retrieval sized pieces, a decision important enough that it gets its own full section below.
Embedding model selection is the second decision. Your options generally fall into three camps. A hosted API such as OpenAI’s text embedding models, sometimes referred to loosely in the market as Ada style embeddings, gives you strong general purpose accuracy without managing any infrastructure. A self hosted open source model, commonly built on architectures such as Mistral 7B, gives you full control over cost and data residency in exchange for owning the infrastructure. And a fine tuned domain specific model gives you the highest accuracy for specialized language, at the cost of a training investment. We cover all three in depth in the next section.
Batch versus real time processing is the third tradeoff. Most production systems need both paths. A scheduled batch job, run nightly or weekly, handles the bulk of your embedding volume, refreshing your product catalog or your documentation library at the lowest possible per token cost. A real time path handles the exceptions that cannot wait, such as a customer uploading a document and expecting to search it immediately, or a new support ticket that needs to be findable the moment it is created. Building both paths from the start avoids a painful rearchitecture later, once your real time volume grows past a trickle.
Quality metrics are the fourth piece, and the one teams skip most often. Before assuming a new embedding model will fix a retrieval problem, measure three things against your current pipeline. Diversity, meaning whether your top results represent genuinely different information or five near paraphrases of the same paragraph. Relevance, meaning whether a human reviewer would actually pick your top results for a given query. And coverage, meaning whether entire sections of your knowledge base are never being retrieved at all, which usually points to a chunking problem rather than a model problem.
Chunking Strategies
If you improve exactly one part of your embedding pipeline, make it this one. Chunk size and chunk structure have more influence on retrieval quality than the embedding model you choose, and they are almost always the first place to look when a RAG system returns results that are close, but not quite right.
Why chunk size matters so much. A chunk that is too small strips away the surrounding context a sentence needs to make sense. A sentence that reads it can cause this reaction under certain conditions is meaningless on its own if the preceding paragraph, which explained what it refers to, was cut into a different chunk. A chunk that is too large has the opposite problem. A 2,000 token chunk covering three unrelated subtopics produces an embedding that is a blurry average of all three, and it will match poorly against a query that is specific to just one of them. The right chunk size sits in a narrow band, typically a few hundred tokens, where a chunk holds one complete idea and nothing more.
Three chunking techniques, and when to use each.
- Fixed size chunking splits text into equal token counts, such as every 300 tokens, regardless of sentence or paragraph structure. It is simple to implement and completely predictable, but it will happily split a sentence, a table row, or a numbered list step right down the middle.
- Semantic chunking splits at natural boundaries, using paragraph breaks, section headers, or a measured shift in topic between adjacent sentences to decide where one chunk ends and the next begins. It costs more compute up front, since you are effectively analyzing the document before you chunk it, but it produces chunks that align with how a human reader would naturally divide the material.
- Hierarchical chunking preserves the document’s own structure. Each chunk is tagged with metadata about its parent section and document, so retrieval can return a specific chunk while still knowing exactly which manual, which chapter, and which heading it came from. This matters enormously for technical documentation and policy manuals, where the section heading itself carries meaning that the chunk text alone does not.
Why sliding window overlap works. A hard, non overlapping cut between chunk one and chunk two risks slicing a single idea, or a single instruction, exactly in half, so that neither chunk fully represents it on its own. Overlapping adjacent chunks by a modest window, commonly 10 to 20 percent of the chunk size, ensures that content sitting near a boundary appears fully intact in at least one chunk, even if it is partially duplicated across two. The tradeoff is a proportional increase in the number of chunks you produce and store. It is a cost almost always worth paying for the retrieval accuracy it buys back.
A concrete example: chunking a customer support manual. Take a 50KB support manual, roughly 12,500 tokens of raw text. Chunked at 300 tokens per chunk with a 50 token overlap, the effective stride between chunks is 250 tokens, producing approximately 50 retrieval ready chunks. The measurable result teams report from this setup is fewer almost right answers, meaning cases where the system clearly located the correct section of the manual but returned a chunk that was missing the one troubleshooting step the customer actually needed, because a fixed, non overlapping cut had severed it from its context. Better retrieval across section boundaries is the direct, practical payoff of getting the overlap right.
Embedding Model Selection
Choosing an embedding model is where most teams either save significant money or quietly overspend for months without noticing. Here is how the three main paths compare on accuracy, cost, and latency.
OpenAI’s text embedding models. OpenAI’s current small embedding model, the successor to the earlier Ada generation that many practitioners still refer to informally as Ada style embeddings, is priced at roughly $0.02 per 1 million tokens on the standard tier, with a batch processing option available at roughly half that rate for jobs that do not need same second results. It remains a strong default for teams who want dependable general purpose accuracy without managing any inference infrastructure themselves. Latency is consistently low, since you are calling a globally distributed, managed API rather than a service you provision and scale yourself. Always confirm current pricing directly against the provider’s published rates before finalizing a budget, since these figures are revised periodically.
Mistral embeddings, and Mistral 7B based open source alternatives. Mistral AI offers a dedicated hosted embedding API, and the broader open source ecosystem has also built strong embedding models on top of the 7 billion parameter Mistral architecture, giving teams a self hosted path with near zero marginal cost per embedding once deployed. The tradeoff is that you take on GPU provisioning, model serving, and ongoing operational maintenance yourself. In exchange, you gain full control over data residency, meaning nothing about your proprietary content ever leaves your own infrastructure, along with the ability to fine tune the model on your own domain language.
LLaMA based embeddings and other open weight options. These models are free in the sense that the weights themselves cost nothing to download and use. But free weights still require GPU infrastructure to serve at any meaningful throughput, and that infrastructure is not free. For low volume or batch only workloads, self hosting an open weight embedding model can genuinely beat the cost of a hosted API. For high throughput, latency sensitive, real time embedding generation, the engineering time required to keep GPU utilization efficient often erases that apparent cost advantage, particularly for teams without existing MLOps capacity.
Embedding quality compared.
| Model | Cost Model | Quality on General Text | Latency | Data Residency |
| OpenAI hosted embedding API | Roughly $0.02 per 1M tokens standard, lower via batch | High, strong general purpose baseline | Low, fully managed | Leaves your infrastructure |
| Mistral 7B based, self hosted | Upfront setup cost, low marginal cost per embedding | High, competitive with closed models | Depends on your own serving setup | Stays entirely in house |
| LLaMA based, self hosted | Free model weights plus GPU infrastructure cost | Good, varies by specific fine tune | Depends on your own serving setup | Stays entirely in house |
| Fine tuned domain model | Training cost plus ongoing hosting | Highest for your specific domain | Depends on your own serving setup | Stays entirely in house |
When fine tuning earns its cost. Fine tuning becomes worthwhile once your domain vocabulary diverges meaningfully from general web text, such as legal contract language, clinical documentation, or internal engineering and product terminology specific to your company. The clearest signal to watch for is a retrieval evaluation that keeps surfacing results which are topically adjacent but substantively wrong, such as a contract clause about indemnification being returned for a query about liability limitation. That pattern usually means a general purpose model is not distinguishing your domain’s fine grained distinctions, and a model fine tuned on labeled query and document pairs from your own data can close that gap directly.
Cost Optimization
In a production RAG deployment, embedding API costs are often the most misunderstood line item in the budget. On a per token basis, embeddings are dramatically cheaper than the language model calls that follow them, which leads some teams to ignore the cost entirely, right up until volume and refresh frequency catch up with the bill.
A worked example: e-commerce with 1 million product descriptions. Assume an average product description of roughly 100 tokens, putting the initial embedding job at around 100 million tokens total.
| Approach | Initial Generation | Ongoing, Weekly Refresh |
| OpenAI hosted API | About $20 a month equivalent for the initial batch | About $40 a month for weekly refreshes of changed listings |
| Self hosted Mistral 7B based model | About $300 one time setup, covering GPU provisioning and deployment | About $50 a month in ongoing compute |
The crossover point is worth calculating explicitly rather than assuming. At low to moderate volume, the hosted API wins on both raw cost and, more importantly, on engineering time. Self hosting starts to make financial sense once volume or refresh frequency is high enough that the one time setup cost amortizes quickly against the per call savings, a threshold that industry benchmarks generally place somewhere in the range of 10 to 15 million embeddings a month, though your own numbers should be the deciding factor, not a rule of thumb.
Batching strategy: 100,000 embeddings in a single job. Most embedding providers support batch or asynchronous processing, and the savings are not just about avoiding network overhead from looping individual calls. Providers frequently price batch endpoints at roughly half the cost of synchronous, real time calls, since batch jobs can be scheduled during off-peak capacity on the provider’s side. For any workload that does not need same second results, such as backfilling 100,000 product embeddings for a new catalog or refreshing a knowledge base overnight, routing the job through a batch endpoint instead of looping synchronous calls is close to a free cost reduction, and it should be the default path for any bulk indexing operation.
Caching: never reprocess identical documents. The single most common source of wasted embedding spend is regenerating vectors for content that has not changed since the last time it was embedded. A content hash, checked against a cache before every embedding call, catches this instantly. If the hash already exists in your cache and the embedding model version has not changed, skip the API call entirely and reuse the stored vector.
Quality Assurance and Monitoring
Detecting stale embeddings and similarity drift. Protecting embedding quality over time means catching two distinct failure modes. The underlying content can change without triggering a re embed, leaving your vector store out of sync with reality. Or the embedding model itself can be upgraded or replaced, meaning old vectors no longer live in the same geometric space as new ones, silently degrading retrieval quality without producing a single error message. Guard against the first with a reliable content hashing pipeline, covered above. Guard against the second by versioning every stored embedding with the exact model ID and version that generated it, and treating any model change as a full re index event rather than an incremental update.
A/B testing embedding model changes. Before rolling a new embedding model, or a new chunking strategy, into production, run both the old and new approach against a fixed, representative set of test queries and compare the retrieved results directly. A shadow deployment, producing embeddings from the new approach in parallel without yet serving them to users, lets you measure the real difference in retrieval quality before committing to a full re index, which for a large corpus can be an expensive, hours long operation you do not want to repeat because of an untested regression.
Retrieval quality metrics: precision at K and NDCG. Precision at K measures, of your top K retrieved chunks, what fraction are genuinely relevant to the query, a direct and easy to explain signal for whether the retriever is finding good material. NDCG, or normalized discounted cumulative gain, goes further by weighting where in the ranking a relevant result appears, since a relevant chunk sitting at position one is worth far more than the same chunk sitting at position eight. This distinction matters in practice because most RAG pipelines only pass the first few retrieved chunks into the language model’s context window. Both metrics require a labeled evaluation set. Even a modest set of 50 to 100 representative queries with human judged relevant results is enough to catch a regression before it reaches production.
A monitoring dashboard: cost, latency, and quality in real time. A minimal production monitoring setup should track three things continuously. Embedding API cost per day, which catches runaway or duplicate reprocessing before the monthly invoice does. Retrieval latency, tracked at both the median and the 95th percentile, which catches infrastructure degradation before users start noticing a slower product. And retrieval precision, sampled on a rolling basis against your evaluation set, which catches silent quality drift long before it shows up as customer complaints. Teams that monitor only cost and latency, and skip the quality dimension entirely, tend to discover retrieval degradation only after users start complaining, by which point the root cause, whether it was a bad chunking change three deployments back or a partial re index that accidentally mixed embedding model versions, is far harder to trace.
A Quick Reference for Technical Leaders
Before you hand this off to your engineering team, three questions are worth asking directly, because the answers predict most of the retrieval problems your company will run into before they happen.
Has anyone measured your chunk size against your actual document types, or was it set once, early on, and left alone. Do you know today, without checking, whether your pipeline is reprocessing content that has not changed. And do you have a labeled evaluation set that would catch a quality regression before your customers do. Most teams building reliable embeddings for the first time can answer yes to one of these. Very few can answer yes to all three, and the gap between those two states is almost always where retrieval quality complaints originate.
Where Most Teams Actually Get Stuck
In practice, the gap is almost never which embedding model is best. It is the unglamorous middle of the pipeline: a chunking strategy decided once, in the first sprint, and never revisited as the document library grew. No caching layer, so the same product catalog gets reprocessed and rebilled every single week. No evaluation set, so nobody notices a quality regression until a customer complaint forces the investigation. These are the cheapest problems in the entire pipeline to fix early, and by far the most expensive to fix after the fact, once they are buried under months of production traffic and half remembered deployment history.
The Naveera Workflow
Naveera does not walk in and hand you a slide deck of best practices you already knew. We run a structured, five stage engagement that mirrors the exact pipeline covered in this guide, so every recommendation is grounded in your actual data, your actual volume, and your actual budget for production RAG systems.
Stage one, the audit. We pull a sample of your existing content and your current embedding pipeline, if one exists, and measure it against the same quality metrics covered above, diversity, relevance, and coverage. Within days, not weeks, you get a clear picture of where retrieval is underperforming and why.
Stage two, the chunking and model design. Using your actual document types, support manuals, product catalogs, contracts, or code, we design a chunking strategy, fixed, semantic, or hierarchical, and recommend the embedding model that fits your volume, your latency requirements, and your data residency constraints, whether that is a hosted API, a self hosted Mistral 7B based model, or a fine tuned domain specific option.
Stage three, the cost engineered build. We implement the pipeline itself, batching, caching, and deduplication included from day one, so you are never paying to reprocess content that has not changed. This is the stage where most of the 60 percent plus cost reductions our clients see actually get built in.
Stage four, the evaluation set. We build your labeled test set, typically 50 to 100 representative queries scored by your own team, and wire up precision at K and NDCG tracking so quality is measured, not guessed at.
Stage five, monitoring and handoff. We stand up a live dashboard tracking embedding cost, latency, and retrieval precision, then hand the fully documented system to your engineering team, or continue to operate it for you, whichever your business needs.
The result is not a one time recommendation. It is a working pipeline, measured against your own numbers, that your team can own from day one.
Bring In Naveera
Naveera has built and tuned embedding pipelines for organizations producing everything from thousands to hundreds of millions of vectors. Reliable embeddings are not an accident. They come from disciplined chunking, a deliberately chosen model, and a pipeline engineered to control embedding API costs before they control your budget. We know exactly where the waste tends to hide in these systems, because we have found it, and fixed it, for dozens of clients across e-commerce, fintech, and enterprise support.
If your team is running production RAG at real volume, and you are not certain whether your pipeline is leaking cost or leaving retrieval quality on the table, Naveera will find out for you and hand your CEO and your engineering team a clear, prioritized plan they can both act on the same week.
Running RAG at scale and want a straight answer on where the waste is? Book a free embedding cost audit with Naveera
