Embeddings and Vector Search: How Semantic Search Actually Works

Vector search came up again and again in this blog’s Advanced RAG series, but there was never a post covering embeddings and vector search themselves from the ground up. This post fills that gap. The conclusion up front: embeddings turn text into numeric vectors that preserve meaning, and vector search finds the nearest ones in that vector space. That is why a query for “refund policy” can find a document titled “payment cancellation terms” with barely any overlapping keywords.

Embeddings: turning text into coordinates #

An embedding model takes text and outputs a fixed-length array of real numbers — a vector. Dimensionality ranges from hundreds to thousands depending on the model. The one property that matters: semantically similar texts land close together in the vector space. “My laptop battery drains fast” and “my notebook doesn’t hold a charge” share almost no words, but their embedding vectors come out close.

A few practical facts are worth knowing alongside that.

• Embedding models are separate from generative LLMs. They are also far cheaper per call, so embedding a large document corpus is usually a small share of total cost.
• Beyond text, images and code can be embedded too, and multimodal embeddings place text and images in the same space.
• Embedding models have input token limits of their own. You do not embed whole documents; you split them into appropriately sized pieces first, and this splitting (chunking) heavily determines search quality. Chunking strategies are covered in Advanced RAG #2.

Similarity: how closeness is measured #

How close two vectors are is usually measured one of three ways.

Most embedding models either output normalized vectors or recommend cosine similarity, so absent a specific reason, use whatever the model provider recommends. One caution: the absolute scale of similarity scores differs across models, so a threshold like “0.8 or above means relevant” has to be re-tuned whenever you switch models.

Vector search: from exhaustive scan to approximate search #

Search ultimately means “find the k vectors nearest to the query vector” (kNN). The simplest implementation computes the distance to every stored vector — an exhaustive scan — and up to tens of thousands of items that is genuinely enough. One line of NumPy matrix math does it, at 100% accuracy.

Scale is the problem. At millions to hundreds of millions of items, exhaustive scans become slow, so you use an ANN (Approximate Nearest Neighbor) index. The flagship algorithm is HNSW, which links vectors into a multi-layer graph and walks down the graph to narrow candidates. In exchange for not comparing everything, you may miss the true nearest neighbors; that accuracy is called recall. ANN indexes universally expose parameters that trade speed, recall, and memory against each other.

Choosing storage: a dedicated DB is not always the answer #

Where to store and search vectors splits three ways.

1. Vector extensions of your existing database: pgvector for PostgreSQL is the representative example. If you already run PostgreSQL, you add a vector column with no new infrastructure, and handle metadata filtering and joins in the same SQL. Practical well into the millions of rows. Elasticsearch, OpenSearch, and Redis also support vector search.
2. Dedicated vector databases: Pinecone, Qdrant, Weaviate, Milvus, and others. They offer indexing, sharding, and filtering optimized for large vector volumes. Consider them beyond tens of millions of items, or when vector search is on your service’s critical path.
3. Embedded libraries: embedding something like FAISS directly in the application. Good for batch jobs and small-scale search with no server, but persistence and concurrency are on you.

The decision rule is simple. Start with the database you already operate, and move to a dedicated one when scale or latency actually becomes a problem. Adopting a dedicated vector DB from day one buys you the operational cost of one more piece of infrastructure before it buys you anything else.

The weakness: exact matching, and hybrid search #

Vector search is strong on semantic closeness but weak on exact string matches. For queries where the literal notation matters more than meaning — error code E4032, model name RTX 5090, an internal project codename — keyword search (BM25) beats vector search. That is why production search systems typically run vector and keyword search together and merge the results; hybrid is close to the standard. The concrete merging techniques are covered in Advanced RAG #3: Hybrid Search.

What you meet in operations #

• Model change = full re-index: embedding vectors are bound to the model that produced them. They cannot be compared against another model’s vectors, so switching embedding models means re-embedding every stored document. With a large corpus, design the re-indexing pipeline in from the start.
• Dimensions and memory: one 1536-dimension float32 vector is about 6KB. Ten million of them is 60GB of vectors alone, and ANN indexes typically hold this in memory. Dimension reduction and quantization can shrink it, in trade against recall.
• Freshness: when a document changes, its chunks must be re-embedded and updated. Any sync lag between the source and the index shows up directly in search results.

Summary #

• Embeddings turn text into meaning-preserving vectors; vector search finds the nearest ones in that space. That is how search works by meaning rather than keywords.
• Small scale is fine with exhaustive scans; large scale trades recall for speed with ANN indexes like HNSW.
• For storage, starting with a vector extension of your existing database (such as pgvector) and moving to a dedicated vector DB as scale demands is the safe order.
• Exact matching is a weakness, so hybrid setups combining keyword search are the practical standard.
• Switching embedding models forces a full re-index. Plan for it in the operational design up front.