How to add AI semantic search to your app

What is semantic search and how is it different from keyword search?

Keyword search matches words. Semantic search matches meaning: a query for “cancel my plan” can find a help article titled “Ending your subscription” even though no words overlap.

It works by converting text into embeddings, lists of numbers where similar meanings sit close together. Searching becomes finding the stored vectors nearest to the query vector, usually by cosine similarity.

Semantic search is weaker at exact matches such as SKUs, error codes or names. That is why most production systems combine both methods instead of replacing keyword search.

Users rarely know the exact words your content uses, which is why semantic search tends to help most in help centres, documentation, catalogs with rich descriptions and internal knowledge bases.

It is also the retrieval layer behind RAG chatbots, so building it well pays off twice if you later add AI answers on top.

Which storage option should you use for vectors?

How to add semantic search step by step (Postgres example)

• Decide what a searchable document is: a product, an article section, a ticket. Combine its title and key fields into one text string to embed.
• Pick an embedding model: a hosted embeddings API, or an open model run through sentence-transformers or Ollama. Note its vector dimension.
• Enable the extension: CREATE EXTENSION IF NOT EXISTS vector;
• Add a column sized to your model, for example ALTER TABLE articles ADD COLUMN embedding vector(768); matching your model’s dimension.
• Backfill: embed existing rows in batches and write the vectors; keep a column recording which model produced them.
• Create an index: CREATE INDEX ON articles USING hnsw (embedding vector_cosine_ops);
• At query time, embed the query with the same model and run ORDER BY embedding <=> $1 LIMIT 20, where <=> is pgvector’s cosine distance operator.
• Apply filters in the same query (tenant, language, visibility) so users only see what they may see.
• Keep embeddings fresh: re-embed a row whenever its searchable text changes, using a queue or background job.

How do you combine keyword and vector results?

Hybrid search runs a keyword query (Postgres full-text search or BM25 in a search engine) and a vector query, then merges the two ranked lists. A common, simple merging method is reciprocal rank fusion, which scores each result by its positions in both lists.

Hybrid search fixes the classic failure where semantic search misses an exact product code or person’s name while still handling paraphrased questions well.

If your app already runs on Postgres, start with pgvector and move to a dedicated engine only when measurements show you need it. One database means one backup, one permission model and no sync job between systems.

A dedicated engine earns its place when collections grow very large, filtering and hybrid ranking become complex, or search load would compete with your transactional database.

• Run both queries with the same filters and take the top 20–50 from each.
• Merge with reciprocal rank fusion or a weighted score you tune on real queries.
• Optionally rerank the merged top results with a cross-encoder reranker for better ordering.
• Show highlights or snippets so users understand why a result matched.

How much does AI search cost to run?

Embedding cost is mostly one-off: you embed each record once and again only when it changes. Queries add one small embedding call each, which you can cache for popular searches.

Storage grows with the number of records times the vector dimension, so smaller-dimension models and fewer embedded fields reduce memory. Running an open embedding model on your own server swaps per-call fees for CPU or GPU time.

Latency matters as much as money. Each search now includes an embedding call before the database query, so a hosted embedding API adds a network round trip. Caching query embeddings and running a small local embedding model close to the database both help keep search responsive.

Reranking adds quality but also time and cost per query, so apply it only to the top few dozen candidates rather than the full result set.

Budget for re-embedding too. Switching to a better model later means embedding every record again, which is a one-off job but worth planning for when you choose the first model.

Common mistakes when adding semantic search

• Mixing vectors from different models or model versions in one index; results become meaningless.
• Embedding whole long documents as one vector, which blurs meaning; split long content into sections.
• Dropping keyword search entirely and breaking searches for exact identifiers.
• Filtering after retrieving the top results, so filtered-out items leave the user with too few matches; filter in the query.
• No evaluation set: collect real queries with expected results and track whether changes improve them.
• Forgetting multilingual users: pick a multilingual embedding model if queries and content differ in language.

When should search become a RAG answer?

Once retrieval works, you can pass the top results to an LLM to write a direct answer with citations, turning search into a question-answering feature. Build good search first; answer generation cannot fix poor retrieval.

RepoLoot’s catalog lists open-source search and vector projects with notes on difficulty, useful when choosing between extending Postgres and running a dedicated engine.

How do you evaluate search quality?

Collect fifty to a hundred real queries from logs or support tickets, and for each one note the results a good search should return. This small labelled set is the most valuable asset in any search project.

Measure simple metrics such as whether an expected result appears in the top five, and read failures by hand. Patterns appear quickly: missing synonyms, over-long chunks, a filter bug or a weak embedding model for your language.

• Log queries that return no clicks; they reveal gaps in content or ranking.
• Compare keyword-only, vector-only and hybrid on the same set before choosing.
• Re-run the set after every change to the model, chunking or ranking weights.
• Track latency too; a search box that feels slow gets used less.

How do you design the search UI around semantic results?

Semantic search changes what users can type, so the interface should invite natural questions as well as keywords. A placeholder such as “Describe what you are looking for” signals that full sentences work.

Always show why a result matched: the title, a relevant snippet and any active filters. Semantic results can look surprising, and snippets help users trust them.

• Keep facets and filters visible; they narrow meaning-based results effectively.
• Debounce search-as-you-type to avoid an embedding call on every keystroke.
• Offer a fallback message and suggestions when results are weak rather than showing unrelated items.

Frequently asked questions

Do I need a vector database to add semantic search?

No. If your app runs on Postgres, the pgvector extension stores embeddings next to your data and supports approximate nearest-neighbour indexes such as HNSW. A dedicated vector database becomes attractive when you have very large collections, heavy filtering needs or want features like built-in hybrid search.

Which embedding model should I use?

Choose based on language coverage, dimension and whether it must run locally. Test two or three candidates, hosted or open, on a sample of your real queries and documents, and keep the one that ranks expected results highest. Public leaderboards help shortlist but do not replace testing on your data.

What is hybrid search?

Hybrid search combines keyword matching, such as BM25 or Postgres full-text search, with vector similarity and merges the ranked lists, often using reciprocal rank fusion. It keeps exact matches for names and codes reliable while still finding results phrased differently from the query.

How do I keep embeddings up to date?

Re-embed a record whenever its searchable text changes, typically through a background job triggered by the update. Store which model produced each vector, and when you switch models, re-embed everything before querying with the new model, because vectors from different models are not comparable.