Pre-filter vs. post-filter

Post-filter: Retrieve top-k, then discard chunks that don’t match your metadata criteria. Simple but wasteful — if most of your data is in the wrong category, you’ll discard most of your results and end up with low effective k.

Pre-filter: Apply metadata filters before similarity search, then retrieve top-k from the filtered subset. This is almost always what you want.

With pgvector, pre-filtering is a WHERE clause:

SELECT id, content, 1 - (embedding <=> $1) AS similarity
FROM document_chunks
WHERE metadata->>'course_id' = $2
  AND metadata->>'content_type' = 'lecture'
ORDER BY embedding <=> $1
LIMIT 10;

The index interaction problem

If your filter selects a very small fraction of rows (< 5%), PostgreSQL may fall back to a sequential scan rather than the index. You can check with EXPLAIN:

For highly selective filters, consider a partial index:

CREATE INDEX ON document_chunks
USING hnsw (embedding vector_cosine_ops)
WHERE metadata->>'content_type' = 'lecture';

Metadata schema design

Metadata that you’ll filter on should be promoted to top-level jsonb keys (or dedicated columns).

Good:

{ "course_id": "CS4320", "content_type": "lecture", "week": 3 }

Multi-tenant isolation

For multi-tenant RAG (e.g., per-user or per-course knowledge bases), metadata filtering is your access control layer. Every query must include a tenant filter, and that filter should be enforced at the application layer:

def retrieve(query: str, course_id: str, k: int = 10) -> list[Chunk]:
    embedding = embed(query)
    return db.query(
        "SELECT ... WHERE metadata->>'course_id' = $1 ORDER BY embedding <=> $2 LIMIT $3",
        course_id, embedding, k
    )

Never let course_id come directly from user input without validation against the authenticated user’s allowed courses.

The short version

• HNSW: Better recall, faster queries, higher memory cost, slower build time. Default choice for most applications.
• IVFFlat: Lower memory, faster builds, worse recall unless you tune it. Better for very large datasets where you can’t afford HNSW’s memory overhead.

HNSW

Hierarchical Navigable Small World graphs navigate a multi-layer graph structure to find approximate nearest neighbors. Once built, queries are fast and recall is high.

CREATE INDEX ON document_chunks
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

• m: Number of connections per node. Higher = better recall, more memory. Default 16 is fine for most cases.
• ef_construction: Search width during index build. Higher = better recall, slower build. 64–128 is typical.

At query time, set hnsw.ef_search to control the recall/speed tradeoff:

SET hnsw.ef_search = 100;
SELECT * FROM document_chunks
ORDER BY embedding <=> query_embedding
LIMIT 10;

IVFFlat

Inverted file index with flat quantization. Clusters vectors into lists buckets at build time; queries search probes buckets.

CREATE INDEX ON document_chunks
USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);

Rule of thumb: lists = rows / 1000 for datasets under 1M rows, sqrt(rows) for larger.

At query time:

SET ivfflat.probes = 10;

More probes = better recall, slower queries. IVFFlat requires the table to be populated before building — you can’t build it on an empty table and add rows later without degrading recall.

Recall testing

Always test recall for your specific dataset. Generate a set of queries with known ground truth (exact kNN via sequential scan), run your index queries, and measure overlap.

-- Exact nearest neighbors (no index)
SET enable_indexscan = off;
SELECT id FROM document_chunks
ORDER BY embedding <=> $1
LIMIT 10;

Performance table

*with probes = 10 on a well-tuned index

Separate concerns: source vs. derived data

Keep source documents in one table and their embeddings in another. This lets you re-embed without touching the source, and query each independently.

CREATE TABLE documents (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  source_url  text,
  content     text NOT NULL,
  metadata    jsonb,
  created_at  timestamptz DEFAULT now()
);

CREATE TABLE document_chunks (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  document_id uuid REFERENCES documents(id) ON DELETE CASCADE,
  chunk_index int NOT NULL,
  content     text NOT NULL,
  embedding   vector(1536),
  metadata    jsonb,
  created_at  timestamptz DEFAULT now()
);

The metadata jsonb column on chunks is important — it’s what enables filtering before or after similarity search.

Audit tables for LLM interactions

Every query sent to an LLM and every response received should be logged. This is essential for debugging, evaluation, and compliance.

CREATE TABLE llm_interactions (
  id           uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  session_id   uuid,
  model        text NOT NULL,
  prompt       text NOT NULL,
  response     text,
  latency_ms   int,
  tokens_in    int,
  tokens_out   int,
  created_at   timestamptz DEFAULT now()
);

Indexing strategy

For vector columns, an HNSW index on document_chunks.embedding is almost always the right choice. See pgvector indexing strategies for details on when to use IVFFlat instead.

jsonb vs. dedicated columns

Use jsonb for metadata that varies across records (tags, categories, source-specific fields). Use dedicated columns for fields you’ll frequently filter or sort on — PostgreSQL can’t use standard indexes on jsonb internals efficiently without expression indexes.