Meaning-aware retrieval on top of the full-text index. The engine embeds text with a real bge-small model (via the dl-embed C++ tool), retrieves candidates with multi-index hashing (MIH) over ITQ bit-codes, and re-ranks by exact int8 cosine — all stored in-relation and snapshot-versioned like everything else.
The vector tier is opt-in and adds a vendored ggml submodule (v0.20.2) plus the model:
# One-time: materialize ggml + the bge-small model (git-lfs tracked). $ git submodule update --init vendor/ggml $ git lfs pull # models/bge-small-en-v1.5-f16.gguf (67 MB) $ make dl-embed # build ./dl-embed (needs cmake) $ ./dl-embed self-test # golden-embedding gate (cosine >= 0.9999 vs reference) # Embed the corpus into the vector index (18 relations + one publish). $ ./dl-embed pipeline --db /tmp/db # Query it. $ ./dl -d /tmp/db vsearch 'affordable GPU rental' --k 10 $ ./dl -d /tmp/db vhybrid 'gpu rental' 'affordable GPU rental' --k 10
Each entity is embedded and binarized to a 256-bit ITQ signature. The signature is split into m=16 bands; for band j the store keeps a fixed-arity relation __sig{j}__(band_u32, entity_sym_id) — a postings index from bit-substring to entity sym-id, the same shape as the full-text postings.
__sig0__..__sig15__ (arity 2) band_value -> entity_sym_id MIH candidate postings __vec_q__ (arity 3) entity_sym_id, chunk_idx, packed_4x_int8_u32 re-rank vectors __itq_basis__ (arity 3) dim_i, dim_j, float32_bits_u32 ITQ encode matrix (pinned)
A query embeds the text with the same model, ITQ-encodes it, then for each band enumerates the query’s substring variants within the pigeonhole budget ⌊r/m⌋, probes each via dl_prefix over __sig{j}__, unions the candidates across bands, filters to live entities, and re-ranks by exact integer int8 cosine. All reads are snapshot-versioned like any other relation, so dl_vector_search_version and dl search --version give you time-travelling semantic search for free.
Semantic vector search over the MIH postings + int8 re-rank.
dl [-d <dir>] vsearch '<query>' [--k N] [--radius R] [--version V]
[--sig <hex64>] [--ivec <hex768>]--version V queries as-of a published snapshot (0 = live). For a programmatic path with no model, pass the pre-encoded query as --sig (8 u32 = 64 hex) and --ivec (96 u32 = 768 hex).
Lexical ∩ semantic hybrid: intersect search results with vsearch candidates, then re-rank the intersection.
dl [-d <dir>] vhybrid '<terms>' '<query>' [--k N] [--radius R] [--version V]
The symbolic half: AND-intersect tokenized terms over __postings__ and rank by co-occurrence. --version N queries as-of a snapshot.
dl [-d <dir>] search '<terms>' [--top N] [--version N]
The C++ embedding tool (ggml-based). It runs the real bge-small-en-v1.5 model in-process — the weights are the actual pretrained model (git-lfs tracked under models/), not a re-implementation. The C++ provides the inference runtime: GGUF load + BERT forward pass + CLS pooling + L2 normalization, a WordPiece tokenizer that auto-detects the llama.cpp WPM vocab convention, and the ITQ fit/encode + int8 quantization.
dl-embed pipeline --db DIR embed corpus + emit vector relations + publish dl-embed encode --db DIR QRY print 'sig_hex ivec_hex' for a query (CLI consumes this) dl-embed embed QRY print the raw 384-float embedding dl-embed tokenize QRY print token ids + strings (debug) dl-embed self-test math/tokenizer checks (+ golden-embedding gate if model present) dl-embed dump-tensors [PATH] list GGUF tensors dl-embed fetch-model download the bge-small GGUF to models/
The dl vsearch/vhybrid commands fork/execve dl-embed encode to embed the query (no shell, no Python). The golden gate (./dl-embed self-test) embeds reference strings and asserts cosine ≥ 0.9999 against the reference model — the numeric proof that the C++ forward pass is correct.
The full design of record lives in the design docs: datalog-dafsa-vector-search.md (MIH over ITQ, the integration seam, honest ceiling) and the int8-in-store note. In short: in-store MIH is competitive with HNSW at ~1e5–1e6 entities and wins the consolidation story — one identity space, one snapshot, one WAL, one crash-recovery story.