Why Vector Search Cannot Be the Decision Layer in Policy Systems

Decision summary

Vector search works as a retrieval mechanism, but was rejected as the source of truth for policy decisions because deterministic enforcement requires explicit structure and traceable execution.

Invariant: The same input must always produce the same policy outcome.

DecisionGraph NormaGraph

Decision ID
DEC-002
Status
Active
Date
Sep 2024

Primary implementation

NormaGraphPolicy Intelligence System

Where this is applied

This shows up in systems like DecisionGraph and NormaGraph, where correctness depends on traceable outcomes.

Problem context

This decision arose while designing a policy intelligence layer responsible for generating and interpreting regulatory rules via LLM workflows.

The system needed to support auditable decisions, repeatable outputs, and clear explanation paths for why a specific rule was injected into a generated policy. Latency was secondary. Correctness and determinism were strictly non-negotiable.

The tempting option

Vector search was an obvious retrieval layer. The project already included RAG-style workflows, and vector infrastructure offers fast semantic lookup, strong ecosystem support, natural language flexibility, and easy integration.

For many AI agents, retrieving the "top 5 most similar policies" is acceptable — even desirable. The temptation was to let that retrieval step act as the policy selection mechanism itself.

Why it failed

Vector search is useful for retrieval, but it fails when promoted into the decision layer for policy logic.

1. Non-determinismThe same query can return different results across runs or model versions.

2. Unverifiable reasoning pathsSimilarity scores do not constitute explainable logic. You cannot reliably justify why one policy matched over another to an auditor.

3. Audit failureIn regulated or governance-heavy systems, "most similar" is not a defensible answer.

Vector search helps find candidate clauses. It cannot decide outcomes.

The chosen approach

I split retrieval from decision-making.

Vector search can still surface candidate clauses or related policy fragments, but final resolution happens in a structured reasoning layer implemented via LangGraph and explicit decision graphs.

Instead of relying on similarity mapping as the source of truth, I built explicit reasoning graphs where policies act as nodes with hard-coded relationships. The orchestration layer separates retrieval, reasoning, and execution, then traverses those nodes conditionally to produce the final outcome.

This produced deterministic execution paths, explicit rule relationships, explainable reasoning chains, and perfectly stable outputs. The cost was higher schema design overhead and reduced linguistic flexibility, but correctness was guaranteed.

Failure Modes

This approach is not perfect. Known weaknesses include:

- Cyclical or conflicting rules require explicit resolution strategies inside the graph. - Graph complexity grows non-linearly with policy scope. - Authoring new graph nodes takes significantly more engineering time than simply uploading a document to a vector index.

When to Revisit

If the system's role shifted from enforcement to advisory drafting guidance, vector-based retrieval could carry more of the system's behavior.

In that scenario, probabilistic suggestions are acceptable, explainability can be approximate, and determinism is less critical. Until then, vector search remains a retrieval primitive, not a decision authority.