RAG knowledge systems: design, evaluation and release

Retrieval-augmented generation (RAG) brings retrieved material into a model's answer. A dependable business knowledge system also needs source ownership, document permissions, retrieval evaluation, supported citations and a clear response when evidence is missing.

This guide covers product FAQs, policy queries, review analysis and internal training knowledge. Start with data pipelines, then continue to agent workflows.

1. Define the question, evidence and user

Begin with a bounded, authorised and testable question set. Translate efficiency goals into actual task time, review time, errors, rework, query cost and user success. Measure a baseline before comparing a pilot.

The original RAG paper combines a parametric model with retrieved non-parametric memory and evaluates specified knowledge-intensive tasks. A business system needs evaluation on its own documents and workflows.

2. Design ingestion, querying, permissions and versions

flowchart LR
  A[Authorised source documents] --> B[Parsing, source and permission metadata]
  B --> C[Chunking and index versions]
  C --> D[Retrieval and reranking]
  U[Authenticated user's question] --> D
  P[Server-enforced document scope] --> D
  D --> E[Evidence checks and answer generation]
  E --> F[Answer, sources, versions and next steps]

Assign stable document and passage IDs. Retain source location, content version, owner, effective date, language, permission scope and withdrawal status. A chunk must remain traceable to its original page, table, clause or passage.

The server authenticates the user and restricts retrieval to authorised documents. Apply the same permissions to citations, search snippets, caches, logs and downloads. Instructions inside a document cannot grant access.

Microsoft's document-level access guidance is one implementation reference. Check the chosen service version, preview status and identity integration before adopting a particular approach.

Updates and deletion

An edit becomes queryable only after parsing, indexing, version publication and cache updates have completed. Withdrawal must also address old passages, derived summaries and citation links. Preserve pending, failed and completed ingestion states rather than reporting unsuccessful work as finished.

3. Compare technical choices using evidence

Evaluate chunking and Top-K rather than assuming a universally best token length. Changing an embedding model or its dimensions requires an index rebuild and version-switching plan. Test the actual store's capacity, filters, durability and backups under the intended workload.

Lost in the Middle investigates information position in specified long-context tasks. It motivates testing evidence placement and distracting material; its experimental findings are not a fixed performance estimate for every current model.

4. Build an operable knowledge system

1. Inventory: establish ownership, permitted uses, access, provenance and publication scope. Separate unapproved or uncertain sources.
2. Parsing checks: inspect tables, scans and important figures. Keep failed documents visible for follow-up.
3. Indexing: create stable IDs, versions and permission metadata. Evaluate chunking, retrieval and reranking with real questions.
4. Query service: authenticate, enforce document scope, handle timeouts and empty results, and return traceable source locations.
5. Answer contract: define coverage, citation format and treatment of conflicting or insufficient evidence.
6. Evaluation: retain the same questions, source versions, model settings and scoring method for each comparison.
7. Controlled pilot: have authorised users complete real tasks. Record review effort, errors and rework.
8. Release and recovery: switch index versions after acceptance; monitor source updates, failed jobs and broken citations, with a rollback path.

5. Evaluate retrieval and answers separately

Retrieval metrics

Label relevant documents or passages for answerable questions. Handle unanswerable, unlabelled and permission-restricted questions separately. This function takes actual passage IDs and calculates per-question Precision@K, Recall@K and reciprocal rank truncated at K. It rejects duplicate IDs. Precision@K retains K as its denominator even when fewer results are returned.

function retrievalMetrics(retrievedIds, relevantIds, k) {
  const validIds = ids => Array.isArray(ids) && ids.every(id =>
    typeof id === "string" && id.length > 0 && id === id.trim());
  if (!validIds(retrievedIds) || !validIds(relevantIds)) {
    throw new Error("Explicit passage ID arrays required");
  }
  if (!Number.isSafeInteger(k) || k < 1) throw new Error("Positive integer k required");
  if (!relevantIds.length) throw new Error("Evaluate unlabelled or unanswerable questions separately");
  if (new Set(retrievedIds).size !== retrievedIds.length ||
      new Set(relevantIds).size !== relevantIds.length) {
    throw new Error("Duplicate passage IDs must be resolved before evaluation");
  }
  const relevant = new Set(relevantIds);
  const top = retrievedIds.slice(0, k);
  const hits = top.filter(id => relevant.has(id)).length;
  const first = top.findIndex(id => relevant.has(id));
  return {
    k, returned: top.length, relevant: relevant.size, hits,
    precisionAtK: hits / k,
    recallAtK: hits / relevant.size,
    reciprocalRankAtK: first < 0 ? 0 : 1 / (first + 1)
  };
}

These values measure retrieval hits. They do not establish answer faithfulness, support for each claim or business impact. When aggregating results, state the number of questions, annotation method, averaging approach and unscorable items.

Answers and user tasks

The Ragas paper distinguishes retrieval and generation evaluation dimensions. Automated scoring can assist triage and regression comparisons; business-critical answers still need checks against original evidence and domain judgement.

6. Handle missing, conflicting and hostile evidence

• No results: retain the question and authorised search scope; request a model, date or document, or refer the task to its owner.
• Conflicting versions: show applicable versions and scope. Follow approved precedence rules or obtain the document owner's decision.
• Unsupported citation: prevent the answer from being treated as verified; retrieve again or seek review.
• Changed permissions: enforce the new access policy and address old caches and download links.
• Embedded requests to disclose data or take action: treat these as source content, not permission to escalate access, transmit sensitive information or invoke tools.

OWASP's prompt injection guidance explains that RAG does not fully eliminate prompt injection. Enforce permissions, tool restrictions and data-handling rules in the application.

7. Release, maintain and accept the system

Protect authentication, document access, credentials, uploads and resource use. Inspect the actual data flows of embedding, generation, telemetry and other connectors. Running one model locally does not establish that every component keeps data on the device.

Exercise backup, recovery, reindexing, rollback and deletion. Retain necessary operational logs while preventing private documents and customer queries from spreading into unprotected monitoring services.

Acceptance should cover normal answers, missing evidence, conflicting versions, broken sources, timeouts, unauthorised access, tenant isolation, withdrawal, model/index changes and process restarts. Report the actual questions, data versions, outcomes, failures and improvement actions.

Further reading

• Customer service workflows
• RAG fundamentals
• Agent workflows
• SEO and GEO