In brief
- Business teams own content and validity; technical teams own processing and availability.
- Every source needs a defined update, failure and deletion process.
- Retrieval and answer quality are monitored with stable test questions and real feedback.
- Changes to models, embeddings, chunking or sources must be versioned and tested before release.
Which roles does operation require?
Technology alone cannot decide which statement is authoritative. Domain owners approve sources and resolve conflicts; platform and operations teams secure data flow, permissions and availability. A named product owner connects value, quality and change.
| Operational task | Lead responsibility | Cadence or trigger | Evidence |
|---|---|---|---|
| Approve source and validity | Domain owner | Before onboarding and after content change or expiry | Approval status, version, scope and accountable person |
| Operate connector, index and deletions | Platform or operations team | Continuously; verify after every synchronisation run | Count reconciliation, delay, failures and deletion confirmation |
| Test permissions | Engineering with security | After role changes, at release and regularly as regression tests | Positive and explicitly denied cases for each role |
| Assess retrieval and answer quality | Product owner with the domain team | Before release and on a fixed monthly or quarterly cycle | Versioned test set, results and accepted error limits |
| Handle false or missing answers | Support route with the domain owner | After a report or recurring failure pattern | Classified case, source correction and new regression test |
| Handle a security or privacy incident | Incident lead with security and privacy | Immediately after detection | Impact, containment, cause, notification and retest |
What does the ongoing data cycle look like?
Each source passes through detection, processing, indexing and quality checks. Operations monitor not only technical availability but also delay, expired content and complete removal of deleted documents.
- Step 1
Inventory
Document the source, owner, purpose, audience and sensitivity.
- Step 2
Synchronise
Carry text, metadata, permissions and deletions into the index.
- Step 3
Validate
Run samples, count reconciliation and known test questions.
- Step 4
Monitor
Watch failures, delay, no-result queries and quality indicators.
- Step 5
Improve
Feed user feedback and new failure patterns back into sources and tests.
Which quality should be measured?
A good model answer can be based on the wrong result; a poor answer can occur despite the right source. Retrieval and generation should therefore be measured separately and together. Averages must not hide critical failure types.
- Recall: is the required passage among the returned results?
- Precision: how many returned passages are actually relevant?
- Authorisation: is unauthorised content always excluded?
- Grounding: does the source support the formulated statement?
- Operations: freshness, latency, cost and failure rate
How are changes and incidents handled?
New models, embedding versions or chunking rules can change results even when source documents stay the same. Such changes need comparison on a fixed test set and a rollback path. Unauthorised or dangerously false answers require a prepared incident process.
- Version configuration and data state
- Test changes in a separate environment first
- Use critical regressions as release blockers
- Be able to disable affected sources or functions quickly
- Document cause, impact, correction and retesting
Example: monthly operation of a service knowledge base
The service team owns manuals and approvals. A daily job carries changes and reports failures. Weekly review covers frequent no-result queries and negative feedback; a fixed test set runs monthly across products, languages and roles. Before a model or embedding change, the team compares old and new and releases only without critical regressions.
What to remember
Plan domain curation, technical synchronisation and measurable quality before the first production use. A knowledge base remains only as good as its operating process.
Sources and further reading
These primary sources provide further detail on definitions, technical foundations or responsible use.
Content reviewed