In brief
- Every indexed passage needs a stable connection to its original source and version.
- Permissions must apply during retrieval, not only when the answer is displayed.
- Every source needs a defined maximum delay for access revocation that matches the sensitivity of its content.
- Monitoring must expose failed updates, stale content and unauthorised results.
Which metadata must be preserved?
Splitting a document creates several searchable passages. Each passage must still point to the original, title, validity and location. Without this lineage, an answer cannot be verified or updated precisely after a source changes.
- Stable source and passage identifiers
- Title, owner, version and validity date
- Original path or verifiable link to the cited location
- Language, document type and approval status
- Access and sensitivity metadata
How are document permissions enforced?
Retrieval must know the authenticated identity and restrict results to that person's rights. A shared service account must not make every indexed document visible. The strongest control happens before text is passed to the model.
Otherwise, an access window exists between revocation in the source system and its arrival in the index. Every source therefore needs a documented revocation SLA: the maximum acceptable delay for revoked access, matched to sensitivity. Event-driven updates can shorten this window. For particularly sensitive sources, the application should also recheck permission against the source system immediately before use. If permission state is unknown, too old or synchronisation has failed, the result is not passed to the model—the system denies access by default.
- Synchronise source permissions or groups with every passage
- Use the user identity from a verified sign-in context
- Filter results securely at search or data layer
- Reflect changes to groups and individual rights promptly
- Test with multiple roles and explicitly denied documents
How does the index stay current?
An index is a derived copy. Source text, metadata and permissions may change independently. Each event therefore needs a documented control path, safe failure behaviour and verifiable evidence.
| Event | Expected control | Safe failure behaviour | Evidence |
|---|---|---|---|
| Content or version changes | Replace affected passages, metadata and embeddings atomically | Mark the old version stale or block it rather than silently presenting both | Source version, index timestamp and passed sample test |
| Access right is revoked | Update rights independently of content within the revocation SLA | Do not pass a result to the model when permission state is unknown or too old | Source-system timestamp, applied timestamp and denied-role test |
| Source is deleted | Remove original reference, passages, vectors and caches completely | Disable the source immediately even if physical cleanup continues | Deletion confirmation and search for remaining derivatives |
| Synchronisation fails | Alert on the error, measure age of the last successful run and bound retries | Remove a sensitive or time-critical source from retrieval after its defined limit | Failure log, last successful version and documented recovery |
What should an answer make visible?
A citation helps only if it supports the statement and the person can open it. When sources conflict, the application should not hide the conflict. If there is no current, authorised evidence, the system should say so or direct the user to the responsible team.
- A direct location rather than only a broad document title
- Validity or version for time-dependent statements
- A clear distinction between supported facts and model wording
- A warning when evidence is missing, conflicting or inaccessible
Example: access change in project documentation
A person leaves a confidential project. Their group membership is removed in the source system. The company has set a maximum revocation SLA of 15 minutes for this source, and an event immediately triggers an index update. Until the update is confirmed, the application rechecks access against the source system. If that check is unavailable, it returns no project passages. Monitoring and a test request then confirm the revocation. Previous chat output is handled by separate retention rules and is not confused with the search index.
What to remember
Treat the search index as security-relevant derived data. Lineage, permissions, updates and deletion must remain traceable for every passage.
Sources and further reading
These primary sources provide further detail on definitions, technical foundations or responsible use.
Content reviewed