Operating guide · AI and automation
Building an internal knowledge assistant with permissions, citations and evaluation
A controlled knowledge assistant for policies, procedures and service information.
Gaps to close
- Improve retrieval without broadening access
- Make answers traceable to current sources
- Identify unanswerable questions
- Avoid creating another unmanaged content copy
- Give source owners a practical review workload
Decisions to settle
- Authoritative source set
- Permission enforcement method
- Citation and refusal rules
- Evaluation and update thresholds
How to know it is working
- Unauthorised sources cannot be retrieved in tests
- Answers cite approved sources
- Unanswerable questions are handled safely
- Source owners and review dates are active
Working documents
- Source and access register
- Assistant behaviour specification
- Evaluation set and results
- Content and service operating runbook
Putting the routine in place
- 01
Knowledge governance
Choose a bounded domain, name source owners, classify access and remove or mark duplicates and superseded material.
- 02
Retrieval and access design
Index approved content with source metadata and enforce the user's information boundary at retrieval time.
- 03
Evaluation
Test correct, ambiguous, stale, unauthorised and unanswerable questions and score retrieval and answer failures separately.
- 04
Pilot operation
Release to a limited group, monitor use, feedback, incidents, source decay and refusal behaviour.
- 05
Content operations
Give source owners a review queue and define what happens when a source expires, moves or conflicts with another approved source.