AI and automation
Can an internal artificial intelligence knowledge assistant be trusted?
A practical design for permissions, source citations, stale information, evaluation and safe rollout.
An internal assistant is trustworthy only within a defined task and measured error boundary. Retrieval with citations can improve traceability, but it does not prove that a source is current, authorised or interpreted correctly. Control source ownership, permissions, indexing, answer behaviour, evaluation and escalation before broad access.
The decision
What changes the answer
- The source repository needs owners and review dates before an assistant can make it reliable.
- Access at retrieval time should respect the user's authorised information boundary.
- Measure unsupported claims, wrong-source use, missed retrieval and unsafe disclosure separately.
Where the decision commonly goes wrong
- Indexing every shared drive before permission and quality review
- Treating a citation as proof that the answer follows the source
- Using user feedback alone without a controlled evaluation set
A sensible way to decide
- 01
Choose a bounded question set
Start with a domain where authoritative internal sources and a human escalation path exist.
- 02
Prepare source governance
Assign owners, classify information, remove duplicates and mark superseded or expired material.
- 03
Design access and answer rules
Respect source permissions, require citations, show uncertainty and refuse outside scope.
- 04
Build an evaluation set
Include correct, ambiguous, outdated, unauthorised and unanswerable questions.
- 05
Pilot and monitor
Review answer quality, source coverage, user behaviour, incidents and content decay before expansion.
Check before committing
- Scope bounded
- Source owners named
- Permissions enforced
- Citations visible
- Evaluation set passed
- Wrong-source failures measured
- Refusal behaviour tested
- Stale-content review scheduled
Material checked for this note
- Australian Government — Voluntary AI Safety Standard
- Office of the Australian Information Commissioner — Small business and the Privacy Act
Links and technical details checked 2 September 2026. Corrections can be sent to info@turnstoneai.com.