Out of Scope Productions is a Third Person Consulting company. Visit TPC →
Out of ScopeProductions
← Quick Start BriefsTechnology · AI · Knowledge · Legal

Before You Switch On the AI Assistant

An internal AI assistant is only as good as the knowledge it reads, the permissions it respects and the tests you ran before anyone trusted it. This brief is a pre-launch checklist for firms putting an assistant over their standards, procedures, project files or asset records, written for engineering, construction and manufacturing teams.

1. Decide what it's for, and who owns it

Start with a task, not a tool. Decide whether AI should do the task at all, then tier oversight by risk: drafting a meeting summary and answering "what's the cover to rebar on this element?" are different jobs and deserve different controls. PRM-BIZ-05 · PRM-GOV-02

  • Name a business owner for every kind of answer the assistant gives, and someone who can pause it today. AIG-01 · AIG-14
  • Pair the investment with a learning plan. Technology without the organizational habits to use it changes little. KIN-TWN-02
  • Measure whether controls operate, not whether a policy exists. AIG-16

2. Feed it governed knowledge

An assistant over an ungoverned shared drive will retrieve the superseded standard as confidently as the current one. Retrieving a withdrawn standard faster is a risk amplifier, not a service. KIN-GOV-04

  • Classify knowledge by authority: approved standards and procedures, live project guidance, learning material, lessons and Q&A, drafts, and archive. Make search and AI show each result's class and source, not a flat list. KIN-GOV-02
  • Index only eligible sources for operational answers, with one owned, current version of each. KIN-TWN-01 · PRM-CTX-03 · PRM-GOV-01
  • Run a content lifecycle: owner, status, version and review date on every object, and when something is superseded, the change reaches the AI index within a set time. KIN-GOV-04 · SOP-DIG-01
  • Fund the curators. Knowledge roles and protected curation time are what keep the index honest. KIN-GOV-05 · KIN-GOV-01

3. Test it on your own questions

Grounding an assistant in your documents reduces invented answers; it doesn't remove them. In the first preregistered evaluation, commercial legal research tools grounded in curated sources and marketed as hallucination-free were wrong 17 to 33% of the time, and incomplete far more often. KIN-TWN-01 · KIN-GOV-02

Figure 1. A gold-set test, before launch and after every major change
Build50 to 200 real questions per domainTaken from the help desk, RFIs and the questions seniors get asked.
BuildSome that should be unanswerableThe right answer is "not found", not an improvisation.
ScoreCorrectnessIs the answer right for this project, edition and jurisdiction?
ScoreCitation accuracyDoes the cited source, version and clause actually say it?
ScoreCompletenessDid it miss the exception that matters?
ScoreCorrect abstentionDid it say "not found" when it should?

Re-run the set periodically, and after any content or model change. Retrieval systems fail at the joints: missing content, the right document ranked too low, an answer lost while combining sources. KIN-TWN-04 · PRM-EVD-04 · AIG-09

Configure the assistant to give source, version and owner on every answer, and to say "not found" rather than improvise. Then make people open the citations: unread means uncited. KIN-TWN-01 · AIG-09

4. Permissions and leakage

An index that ignores access controls turns every user into an administrator. Enforce source permissions at retrieval time, and before release, and after any index or permission change, red-team for answers that carry one project's or client's information into another's. No peer-reviewed leakage rates exist yet; this is implementation judgment, and it's cheap to test. KIN-TWN-05

  • Treat retrieved content as data, never as instructions. A document in the index can carry text that tries to steer the assistant. AGT-04
  • Block silent exit channels in AI output, such as auto-loading images or links that carry data out. AGT-06
  • If the assistant can act, not just answer, give it its own scoped identity, and make "don't touch production" a permission rather than a prompt. AGT-01 · AGT-02

5. Clients, contracts and people data

Client confidentiality. Tag client-restricted content at source, honour contractual reuse limits, sanitize lessons before they're published across the firm, and check client contracts for AI clauses before the assistant reads a single file. Disclose AI use to clients in the contract, and ask your broker in writing about AI exclusions. KIN-TWN-06 · PRM-GOV-04 · AIG-10 · PRM-GOV-05

Search logs and expertise maps are people data. An assistant's logs show who asked what, and how often. In Germany, systems capable of monitoring behaviour or performance need works-council agreement; in France the works council must be consulted on new technologies. Under the EU AI Act, AI that allocates tasks or evaluates people's performance is high-risk, and employers must tell workers first. Quebec's Law 25 requires notice before profiling, including of work performance. Aggregate, minimize, set retention, and give people a way to contest. Canadian federal, UK and US law weren't part of this review: check them for your sites. KIN-GOV-06 · KIN-SIG-01

Keep prompts, sources and outputs as records where the work needs them, and read search and question signals as hypotheses about the organization, not verdicts about individuals. PRM-GOV-03 · KIN-SIG-01

6. Keep the judgment with people

The engineer stays responsible; AI never seals. That's easy to write and harder to keep when the assistant is right most of the time. PRM-ENG-01 · DQ-AI-03

  • Give assistance to less experienced staff first, with a named person responsible for every answer that reaches a client or a site. GOL-AIA-01
  • Check proficiency with the AI switched off. AI that does the task for trainees raises output and can lower skill. PRM-LRN-01
  • Train people to find planted errors, and assess verification behaviour, not prompt-writing. PRM-LRN-02 · PRM-LRN-05
  • Measure the speed-up; don't ask about it. In one software trial, experienced developers measured slower with AI while believing they were faster. PRM-EVD-08

The checklist. A named owner and a pause button. Only governed sources in the index, with authority shown. A gold set with unanswerable questions, re-run after every change. Source, version and owner on every answer, and "not found" when it doesn't know. Permissions enforced at retrieval and tested for leakage. Client contracts checked. Logs handled as people data. Skills checked with the AI off.

Practise it. In The Index, Halverson Lisle Engineering has one day before its assistant launches, in Project: Countdown!