Five questions to ask before using AI with internal company knowledge
A practical guide for IT, operations and leadership teams.
Before choosing an AI interface, decide what a dependable answer needs to look like in your organisation. These five questions make a useful starting point for a pilot and a conversation with potential providers.
1. What information should it use?
Define an initial knowledge set and identify who owns it. Separate approved material from drafts, duplicates and personal notes. Ask whether the tool can be limited to the sources you choose, and how it behaves when those sources do not contain an answer.
For a pilot, choose a bounded workflow with a clear owner. A focused set of questions is easier to evaluate than an instruction to answer anything about the company.
2. Who should be able to see what?
Map access to the person asking, not just the account that connected the repository. Ask how permissions are applied to retrieved content, generated answers and citations.
Test with two users who have different access rights. Include a permission change during the pilot and check what happens to subsequent answers. Use synthetic or explicitly approved test material.
3. Can people check the answer?
Look for citations that open the relevant source, not just a generic reference list. Check whether the cited passage supports the actual statement and whether important qualifications are preserved.
Include questions with missing or conflicting evidence. A useful system needs a clear way to express uncertainty and direct people to further review.
4. What happens when knowledge changes?
Ask about refresh intervals, version handling, deleted content and access revocation. Make someone responsible for source quality, and test an update rather than relying only on a demonstration.
Agree how employees report a wrong answer and who investigates whether the issue came from retrieval, the source itself or the generated response.
5. How will we know it is helping?
Build an evaluation set from representative questions and have the relevant owners review the answers. Consider correctness, source quality, permission handling and the effort needed to complete the task.
Decide the boundaries of the pilot before launch. Which decisions still require review? What would make you pause the rollout? A clear definition of success is more useful than a large volume of questions alone.
The takeaway
Start with an owned knowledge set, realistic test questions and explicit review criteria. Then expand based on evidence.