Table of Contents
Summary
Have you ever wondered how Whistic ensures the accuracy of its answers, whether you can inspect AI-generated responses, and how you can control or adjust the results Whistic AI returns?
This article will cover all the tools available for verifying accuracy, auditing, and editing answers.
🔍 Understanding how Whistic AI ensures the accuracy of its results
Firstly, let's understand how Whistic AI ensures the accuracy of its results.
Q: How does Whistic ensure the accuracy of its results?
Whistic continually improves and refines how we process and embed documents, upgrades to leverage newer models, and develops new tools to enhance our AI capabilities. With each update, we validate results against an extensive and growing suite of representative scenarios to ensure consistently high quality.
Q: How does Whistic consider match thresholds?
Whistic regularly adjusts match thresholds based on performance monitoring to maintain high accuracy and deliver the best possible results.
Q: How well is AI continuing to learn and provide answers for unknown questions over time?
Whistic AI does not learn in the sense of most AI LLMs. It improves when it gets access to the right data, as additional sources are added, or when incorrect or imprecise answers are edited and saved back into the Knowledge Base to act as additional sources for future questions. It also improves as we adjust our prompt engineering to increase the accuracy and specificity of the responses.
Q: Is Whistic's AI model optimized through fine-tuning, adapter tuning, etc.?
Whistic relies on state-of-the-art models developed by industry leaders and hosted securely on AWS Bedrock, where we ensure that our customer's data is secure and not used for unauthorized training. Our primary method for improving AI responses involves upgrading our models as the industry improves and improving our prompts to increase the accuracy and specificity of responses. In the future, our approach could involve fine-tuning or the like, but if that happens, it will continue to be limited by the bounds of our commitment to customer privacy and security.
🛠️ Tools allowing you to inspect and audit Whistic AI accuracy
Acting as a Whistic AI in the assessment process, and as part of each of these functions, Whistic AI provides additional tools that allow the user to inspect and audit the accuracy of the answers it provides, and to edit those answers as needed. These include:
1. An AI confidence level and explanation/Info Tooltip that indicates whether the sources provided sufficient information to fully answer the question and whether any other factors (such as contradictory sources, whether the question was ambiguous, etc.) should lower the user's confidence in that answer.
Note that the explanation in the Info Tooltip is specific to each question and will adjust on a question-by-question basis to provide the reasoning behind the AI's confidence.
In general, the AI Confidence rating reflects the following:
- High Confidence: the question is clear and was able to be fully answered using the existing sources, without requiring any inference /reasoning on the part of the AI.
- Medium Confidence: The question was able to be at least partially answered based on the sources, but it may have required some inference on the part of AI.
- Low Confidence: The question was ambiguous or the sources provided contradictory answers. Significant inference was required on the part of AI to generate the answer.
2. A detailed Answer Explanation of how AI arrived at the answer.
3. A prioritized list of the most relevant sources related to that answer, including an indicator of which sources were used to generate the answer.
Sources are prioritized based on how closely the information they provide matches the question.
The specific source(s) that were Used by AI to generate the answer are clearly identified at the top of the Sources list.
4. Each source provides a description of where the information was found in the document, as well as a link to easily navigate into the source document to view the relevant information in context.
5. Smart Search History provides an editable history of the questions asked, the answers provided, and the sources used to answer that question. This allows admins to correct ambiguous or incorrect answers (for the future) and to help improve AI answers for their organization over time. Learn more about Smart Search History