Compliance: How to Create Controls, Run Tests, and Track Evidence

Table of Contents

• Summary
• Getting Started
• Creating a Control
• Creating a Control Test
• Running a Test
• Viewing Test History
• Managing Stored Credentials
• FAQ

Summary

Compliance gives your security and compliance team one place to define the controls you enforce, test that they're working, and house the evidence that proves it — all with a complete, timestamped history over time.

Rather than simply documenting that controls exist, you can prove it. Add credentials so Whistic's AI can navigate secured platforms and capture evidence on a set schedule, automatically storing results over time. Run tests manually or let the agent do the work — every result is logged with a full history your auditors can review at any time. Whether you're preparing for a review or maintaining ongoing compliance, Compliance keeps your program organized and audit-ready.

Press Start Demo below to begin watching this short demo of our newest product! 

🚀 Getting Started

Who can use this feature:

• Users on plans with the Compliance feature enabled
• Admin-level access is required to create and manage controls & tests

If You Don't Have Access Yet

If you navigate to the Compliance page and see an overview screen instead of your Compliance dashboard, this means the feature has not yet been enabled for your organization. From this page, you can:

• Watch a short demo to learn more about the feature
• Start Free Trial or Contact sales to discuss adding Compliance to your plan

If you are interested and have the option to Start Free Trial, go here to learn more. 

🛡️ Creating a Control

A control is a security policy or requirement your team enforces — for example, "All production databases must be encrypted at rest." Controls are the foundation of this feature; tests are then attached to each control to verify it's actually working.

1. Navigate to the Compliance tab in your Whistic account

2. Select Controls from the menu

   

3. Click Create Control

   
4. Enter a Title and Summary for the control

   • Example — Title: Ethics Policy | Summary: All employees must acknowledge the company ethics policy annually

     
5. Click Create Control — the control will now appear in your Controls library with a status, due date and more
6. From the Controls library, you can sort controls by due date to prioritize what needs attention next

🧪 Creating a Control Test

Once a control is created, you will automatically be presented with the Test page. A control test defines how a specific control will be verified. Each test includes step-by-step instructions, pass/fail criteria, and a recurrence schedule so it runs automatically on a set cadence.

There are two test types available:

Test Type	How It Works	Best For
Manual	You follow documented steps and upload evidence (screenshot, PDF, etc.)	Controls that require human review or cannot be automated
Browser AI	An AI agent navigates to a URL, follows your instructions, and captures a screenshot and other details requested as evidence	Checking web-accessible sources like shared drives or portals

To create a test:

1. Open a control from your Controls library
   1. On this page you will see your control summary, and the following two sections: Test and Test History
2. Click Add Test
3. Fill in the test details
   • Title - Provide a title to identify it by
   • Summary - High level summary of the test you will be conducting

   • Select a test type: Manual or Browser AI

     
   • For Browser AI test you will need to provide:
     • Set a recurrence cadence (e.g., every 12 months) so the test is scheduled automatically and due dates stay current
     • Instructions — Step-by-step guidance for what to check
       • Example: Go to our Google Drive, find the employee handbook, locate the ethics policy section, and take a screenshot
     • Criteria — What counts as a passing result
       • Example: The ethics policy must be visible in the screenshot

     • If using a Browser AI test that requires a login, attach stored credentials (see Managing Stored Credentials)

       
4. Click Create & Run

▶️ Running a Test

1. Depending on the test type:
   • Manual — Follow the instructions, then upload your evidence (screenshot, PDF, or other supported file)
   • Browser AI — The agent runs automatically. A status indicator tracks the progress: In Progress → Evidence Gathered → Pass or Fail

(Browser AI status icons)

(Manual & Browser AI - pass/fail icons)

2. Review the result — the test results view displays:
   • Control name and test type
   • Instructions and pass/fail criteria
   • Date and time of the run
   • AI-generated notes explaining the result
   • All evidence, including screenshots
     • For Browser AI test the executor will be listed as [e-mail]

3. Mark the test as Pass or Fail (Browser AI will do this for you)

   
4. Optionally, click the pencil icon to add or edit notes on the results panel

❌ Reviewing a Failed Test

If a test is marked as Fail — either by the Browser AI or manually — Whistic will flag the associated control as non-compliant. This is your signal to investigate and determine next steps.

To review a failed test from the Compliance drop-down:

1. Open the control from your Controls library or by going to Test History
2. Navigate to the Test History section and select the failed test run from the rows below (clicking the pencil icon from the Tests section will open the Test Details)
3. Review the test results, which include:
   • The instructions and pass/fail criteria that were used
   • AI-generated notes explaining why the test did not pass (if Browser AI test)
   • Any screenshots or evidence captured during the run
4. Determine the cause of the failure — common reasons include:
   • Incorrect instructions or criteria — the test was set up in a way that didn't accurately reflect what should be checked; update the test and re-run
   • Page blocked or inaccessible — the Browser AI was unable to reach the target URL due to access restrictions, a site change, or downtime; verify the URL is correct and accessible, then re-run
   • Legitimate non-compliance — the control is genuinely not being met and action is required before the test can pass
5. Once you've identified the cause, take the appropriate action:
   • Edit the test instructions or criteria if the setup was incorrect
   • Resolve the underlying issue if the control is out of compliance
6. Optionally, click the pencil icon to add notes documenting your findings before re-running

Note: The control will remain flagged as non-compliant until a passing test result is recorded.

Here is a manual test failure as an example:

📋 Viewing Test History

Every test run creates a permanent, timestamped record that includes the control name, test type, instructions, criteria, date and time of the run, AI-generated notes, and all evidence. You can view the full history for any control at any time, making it easy to demonstrate continuous compliance to auditors.

To export your controls and test history, use the Actions dropdown in the middle of the page from the Test History page and select Export Results.

🔐 Managing Stored Credentials

If a Browser AI test needs to log into a site to gather evidence, you can store credentials securely in Whistic.

When adding credentials, you'll provide:

• Name — A label for the credential set (e.g., Whistic Staging Login)
• Domain — The domain where these credentials will be used (e.g., whistic.io) — credentials are restricted to that domain and its subdomains only
• Username and Password — Stored encrypted; raw values are never exposed in test definitions, logs, or exports

Note: Credentials are encrypted at rest and scoped to a specific domain to prevent misuse. Browser Agent login supports username and password only — sites that require multi-factor authentication (MFA) cannot be automated at this time. Use a Manual test for those cases instead.

❓ FAQ

What is a control?

A control is a security policy or requirement your team enforces — for example, "All production databases must be encrypted at rest." In Whistic, you create a control with a title and summary, assign an owner, and attach tests to verify it's working.

What is a control test?

A test defines how to verify a specific control. You provide instructions, pass/fail criteria, and a test type. Each test run creates a permanent, timestamped record with evidence.

What's the difference between a Manual test and a Browser Agent test?

Manual tests require you to follow the steps and upload evidence yourself. Browser Agent tests use AI to navigate to a URL, follow your instructions, and capture a screenshot automatically. In both cases, you review the result and mark it as pass or fail.

Can I schedule tests to run automatically?

Yes — set a recurrence cadence (e.g., every 12 months) when creating the test. Whistic tracks the next due date, and you can sort controls by due date on the Controls page to stay on top of what's coming up.

Can I edit the notes on a test result?

Yes — click the pencil icon on any test result to override or add to the AI-generated notes with your own context.

Can I export my controls and test history?

Yes — use the Actions dropdown on the Controls page to export your controls and test history.

How are stored credentials secured?

Credentials are encrypted at rest and scoped to a specific domain and its subdomains. Raw values are never visible in logs, test definitions, or exports.

What should I do if the Browser Agent can't find what I'm looking for?

The Browser Agent is best-effort. If the result doesn't meet your criteria, mark the test as fail, add a note, and review your instructions or URL for accuracy before re-running.

What happens if a target website changes or goes down during a scheduled test?

The test will run and log a failure. Use that failure as a signal to open the test, update the URL or instructions, and re-run.

What compliance frameworks can I use this for?

Controls & Control Tests is framework-agnostic — you can use it to track controls relevant to SOC 2, ISO 27001, HIPAA, or any other framework your team works within.

If I archive a control what happens to the test and history?

If controls are deleted, tests and history associated with them will be deleted.

What happens when a test fails?

When a test is marked as Fail, the associated control is flagged as non-compliant. You can open the test result to review the AI-generated notes, evidence, and the instructions and criteria used to determine the cause. Common reasons include incorrect test setup, an inaccessible or blocked page, or a genuine compliance gap. Once the issue is resolved, you can update the test and re-run it. The control will remain non-compliant until a passing result is recorded.

What should I do if the Browser AI marks a test as failed but I believe the setup was incorrect?

Open the failed test result and review the instructions, criteria, and AI-generated notes. If the test wasn't configured correctly, edit the instructions or criteria to better reflect what should be verified, then re-run the test. If the page was blocked or inaccessible, confirm the URL is reachable and try again. You can also add notes using the pencil icon to document your review before re-running.