Most Companies’ AI Risk Comes From Tools They Bought. A Practical Plan to Manage It

Most of the public debate about AI regulation is about the companies that build frontier models. Most of the actual AI risk inside ordinary businesses comes from somewhere else: the transcription tool sales installed last quarter, the AI features a CRM vendor switched on in an update, the coding assistant engineers use with their own accounts, the chatbot a marketing agency put on the website. None of these companies trained a model. All of them are now responsible for what those models do with their data.

This is the gap privacy, legal and security teams keep running into. Frontier-model laws such as California’s SB 53 set duties for a handful of developers. Comprehensive AI statutes are still arriving unevenly. Meanwhile, the organizations buying and using AI tools have little specific guidance about how much diligence is enough. What they do have is a set of older disciplines, vendor management, acceptable use policies and security controls, that can be adapted quickly if teams stop waiting for an AI-specific rulebook.

Why third-party AI is a different kind of vendor risk

Buying an AI tool looks like buying any other SaaS product, and the standard security questionnaire will still be sent. The differences show up in places that questionnaire rarely covers.

• The data may become training material. A traditional processor stores and processes your data. An AI vendor may also use prompts, files or outputs to improve its models unless the contract or settings say otherwise.
• The product changes without a procurement event. Vendors swap underlying models, add features and change retention defaults through ordinary updates. The tool you assessed in January may not be the tool you are running in October.
• The supply chain is deeper than it looks. Many AI products are wrappers around a model provider’s API, with their own subprocessors for hosting, logging and evaluation. Your data may pass through three companies to answer one prompt.
• Outputs carry their own risk. Errors, fabricated content and biased recommendations become your problem the moment an employee relies on them, especially in hiring, lending, healthcare or customer communications.
• Adoption outpaces review. Employees can start using a browser-based AI tool in minutes, long before procurement or privacy knows it exists.

The NIST AI Risk Management Framework names this problem directly. Its core functions include a governance category requiring “policies and procedures” that “address AI risks and benefits arising from third-party software and data and other supply chain issues,” a mapping category covering risks of “all components of the AI system including third-party software and data,” and a management category requiring that third-party AI risks be “regularly monitored” and that pre-trained models be monitored as part of routine maintenance. The framework is voluntary, but it gives teams a shared vocabulary and a defensible structure. Our explainer on the NIST AI RMF for privacy, legal and compliance teams walks through the full framework.

Regulatory pressure is arriving through privacy law

For many U.S. businesses, the first binding AI obligations come from privacy regulation rather than AI statutes. In California, regulations approved in September 2025 added risk assessment and automated decisionmaking technology rules to the CCPA framework. According to the California Privacy Protection Agency, businesses subject to the risk-assessment requirements had to begin compliance on January 1, 2026, and must submit attestations and summaries to the agency by April 1, 2028. Businesses that use ADMT to make significant decisions must comply with those rules beginning January 1, 2027.

Those rules apply regardless of whether the business built the technology. A company that uses a vendor’s AI screening tool in hiring is the one making the decision, and the assessment and notice duties attach to it. That is the practical reason third-party AI governance cannot stay a procurement formality.

A pragmatic program in six steps

The goal is a program proportionate to the risk, not a review board that blocks every tool. These steps build on processes most organizations already run.

1. Find what is already in use. Start with expense reports, single sign-on logs, browser extension inventories, network traffic to known AI domains and a short survey of team leads. Include AI features embedded in existing vendors’ products, not just stand-alone tools. Expect the list to be longer than anyone thought.
2. Sort tools into risk tiers by use, not by vendor. The same model can be low risk for drafting internal emails and high risk for summarizing patient records. Tier on three factors: the sensitivity of the data going in, the consequence of a wrong output and whether the output affects a decision about a person. A simple three-tier scheme works for most companies.
3. Ask transparency questions that match the tier. For low-risk tools, confirm data retention, training use and the ability to delete. For higher tiers, ask which underlying models the product uses and how the vendor notifies customers of a model change, which subprocessors receive data, what testing the vendor has done for accuracy and bias in your use case, how outputs are logged and whether the vendor publishes system or model documentation. Where the vendor relies on a frontier developer covered by California’s SB 53, that developer’s published framework and transparency reports are worth reading too.
4. Put the answers in the contract. A questionnaire answer is a statement; a contract term is an obligation. Key terms include a prohibition on training with your data unless you opt in, retention and deletion timelines, notice of material model or subprocessor changes, security incident notification, audit or evidence rights and allocation of responsibility for output-related claims.
5. Write a use policy people can follow. Effective policies are short and concrete. They list approved tools for each tier, name the data categories that may never be entered into unapproved tools, require human review for outputs used in decisions about people or sent to customers, and explain how to request a new tool. A policy that bans everything will be ignored; one that offers an approved path will be used.
6. Back the policy with controls. Use enterprise accounts with single sign-on so access can be revoked. Turn off training and chat history settings where the vendor allows it. Apply data loss prevention rules to block sensitive data types from reaching unapproved AI domains. Log usage for higher-risk tools. Reassess on a schedule and whenever the vendor announces a model change.

Common failure points

Even well-designed programs break down in predictable ways. A few patterns are worth checking for before an auditor or regulator does.

• Approving the vendor, not the use. An approved tool gets used for a new purpose, such as moving from marketing copy to customer support transcripts, without a second review.
• Trusting a setting nobody verified. The admin console says training is off, but individual users signed up with personal accounts before the enterprise plan existed. A setting is only a control if it covers every account.
• Treating the questionnaire as the assessment. A completed questionnaire documents what the vendor said. It is not evidence that the controls work in your environment, which is why our guide to closing the AI audit gap focuses on testing.
• Forgetting the fourth party. The vendor’s model provider and hosting subprocessors handle your data too. Our piece on what third-party due diligence actually requires explains why programs that stop at the first tier fall short.
• Writing a policy about architecture instead of behavior. Employees need to know what they may do with a tool, not how transformers work. As we argued in generative AI governance starts with use, rules tied to tasks are easier to follow and to audit.

A worked example (hypothetical)

Consider a regional insurer whose claims team wants an AI tool that summarizes adjuster notes and drafts letters to policyholders. The notes contain health information, financial details and names. The drafted letters go to customers, and the summaries inform claim decisions.

Under the tiering above, this is the highest tier on every factor. The transparency review would ask which model the tool uses, whether claims data is used for training, where data is hosted and for how long, and what accuracy testing exists for insurance terminology. The contract would bar training on the insurer’s data, require deletion on termination and require notice before the underlying model changes. The use policy would require an adjuster to review every letter and every summary used in a decision. Controls would include single sign-on, logging and a quarterly sample review of outputs against source notes.

Compare that with the same insurer’s marketing team using a general chatbot to brainstorm campaign slogans with no customer data. That is a low-tier use: confirm the enterprise account has training disabled, add it to the approved list and move on. Proportionality is what makes the program sustainable.

Where Captain Compliance fits

Captain Compliance’s AI governance module is designed to inventory the AI an organization builds, buys and permits employees to use, run risk and impact assessments, review vendors and agents, assign controls and keep the evidence behind each approval. Linked to the platform’s data mapping, it can show which systems and data categories each AI tool touches and which vendors sit behind it. That gives teams one record for procurement, privacy and security decisions instead of three spreadsheets. It does not replace testing a tool in your own environment, and no platform can certify that a vendor’s model behaves as promised. For individual-level risks to watch, see our guide to generative AI privacy risks in personal use, and for terminology, the AI governance dictionary.

Frequently asked questions

Do we need a separate AI vendor questionnaire?

Usually an AI supplement to the existing questionnaire is enough. It should cover training use, model provenance and change notice, subprocessors, output testing and logging. Reserve the full supplement for higher-risk tiers.

Is the NIST AI RMF mandatory?

No. It is a voluntary framework. Many organizations use it because it is widely recognized and maps well to other standards and regulatory expectations.

Are we responsible for a vendor’s AI mistakes?

Often, yes, at least toward your customers and regulators. If a tool you chose produces a wrong decision about a person, the organization that made the decision is usually the one expected to answer for it. Contracts can allocate some cost back to the vendor, but they rarely remove your own obligations.

How often should AI vendors be reassessed?

On a fixed schedule tied to the risk tier, such as annually for low-risk tools and more often for high-risk ones, and whenever the vendor changes the underlying model, adds a subprocessor or changes data retention.

What about employees using free AI tools on their own?

Blocking alone tends to push use further out of sight. Pair technical controls on sensitive data with an approved alternative that is just as easy to use, and make the request process for new tools quick.

The tool you approved last year is not the tool you are running

The hardest thing about third-party AI is that it does not stay still. Models are replaced, features appear overnight and defaults shift in release notes no one reads. A program built on a one-time approval will drift out of date within months. The organizations that manage this well will not be the ones with the longest policy. They will be the ones that notice when the vendor changed something, and ask again.