The steering committee reviews nine slides. Slide four has a vendor logo and a green risk rating. The agenda item says to approve the enterprise AI assistant. The thing they are actually approving is a JSON configuration file and a service principal nobody in the room has access to.
When the committee asks what the assistant can do, the project sponsor describes a use case: summarizing policy documents and drafting support ticket responses. The committee approves the use case. They do not approve the data flow.
The assistant does not have the querying user’s permissions. It has the permissions of the identity it runs as. In most enterprise deployments, that identity is a service account created to build the retrieval-augmented generation index. That service account was granted broad read access to SharePoint, Confluence, and the shared drives to ensure the vector database had enough context.
When a regional sales director asks the assistant to summarize the compensation structure for a new role, the assistant does not check the director’s file permissions. It checks its own. It retrieves the document from an unsecured human resources staging folder from three years ago. The assistant synthesizes the exact salary and equity grant into a bulleted list. The committee approved a summarization tool. The incident report will describe a permission boundary failure.
The pilot ran for six weeks with twelve users in the IT department. Their permissions were the narrowest in the company. That is why the pilot worked. When the tool rolls out to forty thousand employees, it encounters permissions accumulated over fifteen years of reorganizations, acquisitions, and exceptions. Users have read access to thousands of sites they do not know exist. The AI does not create these access violations. It makes existing access violations instantly exploitable at machine speed. A human reads a few hundred documents a day. The assistant reads everything the service account can reach in an hour.
Reading data is a compliance issue. Writing data is an operational hazard. The governance framework treats both as the same agenda item. A vendor pitches a customer service routing tool. The stated use case is drafting response templates. The committee approves it. During implementation, the vendor instructs the engineering team to grant the tool write access to the ticketing system so it can automatically close tickets it thinks are resolved. The interface is still just a chat window. The backend is now an autonomous actor.
The first time the assistant acts on its own—canceling a calendar meeting, updating a database record, sending an email to a client—is the first time the organization finds out what write scope actually means. A human reviewer cannot catch a record change they never see. Accuracy is a quality metric. Rollback is a capability. If the assistant makes a wrong write action at machine speed, the blast radius is determined by the OAuth scopes, not the prompt instructions.
Committees approve software based on the features it has on the date of the meeting. AI vendors ship capability updates continuously. The governance calendar does not match the release notes. An enterprise approves a text-generation tool in March. By August, the vendor pushes an update that gives the tool a code interpreter, web browsing, and integration with local file systems. The committee’s approval of the product name remains intact. The risk profile has completely changed.
Every new connector is a new data flow wearing a checkbox. Adding a connector to a new shared drive or a third-party CRM is a configuration change in the admin portal. It enables a new data flow without triggering a new review.
Data loss prevention policies guard the door. The assistant lives inside. DLP scans for files leaving the network boundary. It does not see a user pasting a proprietary database schema or an unredacted contract into a chat window, because a prompt is not a file. The data never left the tenant. The meaning did.
Furthermore, the assistant reads in place and returns derived content. Summarization is a new data flow that produces a new artifact. Forty documents that are individually unclassified become sensitive when summarized together. No existing data classification scheme catches the derived artifact.
When the incident happens, the first question in the review is what the model read to generate the wrong answer. The logs show the prompt and the completion. The retrieval set is not logged. The investigation stalls. To govern the deployment, you need to reconstruct five things: the user identity, the connector credential, the documents retrieved, the text sent for inference, and the actions executed. Most deployments log two. If you cannot reconstruct the retrieval set, you did not govern the system. You only approved it.
The minimum viable governance artifact is not a signed memo listing the vendor and the business sponsor. It is the scope consent screen, the connector list, the identity the assistant runs as, and the log of what it did. Every user who clicks “Allow” on the consent screen is making a governance decision the committee never sees.
The approval packet describes what the tool is supposed to do, but the JSON configuration file determines what it can actually reach.