The Pilot Costs Nothing. Production Costs a Lot.

The pilot invoice was zero. Finance signed off. Then the credits expired. The first production bill for an AI feature is often the first time a finance team sees a variable cost line item that isn’t raw cloud compute.

During the proof of concept, the architecture is designed to get the best possible answer. In production, the architecture has to get an acceptable answer at a sustainable margin. Most organizations skip the redesign. The pilot budget covered twenty users and a controlled, clean document set. The production forecast used the exact same cost per request after access went to the entire service desk and the application began retrieving from the full, messy knowledge base.

A user types a five-word query. The application retrieves ten pages of vector search results, appends a fifteen-hundred-word system prompt, and includes the entire previous conversation history. The user thinks they are sending five words. The billing meter charges for the entire payload. LLMs do not remember the previous turn of a conversation. To maintain context, the application must pass the entire history back to the API with every new message. A ten-turn conversation grows exponentially in token cost with each turn, even if the user is only typing a correction.

The invoice charges for attempts, not just answers. Traditional enterprise software fails fast. A malformed database query throws an error immediately. If an LLM returns a hallucination, a poorly formatted JSON object, or a blocked response, the application often runs a validation script and silently prompts the model to try again. A single user request can trigger three or four background API calls before returning a result. An architect reviewing the logs of an internal HR chatbot once found that a single frustrated employee asking for a dental claim form had maxed out the context window and cost the company three dollars in API calls to find nothing.

This variable cost structure creates a mismatch in the procurement process. Finance teams know how to budget for seats. They do not know how to budget for tokens. Procurement receives two proposals: a per-seat agreement with a minimum commitment, and an API consumption rate card. The per-seat license often looks more expensive per user than raw API consumption would be for the median employee. But procurement is paying for budget predictability. A per-seat license is an insurance premium that shifts the risk of runaway inference costs back to the vendor. Consumption pricing is cheap at low volume but explodes at scale.

Procurement sits on a call with a cloud vendor account manager, trying to negotiate an enterprise discount on inference. The vendor wants a committed annual spend to unlock the discount tier. Procurement is looking at a spreadsheet from the engineering team that estimates annual token consumption with a massive margin of error and no historical baseline. You cannot negotiate a consumption contract without usage data, and you cannot get accurate usage data without running production.

The budget owner rarely controls the bill. An application team can add more retrieved documents, extend the conversation window, or change models to improve accuracy, and the resulting expense lands silently in a central platform budget. A platform team will see the invoice rise after a release while request counts remain flat. An application change silently included more search results in each call. Nothing in the monthly user report explains the increase.

Average monthly usage hides the capacity question. A service that is affordable across a month may still hit rate limits during a concentrated morning workload. A support manager expands access from a handful of specialists to the whole service desk. Staff use the assistant most heavily during an outage, when tickets are longest and follow-up questions are frequent. The month with the least tolerance for delay is also the month with the highest inference bill.

The pilot established that people would use the system. It did not establish how often they would come back when the first answer failed. Model choice is a production setting with a budget consequence, not just a quality setting. If the tool is built tightly around one flagship model, the team surrenders its only mechanism for cost control. Routing simple queries to smaller models on the backend requires engineering logic that is often skipped after a successful pilot.

The first production invoice is not a billing anomaly. It is the first accurate measurement of what the application actually does.