witn vs Building In-House
Considering a DIY outcome-based billing system? Learn the hidden costs of settlement, idempotency and auditing that turn a simple project into a major build.
By George Kats
Your engineering team has a new feature request: bill customers for AI agent outcomes. The initial scope seems manageable. A new resolutions table in the database and a scheduled job to scan it should do the trick. This approach, however, often overlooks the true complexity of a reliable outcome-based billing system.
The True Scope of an Outcome-Based Billing System
The resolutions table is the visible 10 percent of the project. The submerged 90 percent is a stateful stream-processing and correctness problem that can consume quarters of senior engineering time. This is not a question of whether a competent team can build it. The question is whether it is worth the opportunity cost. The hidden work involves challenges that a simple database scan cannot solve: settlement windows, idempotency, out-of-order events, audit trails and rerating.
These are not edge cases. They are fundamental requirements for a system that customers and finance teams can trust. Building this layer correctly is a significant engineering investment. In fact, it is a product in itself. We know because we built it. Our technical deep dive on the resolution layer details exactly what you would be rebuilding, from the state machines to the concurrency controls.
Ensuring Correctness with Settlement and Idempotency
A resolved outcome is not yet a charge. It first enters a configurable settlement window, a period during which a cancellation credit or reopen event can reverse it. Only after the window closes cleanly does the outcome commit to an invoice. A simple cron job has no native concept of this waiting period. Adding it introduces significant state management complexity, turning a straightforward scan into a web of timers and conditional logic that is difficult to test and maintain.
Beyond settlement, there is the problem of idempotency. Client applications retry, workers crash and message queues redeliver. An idempotent billing system must guarantee that a customer is charged exactly once for an outcome. A naive consumer that calls a payment API directly can easily double-charge on restart, a classic dual-write problem. Correctness requires deterministic event IDs for deduplication and a transactional commit process, not a simple boolean billed column. Failures here directly damage revenue and customer trust, which is a high price to pay for what seemed like a small feature. This is a core part of the missing layer in AI agent monetization.
Data Integrity for Out-of-Order Events and Audits
Distributed event streams rarely arrive in perfect chronological order. A system that processes events based on their arrival time, such as a cron job scanning records by created_at, will silently produce incorrect results. For example, a cancellation event might arrive before the creation event it corresponds to, leading to a missed reversal. Correct evaluation must key off immutable event timestamps and be prepared to re-evaluate an outcome's state whenever a late-arriving event is received. This requires a more sophisticated model than a simple sequential process.
This historical accuracy becomes critical when a customer asks "why was I charged for this?". Your system must be able to answer with a complete audit trail. This means tracing every charge back to the specific outcome and the exact sequence of events that triggered its resolution. Retrofitting this capability onto a mutable table after the fact is exceptionally difficult. An audit trail must be an immutable, foundational part of the system's design from day one. As we've discussed before, transparent invoicing is essential for preventing billing disputes and maintaining customer confidence.
Future-Proofing with Simulation and Scale
Your pricing models and outcome definitions will evolve. Before deploying a change, you must understand its impact on revenue. This requires the ability to simulate the new rules against historical data. A proper rerating and simulation engine is a whole second system, typically an event archive paired with a fast columnar query engine. Most teams building in-house never budget for this, leaving them blind to the financial impact of their pricing adjustments. This makes it impossible to test an outcome pricing model before it goes live.
As your business grows, your billing system for AI agents must scale with it. This means handling potentially millions of concurrent outcome evaluations, each its own state machine, without race conditions or performance degradation. This is a classic distributed systems problem that requires techniques like compare-and-swap versioning to avoid slow, expensive distributed locks. A simple database table with row-level locking will not suffice at scale. Instead of hand-coding this logic, a platform approach allows you to define complex conditions in simple terms.
In-House Build vs witn at a Glance
The following table summarizes the trade-offs between a typical in-house build and using a managed platform.
| Concern | In-house resolutions table + cron | witn |
|---|---|---|
| Settlement windows | Manual logic, complex state management | Built-in, configurable per outcome |
| Idempotency/exactly-once | Requires deterministic IDs, transactional commits | Handled automatically by the platform |
| Out-of-order events | Prone to silent errors if using arrival time | Correctly evaluates based on event timestamps |
| Audit trail | Difficult to retrofit, often incomplete | Immutable log of all events for every charge |
| Rerating/simulation | Requires a separate, unbudgeted system | Built-in tools for forecasting and analysis |
| Time to first correct invoice | 2+ quarters with 3-4 senior engineers | Hours to days |
| Ongoing operational load | Permanent on-call, maintenance, capacity planning | Managed by witn as infrastructure |
The Real Cost is Opportunity
Building this system correctly requires three to four senior engineers for at least two quarters to produce the first correct invoice. This does not include the permanent on-call load for maintenance and capacity planning. The real cost is the opportunity cost. Those are the same senior engineers who should be improving your core AI agent, which is the product your customers are actually buying.
The build vs buy billing software decision is a strategic one. Before dedicating a team to this project, engineering leaders should ask a few critical questions:
- Is billing correctness a core competency we want to own and maintain?
- Who carries the on-call pager for a double-charge incident at 2am on a weekend?
- What features in our core product will not get built while we are building and maintaining this billing system?
Frequently asked questions
Can't we just add a resolutions table?
How long does building this actually take?
What is the hardest part to get right?
When does building in-house make sense?
Billing as Infrastructure Not a Project
A correct and reliable outcome resolution layer is not a feature to be shipped and forgotten. It is critical infrastructure that underpins your revenue and customer trust. witn provides this layer as managed infrastructure. Your team defines what a billable outcome looks like and sends the corresponding events. Our platform handles the complex work of resolving that outcome into a correct, settled and fully auditable charge.
Focus your engineering talent on building a better agent, not a billing system. Read the docs.
The complete monetization playbook
How to price, verify and bill the work your AI agent delivers. A practical playbook for founders, product leads and engineers, from choosing a pricing model to operating outcome-based billing in production. 17 pages, free download.