Many pilot programs can show activity—visits completed, referrals accepted, devices deployed—but struggle to prove economic value in a way that survives contracting. “We think we avoided ED visits” is not a contracting narrative. A contract-ready narrative starts with design choices: what you count, how you attribute, and how you separate real impact from documentation shifts. If you’re working within Pilot Evaluation & Learning Loops, treat economic evaluation as part of operations, not finance. Programs frequently position the same discipline as a bridge into New Service Models, because payers and systems fund models they can price, govern, and renew.
Two expectations recur across payers, Medicaid managed care, and system leaders. First, they expect an attribution story they can defend internally: who would likely have used acute care, what comparison you used, and what rules you applied to avoid double counting. Second, they expect evidence packaging that ties cost to workflow: a cost model anchored in actual staffing and process steps, with a clear method for handling missing data and a documented governance approach for adjustments.
Start with the contracting question, not the spreadsheet
Economic evaluation is not about perfection; it is about credibility under scrutiny. Most buyers want to know: (1) what does it cost per enrolled person (or per episode), (2) what utilization changes are plausibly attributable, and (3) what is the time horizon for savings versus investment. If you cannot answer those in plain language, the pilot may still be clinically meaningful—but it will be hard to fund at scale.
Define your unit of analysis and your decision threshold
Before collecting cost data, define what “success” means in a way that maps to purchasing. Is this a per-member-per-month service? A per-episode service (e.g., post-discharge 30 days)? A capacity model (mobile response hours)? Then define decision thresholds: what effect size (timeliness, avoided utilization, reduced length of stay, reduced readmissions) would be sufficient to justify expansion, and what level of uncertainty is acceptable. These thresholds shape what you measure and how you interpret ambiguous results.
Operational Example 1: Time-driven cost capture linked to workflow steps
What happens in day-to-day delivery
The pilot builds a simple “time-driven” cost map from the real workflow: intake, eligibility verification, first outreach, assessment, clinical review, follow-up, escalation handling, and closure. Each step has a role owner (e.g., coordinator, RN, paramedic, CHW), a typical time range, and a documentation artifact (template or timestamp). Staff do not fill in new forms; instead, the program pulls timestamps from existing systems (scheduling, documentation, call logs) and uses periodic sampling to validate step durations. Supervisors review monthly outliers (cases with unusually high time) to determine whether the variance is clinical complexity, operational inefficiency, or documentation gaps.
Why the practice exists (failure mode it addresses)
Pilots commonly understate true cost because they ignore coordination time, supervision time, and exception handling (no-shows, device failures, repeated outreach). Alternatively, they overstate cost because they allocate generic overhead without linking it to delivered steps. Time-driven cost capture exists to prevent both failure modes by anchoring cost in actual delivery mechanics that can be explained and replicated.
What goes wrong if it is absent
Without a workflow-linked cost model, the pilot’s economic story becomes fragile. Finance teams may apply broad averages that don’t match operations, leading to cost figures that front-line leaders can’t explain. In contracting discussions, payers quickly detect this: “Why is your cost per person higher than your staffing model implies?” or “Where did the supervision time go?” The result is lost credibility even if outcomes are strong, because the program cannot show what it truly takes to deliver reliably.
What observable outcome it produces
A time-driven model produces defensible unit costs and operational insight. You can show cost per enrollment, cost per completed assessment, and cost per escalation handled—each backed by timestamps and role rates. You can also show where efficiency improved across learning cycles (e.g., reduced eligibility time after redesign). Evidence includes the step map, sampling validation notes, and variance reviews that explain outliers rather than hiding them.
Operational Example 2: Avoided utilization logic with explicit rules and guardrails
What happens in day-to-day delivery
The pilot defines “avoidable utilization candidates” prospectively. For example: high-frequency 911 callers; patients recently discharged with known risk factors; people with repeated ED visits for ambulatory-care-sensitive conditions; or residents in settings where mobile response can stabilize a situation. The program sets explicit rules for counting an “avoid”: what event must have been trending (symptom deterioration, caregiver escalation, device alerts), what intervention occurred (home visit, clinician consult, medication coordination), and what evidence supports that the trajectory changed (stabilization markers, documented disposition, follow-up outcome). A small review panel meets biweekly to adjudicate ambiguous cases using a standard checklist and records decisions.
Why the practice exists (failure mode it addresses)
Retrospective “we probably avoided” claims are the fastest route to skepticism. Avoided utilization logic exists to prevent confirmation bias and to show that the pilot applied consistent rules. It also supports oversight expectations: decision-making that is auditable and governed, rather than dependent on narrative interpretation after the fact.
What goes wrong if it is absent
If rules are not defined up front, staff will naturally remember the strongest stories and forget the neutral or negative cases. The program then reports inflated avoided utilization and cannot reconcile discrepancies when claims data arrives months later. Payers will discount the entire evaluation because the counting method looks like marketing, not measurement. Internally, the pilot team also loses the chance to learn: without adjudication, you can’t identify which interventions truly changed trajectories and which just added activity.
What observable outcome it produces
With explicit rules, the pilot can present avoided utilization as a disciplined estimate with bounds, not a boast. You can show counts by cohort, confidence categories (high/medium/low), and reasons for exclusions. Evidence includes the adjudication checklist, panel logs, and follow-up outcomes that verify stability (e.g., no ED visit within a defined window; documented resolution; reduced repeat calls). This is the kind of packaging that supports renewal conversations.
Operational Example 3: Comparator design that commissioners can understand
What happens in day-to-day delivery
The pilot establishes a practical comparator strategy that fits the data available. Common approaches include: a pre/post comparison for the same cohort with a defined baseline period; a matched comparison group from similar referrals not enrolled due to capacity; or a stepped rollout where later sites serve as temporary comparators. Operationally, this requires a clean enrollment definition (start date, eligibility rules, exclusion criteria) and a consistent follow-up window. The evaluation lead maintains a “comparator memo” that documents the choice, its limits, and any changes made over time, with leadership sign-off when changes occur.
Why the practice exists (failure mode it addresses)
Pilots often fail economic scrutiny because they lack a credible counterfactual: what would have happened without the pilot. Comparator design exists to prevent over-claiming and to make impact legible to decision-makers who must justify funding. Even a modest comparator, clearly described, is stronger than an uncontrolled narrative.
What goes wrong if it is absent
Without a comparator, any observed changes can be explained away as regression to the mean, seasonal trends, shifting referral patterns, or documentation effects. Commissioners will hesitate to scale because they cannot separate pilot impact from background noise. Internally, teams may also misread the data—expanding a model that appears to work but is actually serving a different risk profile over time.
What observable outcome it produces
A clear comparator approach produces results that can be defended and repeated: changes in utilization rates, time-to-follow-up, or cost per episode anchored to a defined baseline or comparison cohort. Evidence includes enrollment definitions, the comparator memo, and a consistent set of measures reported each cycle. This also improves learning loops: when results shift, you can investigate whether operations changed or the comparator assumptions changed.
Packaging the evidence for renewal and scale
Contract-ready evaluation outputs usually include: unit cost (with workflow basis), utilization impact estimate (with explicit rules), and a sensitivity view (best case, plausible case, conservative case). Decision-makers expect transparency about limits: missing claims data, small sample sizes, or selection bias. Counterintuitively, acknowledging limits increases credibility—especially when paired with a clear plan for strengthening evidence in the next cycle (data sharing agreements, expanded cohorts, or stepped rollout design).
A simple checklist before you publish pilot “results”
- Can operations explain the cost model in the same language as the workflow?
- Are “avoids” counted using rules defined before outcomes were known?
- Is there a comparator story (even modest) that a commissioner can repeat?
- Is there a dated audit trail for method changes and adjudication decisions?
- Do you present conservative and plausible scenarios, not just a single headline number?
Economic evaluation does not need to be academic to be rigorous. It needs to be operational, governed, and packaged for real purchasing decisions—so your pilot can move from “interesting” to “funded.”