Questions to Ask Before You Deploy Predictive AI in L&D

Every health system evaluating predictive AI for Labor & Delivery eventually sits through the same demo. A dashboard lights up, a risk score appears next to a patient’s name, and someone in the room asks the only question that matters: should we believe it?

There is now a structured way to answer that. ONC’s HTI-1 rule created a certification criterion for Decision Support Interventions that requires certified health IT developers to publish a defined set of source attributes for every predictive model they supply: how it was trained, on whom, how it was validated, and how it should be maintained. Certified developers have been subject to it since January 1, 2025.

Most AI tools sold into L&D are not themselves certified health IT. That does not make the attribute list irrelevant. It makes it the best available buyer’s checklist: a set of questions a vendor with a defensible model can answer and a vendor without one cannot.

The Questions That Separate Real Models from Demos

What population was this trained on, and does it look like ours?

Ask for the training cohort in specifics: number of deliveries, date range, number and type of sites, payer mix, nulliparous share, induction rate, baseline cesarean rate, and racial and ethnic composition. A model trained entirely at academic quaternary centers will behave differently on a community unit with 900 deliveries a year and no in-house maternal-fetal medicine.

The follow-up matters more than the answer: has it been validated on a population like ours, and can we see those results separately? Training data and validation data are different claims, and vendors sometimes present one as the other.

What exactly does it predict, and at what moment?

“Predicts adverse outcomes” is not a specification. A usable answer names the outcome, the prediction horizon, and the decision it is meant to inform. For example: probability of cesarean delivery given the labor course as of this hour, or probability of postpartum hemorrhage exceeding a defined blood loss threshold at the time of delivery.

Vague outcome definitions are not just a documentation problem. They make it impossible to audit the model later, because you cannot count events you never defined.

How was performance measured, and was calibration measured at all?

Nearly every vendor will quote discrimination, usually an AUC. Far fewer will show calibration, which is the property that actually determines whether a number can be acted on. A model with an AUC of 0.82 that systematically overstates risk by a factor of two will generate a steady stream of interventions that were never warranted, and clinicians will learn to ignore it within a month.

Ask for a calibration plot. Ask for sensitivity and positive predictive value at the specific threshold the product ships with. Ask what fraction of patients cross that threshold. The alert rate is the number that determines whether your nurses will tolerate the tool.

Does performance hold across subgroups?

Maternal outcomes carry well-documented disparities, and a model trained on historical care patterns can learn them. Ask for performance stratified by race and ethnicity, payer, parity, and BMI. A vendor who has not run that analysis has not finished building the product.

Pay particular attention to any input that encodes access to care rather than physiology. Prior prenatal visit count, for instance, is predictive precisely because it reflects who was able to get care.

How will we know if it drifts?

A model deployed in 2026 sees a different population in 2028: different induction practices, different guidelines, a different documentation template upstream. Ask what is monitored, how often, who reviews it, and what triggers retraining or withdrawal. Then ask for the monitoring output to be contractually available to you, not just to the vendor.

What Good Looks Like in Practice

A vendor who has done the work will hand you a model card without being asked, will distinguish training from validation cohorts unprompted, will show calibration alongside discrimination, will name the failure modes, and will tell you which patients the model should not be used on.

A vendor who has not will answer in AUCs and adjectives.

One More Thing to Evaluate

Accuracy is necessary and not sufficient. The model also has to arrive somewhere a clinician is already looking, at a time when the decision is still open, with enough context to act on. A perfectly calibrated prediction delivered in a separate portal that nobody logs into has an effective accuracy of zero.

Evaluate the integration with the same rigor as the algorithm. That is usually where these deployments succeed or quietly fail.