Shipping an AI Feature to Production: Pre-Launch Checklist

A demo shows what the model can do on a good day. Production has to survive the bad days: odd inputs, outages, slow responses, angry users and changing models. These are the things I check before putting an AI feature in front of customers.

Scope it narrowly

Pick one job the feature does well and say what it will not do. A narrow assistant that is right 95 percent of the time on one task earns more trust than a general one that is sometimes wildly wrong. Block off-topic requests with clear messages.

Have an evaluation set

Before launch you should have a set of real cases and a pass rate you accept (see how to judge AI output). After launch, add every reported failure to the set so it cannot silently return.

Plan for failure

• Timeouts and retries with backoff; providers have outages and rate limits.
• Fallback behaviour: a plain, non-AI path or a clear "try again later" message.
• Validation of model output before using it. If a program consumes it, check the schema.
• Kill switch: a configuration flag to turn the feature off in seconds.

Keep humans in the loop where it matters

For anything irreversible or customer-facing (messages, money, deletions), let the model draft and a person approve, at least until measured accuracy justifies more autonomy. See the security post for why.

Privacy and data handling

Decide what data goes to the model provider, and send the minimum. Check the provider's data-retention and training terms for your plan. Do not put secrets or unnecessary personal data in prompts. Tell users when they are interacting with AI and how their data is used.

Monitor what actually happens

Log inputs, outputs, latency, token counts and errors (with privacy in mind). Review a sample of real conversations weekly. Add a thumbs-up/down control so users can flag bad answers. Track cost per feature (see AI cost and tokens).

Version everything

Pin the model version where the provider allows it, keep prompts in version control, and roll out changes gradually. Provider-side model updates can change behaviour, so rerun your evaluation set on a schedule.

Be honest with users

Say what the feature is, what it can get wrong, and how to reach a person. Users forgive limits they were told about, and not surprises.