Write the Rollback Plan Before Going Live (Rollback First)

Last week, I monitored a release where a boundary data exception occurred just 20 minutes after the new code went live. Troubleshooting, root cause analysis, and preparing a hotfix burned through two hours. In the post-mortem, we uncovered a more painful truth: if we had spent just 10 minutes before deployment writing a "rollback ticket," we could have skipped at least an hour and a half of that downtime. The first step in any incident window should always be "rollback first, investigate later." Yet, no one had confirmed beforehand how to roll back or to which state.

The "Rollback First" habit is simple: **Before performing any irreversible or semi-irreversible operation, write down "how to revert" instead of figuring it out after things go wrong.**

When to Use It

- **Deployments**: For any code, configuration, or data migration entering production, you must answer three questions before release: What are the rollback triggers? How many steps and minutes will the rollback take? How do we handle data cleanup after rolling back?

- **Database Changes**: Before adding columns, changing types, or running migration scripts, confirm whether the old version of the service can run directly on the new schema. If not, adopt an "add-then-modify" strategy across two separate releases.

- **Deletions**: Deleting files, branches, configuration items, or clearing caches. Before hitting `rm`, think about where the `trash` or backup will reside.

- **External Commitments**: Before sending bad news or change notifications to clients or management, plan how you will respond to their follow-up questions and consider whether you can retract the statement if it turns out to be incorrect.

- **Agent Operations**: Before letting automated scripts perform batch operations, ensure each batch is reversible, or at least that input snapshots are taken.

When Not to Use It

- **One-off Experiments**: In local experimental environments, drafts, or scenarios where you can easily start over, writing a rollback plan is mere formalism and a waste of time.

- **Pure Additive Changes**: Adding an unreferenced configuration item or a new table without touching existing logic—these "naturally reversible" operations only require a brief confirmation, not a full ceremonial process.

- **Scenarios Where Rollback Costs Exceed Business Costs**: In rare cases (e.g., shutting down an already leaked API endpoint), the right decision is to deploy quickly and directly; rolling back would be the wrong move. Determining "whether rollback is possible" takes precedence over "whether to rollback."

Minimum Checklist for a Rollback Plan

A qualified rollback plan fits neatly into the release notes:

1. **Trigger Conditions**: Define specific signals that mandate a rollback, rather than adopting a "wait and see" approach. (Examples: Error rate >1%, P95 latency of core APIs doubles, or more than 3 customer support tickets reported.)

2. **Rollback Actions**: Be specific down to commands or buttons. "Roll back to the previous version" is not an acceptable answer; "`git revert` this commit and redeploy via CI pipeline #x" is.

3. **Estimated Time**: Estimate the minutes from the decision to roll back until service recovery. If a rollback plan exceeds 30 minutes, you should rehearse it before going live.

4. **Data Cleanup**: How do you handle dirty data after rolling back? Should newly written records be kept, masked, or deleted?

5. **Executor**: Name the person responsible. Saying "find the on-call engineer to roll back" during a midnight incident effectively means no one will roll back.

Common Pitfalls

- **The rollback plan exists only in your head.** During post-mortems, everyone claims they "knew how to roll back," but during incidents, they realize the DB schema is no longer compatible with the old code. Writing it down makes it real.

- **Confusing "rollback" with "fix."** The default action should be to roll back first to stop the bleeding, then investigate calmly. Many teams attempt hotfixes directly in production when issues arise, turning one incident into two.

- **Lack of Rehearsal.** A rollback plan that hasn't been tested is merely a guess. Before major changes go live, run through the full rollback process in staging. This usually takes 10 minutes and eliminates half the illusions in your plan.

- **Rolling back code but not configuration.** If you revert the code to an older version but leave the new configurations in place, the service still won't start. Your rollback plan must list code, configurations, and data together.

One-Sentence Summary

The courage to deploy doesn't come from "I think it's fine," but from "Even if there's a problem, I can revert within 15 minutes." A rollback plan isn't bureaucratic red tape; it's the lowest-cost insurance policy you can buy for yourself.