Running the Pipeline Three Times a Day: Idempotency Matters More Than Speed

Our content pipeline triggers three times a day. The real nightmare isn’t “failed to publish,” but “published twice.” A single duplicate publication doesn’t just wash away that specific piece of content; it erodes trust in the entire pipeline.

Where Do Duplicates Come From?

Let’s review the background: Daily updates are handled by cron jobs, scheduled at 09:00, 14:00, and 20:00. Each run generates one original article in zh-CN, followed by a three-language publishing process. Everything ran smoothly for the first few weeks until one evening, when network jitter caused the script to time out and exit during the reporting phase. The cron job’s retry mechanism kicked in and re-executed the task, resulting in two articles with identical slugs and body content being published on the same day.

The initial fix was manual: we checked the publication logs, identified the duplicate pair for that day, and manually deleted one of them. During this cleanup, we realized that any step relying on “manual intervention as a safety net” is a breeding ground for duplicates. Manual cleanup lacks consistent standards—Person A might decide to delete Article X, while Person B thinks Article Y should go. The more we cleaned, the messier it got.

Three Lines of Defense, From Outside In

The first line of defense is naming conventions. Articles published on the same day share the same slug prefix, formatted as an eight-digit date. This prefix serves as a queryable fingerprint: before publishing, we send a GET request to the API asking, “Has an article already been published today?” If we find a match, we verify it; if not, we proceed with publishing. The cost of this step is just one GET request, but it eliminates reliance on anyone’s memory for duplicate detection.

The second line of defense is a pre-publishing gatekeeper. When an article with the same name is found, the script neither automatically skips nor overwrites it. Instead, it offers three options: Publish, HOLD, or BLOCKED. The HOLD state is often overlooked—intuitively, one might think duplicates should simply be deleted. However, a duplicate usually indicates that an unknown task instance has already run. The correct action is to investigate why it ran, not to delete the superficial data.

The third line of defense is the write-side safety net. The publishing script does not rely on a “check-then-write” pattern to prevent concurrency issues. Instead, it delegates idempotency to database unique constraints, enforcing uniqueness on the combination of slug and locale. If a conflict occurs, the write is simply ignored. The second call neither throws an error nor overwrites the first result; it becomes a harmless no-op.

Two Pitfalls We Encountered

Pitfall 1: There is always a window between checking and writing. There are a few milliseconds between duplicate detection and the actual write. If two tasks execute concurrently, both may pass the “no article today” check. At this point, the only thing that can stop duplicates is the database unique constraint. Therefore, this constraint must be enforced at the database level, not just in application code.

Pitfall 2: Validation failure does not equal write failure. Initially, we treated “validation failure” as a signal to retry. This led to a scenario where the write actually succeeded, but the cache wasn’t ready yet. The validation read stale data, triggered a retry, and a second write attempt came in concurrently. Later, we decoupled these two concerns: write status is determined solely by the database return value; content validation performs read-only checks, triggering alerts on failure but never triggering a rewrite.

How It Runs Now

After implementing this mechanism, each of the three failure modes maps to a specific action: if nothing is found, publish; if a match is found, HOLD; if a write conflict occurs, perform a no-op. Duplicate publications have dropped to zero. The cost? One preliminary GET request, one unique constraint, and an additional HOLD state.

I won’t boast about the technical sophistication of this solution—it’s all straightforward operations. The truly valuable insight boils down to one sentence: Ensure that executing an automated task twice yields the same result as executing it once. Only when you achieve this can you confidently let it run three times a day or thirty times a month, without staying at your desk all night listening to alerts.