SmallFireDragon Lab

Lab Articles

Deep dives into tech, design, and AI exploration

No Overnight Demo Runs: Moving Preflight Checks to CI After a Simulation Environment Freeze
Article

No Overnight Demo Runs: Moving Preflight Checks to CI After a Simulation Environment Freeze

Last month, we delivered an automation script for a simulation environment to a team. During the final batch run demonstration before delivery, the system crash

Read More → →
Troubleshooting Retrospective: Assigning a "Handling Budget" to False Positives Saved the Detection Team Two Person-Months
Article

Troubleshooting Retrospective: Assigning a "Handling Budget" to False Positives Saved the Detection Team Two Person-Months

Last month, we deployed a content anomaly detection system for the business side. Within the first week, we ran into a familiar problem: the model was too "sens

Read More → →
Post-Mortem: How a "Silent Pipeline" Let High-Value Detection Windows Slip Away for Two Weeks
Article

Post-Mortem: How a "Silent Pipeline" Let High-Value Detection Windows Slip Away for Two Weeks

Last month, we spent an entire week troubleshooting the issue of "model performance degradation on specific categories" within one of our identification pipelin

Read More → →
The P99 Latency Doubled, But It Wasn’t the Model’s Fault
Article

The P99 Latency Doubled, But It Wasn’t the Model’s Fault

Last month, I helped an internal team troubleshoot performance issues with their intent classification service. Regional monitoring showed that P99 latency had

Read More → →
After the Automation Script Retried Itself to Death for the Third Time, We Separated Observation from Execution
Article

After the Automation Script Retried Itself to Death for the Third Time, We Separated Observation from Execution

Last month, during the 2 AM release window, the same bug triggered three alerts. The first time, the script created its own mess and cleaned it up. The second t

Read More → →
Failover for Local Inference Routing: The 38 Minutes We Lost in a Drill
Article

Failover for Local Inference Routing: The 38 Minutes We Lost in a Drill

The customer demo was scheduled for 3 PM. During the morning routine inspection, I switched the primary backend from Pool A to Pool B, aiming to test a real fai

Read More → →
Three Lines of Configuration We Missed, Discovered Only a Month After Migrating to Local Inference
Article

Three Lines of Configuration We Missed, Discovered Only a Month After Migrating to Local Inference

Last month, we switched our production inference from a cloud API to a local router. The metrics looked impressive: P95 latency dropped from 1400ms to 220ms, an

Read More → →
How Renting a Local Inference Cluster Led to "False Health": Three Pitfalls Encountered During Downtime Drills
Article

How Renting a Local Inference Cluster Led to "False Health": Three Pitfalls Encountered During Downtime Drills

Last week, during the final stretch of integrating our local inference cluster into production, everything looked perfect: CI was all green, monitoring was all

Read More → →
Practical Guide to Route Migration: Five Pitfalls When Moving from Legacy to Local
Article

Practical Guide to Route Migration: Five Pitfalls When Moving from Legacy to Local

Last month, we migrated our team’s local inference routing from legacy nodes on the intranet to local ports on individual workstations. It sounded like a simple

Read More → →