Calculate Your Retry Budget: What Does a Single LLM Failure Really Cost?
Last week, a friend’s production service triggered alerts in the middle of the night. Everything looked normal on the model provider’s side, but 23% of our traf
SmallFireDragon Lab
AI Science
Making complex AI concepts understandable for humans
Why LLMs "Confidently Spout Nonsense": An Engineering Perspective on Hallucinations
The most frequently asked question in AI system development is: Why do models make mistakes? The term "hallucination" sounds mysterious, but when broken down, i
Slow Isn’t About the Model: Four Quantifiable Components Hidden in AI Service Latency
Last week, I helped a team troubleshoot a production issue: their customer service bot was responding too slowly, leading to user complaints. The boss’s immedia
Why "Switching to a Larger Model" Is Often Not the Answer: AI System Selection and Degradation Chains
Recently, a friend was backed into a corner by a production incident. Their immediate reaction was, "Let's switch from GPT-4 to a bigger, more expensive model."
Inference Timeouts Are Not Black Magic: Set Three Numbers for LLM Calls
The dilemma over timeouts for LLM calls often plays out like this: 20 seconds is too short, occasionally killing long prompts; 120 seconds is too long, causing
Pinning Down the Seed: Minimal Fixes for LLM Reproducibility
Reproducibility is the cheapest quality signal in engineering: only when you run the same input and code twice on the same day and get consistent results can yo
Build an LLM Evaluation Suite in 15 Minutes: No MLflow, No Platform
Every time you switch models, tweak prompts, or adjust the temperature, you want to know if the results are getting better or worse. But often, you’re stuck rea
Adding Rate Limiting to Large Language Models: Why Higher QPS Limits Aren't Safer
Many teams treat "QPS limits" as a large number that they gradually decrease until business alerts trigger, at which point they lower it slightly more. However,
How to Implement Retry Logic for LLM API Calls: Which Errors to Retry and Which to Avoid
Anyone who has worked with LLM services has written this code: a call fails, so you wrap it in a for loop to retry three times. It seems robust, but half of pro