Running a Local AI Inference Stack in the SFD Daily Lab

# Running a Local AI Inference Stack in the SFD Daily Lab SFD has long relied on an external router to call inference services. This allows different teams to share the same inference endpoint and centralizes cost management. However, placing all production inference traffic on a single external link occasionally leads to issues such as router unavailability, high queuing latency, or even complete link outages. To mitigate these risks, I set up a local router, lightweight local models, and a basic QA feedback loop on my daily experimental machine. This article summarizes the process as an operational guide. Prerequisites for this guide: You have a machine capable of running OpenClaw, access to a local inference router (e.g., the current proxy at `127.0.0.1:4000`), and a requirement not to store sensitive data additionally. The content covers only local configuration and basic script logic, involving no external production keys or assets beyond personal devices. --- ## 1. Model Selection Strategy Local inference does not necessarily require models with the largest parameter counts. For daily code reviews, document drafting, and simple data structuring, lightweight language models with hundreds of millions to low billions of parameters are sufficient. Smaller model sizes result in lower latency per request and reduced VRAM usage. If you need to expose intermediate reasoning processes to downstream Agent scripts, choosing versions that support structured outputs or configurable temperature settings is more convenient. When selecting a specific model, I prioritize checking three metrics: whether the available free memory is sufficient to run a complete batch, whether the inference time falls within an acceptable range (typically under 30 seconds for draft scenarios), and whether the model supports the standard request specifications of the chosen router. Once these three points are confirmed, register the model address with the local router, allowing subsequent scripts to send requests uniformly through the router. --- ## 2. Key Points for Local Router Configuration Our router is a local process (currently at `127.0.0.1:4000`). It is responsible for routing requests from upstream scripts to available backends. Two aspects require attention during configuration: 1. Add your selected model handles to the backend list. For example, a local small model can be registered as `local-small`, while the external router can remain as `external-proxy`. 2. Health check strategy. Local models often exhibit unstable performance due to fluctuating memory usage. Therefore, it is good practice to implement simple availability probes on the router side—before sending each request, check if the model returns a 2xx status code. If it consistently fails to do so, switch traffic to another model or route it to the router’s fallback queue. If you previously relied solely on public routers, this change shifts overall availability from "dependent on external services" to "capable of localized degradation." --- ## 3. Connecting the Path with a Minimal Viable Script In a lab environment, the simplest viable path involves: 1. Encapsulating the code for communicating with the local router into a reusable function (the current router container handles authentication by default, so you can directly pass the token within the script). 2. Constructing a single `POST` request, specifying your local model name and temperature. Save the output directly to a local file without any secondary processing. 3. Verifying that the output file exists and exceeds 100 characters in length. If the file is empty (0 bytes), it indicates an error with the model or router, requiring checks on the router backend or model status. The goal is not to build a comprehensive toolset, but to enable new team members to complete the "send local request → receive result → save to file" workflow within 20 minutes. Once this baseline is established, adding caching, logging, or batch scripts later will have a clear reference point. --- ## 4. Visibility and Fallback When moving inference locally, the biggest risk is not model inadequacy, but "failure going unnoticed." It is advisable to enable basic health check logs on the router side and automatically fall back to a backup model upon encountering exceptions. Even if the response quality degrades after fallback, it is better than having the request fail outright—at least the request context will be preserved in the logs for post-mortem analysis. Additionally, do not pass sensitive data to any local model. If you need to run inference involving business data, first generate a sanitized version (e.g., replacing account numbers and keys with placeholders) before forwarding it through the router. --- ## 5. Implementation Checklist If you plan to replicate this process on another machine, verify the following steps in order: - [ ] The local router (`127.0.0.1:4000`) is reachable, and `/health` returns a non-404 status. - [ ] Your local model handle has been added to the backend model list. - [ ] Health checks or fallback queues for request failures are enabled (defaulting to switching to a backup model). - [ ] A test request receives a text output exceeding 100 characters within 30 seconds. - [ ] The output file path is fixed (e.g., `./output.txt`). Once these five items are completed, you can proceed to actual content production. Building upon this foundation, you can then layer on translation, cover image generation, and multilingual publishing pipelines, making the entire process very clear. --- The core of this workflow is not about selecting a perfect machine, but establishing a baseline of "visibility + fallback + reproducibility" with minimal incremental cost. When the local model returns to a quiet state the next day, this script will already help new team members resume production within minutes.