How I build with LLMs
Two shapes of system here. One is conversational — a health coach with tools and memory — and the rest are bounded jobs: a component that gets a defined task, returns structured output, and is checked before anything it produced reaches a user. The rules below cover both, because the chat agent is the harder case, not the exception to them.
These are the rules I’ve ended up with after running ten of them. Every claim below is traceable to a project on the index or to the code behind it.
01Agents, when the work is a conversation
Not every job is a one-shot. MyAlivea is a conversational health coach: a LangGraph state machine over LangChain tools, with the graph’s state checkpointed in Postgres. The model decides which tool to call — pull a lab history, compare a test across twelve months, record a weigh-in, move a goal — so the answer is grounded in the user’s own record instead of recalled from training.
- Thirteen tools over the user’s own data: lab history, results on a date, a twelve-month comparison, medications and conditions, weight, goals and check-ins
- Memory that survives a restart: the LangGraph checkpointer keeps graph state per conversation, so a new tab, a redeploy or a resumed thread carries on where it stopped
- Compaction instead of truncation: past ten recent messages and an 8,000-token estimate, older turns are summarised, marked compacted and replaced by one summary message, while pinned messages are exempt
- Streaming through
astream_eventsto SSE, so tool calls and tokens appear while they happen
See MyAlivea.
02Bounded jobs, not conversations
Every model call asks for JSON against a schema: ingest this table of contents, classify this headline, estimate this probability, draft this report from these facts. The call has a contract test, so a prompt change that breaks the shape fails in CI rather than in front of a user. Adding a new kind of job is one file plus a registry entry — not a new pipeline.
See Cadence (ingest, enrich, requirements, paperwork — all the same shape) and Predict’s estimator registry.
03Cache by content hash
Results are keyed by a hash of job, model and input. The same textbook is processed once rather than once per family; a re-run of a backtest is byte-identical and costs nothing. Caching at this level is also what makes the cost of an AI feature predictable instead of a surprise at the end of the month.
See Cadence and Predict (search results cached per forecast).
04Models write sentences, not statistics
Generated prose is produced from a facts blob, and a number guard rejects any figure that isn’t in the facts. The model never originates a number that reaches a user. If the model is down, the email or the page still ships, minus the sentences.
See Gridiron’s weekly email and Cadence’s morning and weekly emails.
05Measure before trusting
Predict ships with a backtesting harness: multi-class Brier, log loss, expected calibration error, and paired bootstrap intervals against the market, with searches date-bounded at the forecast date so the model can’t see the future. The current score is published on the project page, and it is worse than the market. That is the point of measuring.
See Predict.
06Scheduled and event-driven, with workers
Schedulers only enqueue; workers run. Scaling is adding workers, not adding cron entries. Every job is idempotent per (user, period, kind), so a retry is safe and a restart loses nothing — the queue row is the state.
See Gridiron, MyAlivea’s thirteen jobs, and Datacenter Radar’s feed probing.
07LLMs are swappable
The projects are LLM-agnostic: the provider is configuration, not architecture. Every app talks to the model through one client behind one interface, so the same job runs against OpenAI, Anthropic — direct API or through AWS Bedrock — or the DeepSeek API. Cadence gets all three from LiteLLM behind a single call; the others keep the same seam over an OpenAI-compatible client or a provider SDK. Choosing a model for someone else’s constraints — data residency, cost, latency, vendor — is the evaluation harness in 04 pointed at a different endpoint, not a rewrite.
Today that configuration is DeepSeek, because token cost decides it when you are running this much batch work. Anthropic Claude still runs the extraction jobs on ShelfCaster and Epic Fury, and Stock Radar carries an Ollama path for local runs.
See the Stack line on each project page.
08How the work gets built
Development is its own stack, and it has the same shape as production: a model in a loop with tools, and a check it cannot fake. Interactive work happens in Claude Code on a Max subscription — reading, editing, running the tests. Longer, cheaper iterations go to a custom harness built on OpenClaw, running DeepSeek: point it at a repo and a task, it reads the code, runs the backend and frontend suites in their containers, and comes back with a diff and the test output. Anything it produces is reviewed like a colleague’s pull request, which is the only reason to trust it.
This site was built that way: an OpenClaw agent on DeepSeek, deployed from the same machine that runs the apps.
09Somebody has to notice
Every service exposes /api/health, every container has a healthcheck, and everything runs with restart: unless-stopped — a dead process comes back before anyone is told. What won’t fix itself is silence, so the workers mail or text when a student goes quiet, when a course slips, when a batch fails, and the table that records each send is the same one that stops it sending twice.
Cost is bounded the same way: cache hits, batched jobs, per-family rate limits.
10Know when to turn it off
The Inverse generated the strongest counter-argument to any news story on demand. It worked, and that was the problem: adversarial takes read as insight without evidence behind them. The containers are stopped and the entry stays up, labelled offline.
See The Inverse.