A full look at the local AI inference engine Opta built for itself: what it is, how it runs, and the optimisations that make running models on your own machine a real option.
Every part of Opta eventually calls a model. That model can run in the cloud, or it can run on a machine you own. Opta Local is the engine that makes the second option viable: Opta's own local inference host, the engine that serves models on your hardware.
The local-inference tools that already existed treat local AI as a commodity runtime. They load a model, return tokens, and do little more. They are general-purpose, third-party, and not designed to be operated by an autonomous system. For Opta that fell short on two counts.
It needed to be engineered to be optimal. A commodity runtime leaves real performance on the table: repeated work that could be cached, models that reload from scratch, hardware paths that go unused.
It needed to be fully operable by AI. Opta's agents load models, swap them, and read live state on their own. That requires a clean, programmatic surface and honest observability, not a desktop app meant for a human to click.
So Opta built its own. Opta Local is Python, FastAPI and MLX, written to a published specification, and it is the only service in the Opta system authorised to host an inference engine. Everything else, the coordination, memory, research and routing services, calls into Opta Local over HTTP.
Opta Local runs models with MLX, Apple's machine-learning framework built for Apple Silicon. On a Mac with an M-series chip, the CPU, GPU and memory share one pool. MLX uses that unified memory directly, which is what makes large models practical on a single desktop machine.
Opta Local speaks the same HTTP shape as the OpenAI API. That is a deliberate design choice. Any tool or library that already knows how to talk to a cloud model can point at Opta Local instead, with no rewrite. The endpoint changes; the code does not.
Alongside the standard endpoints, Opta Local adds a small set of Opta extensions for things a commodity runtime tends to hide: explicit model load and unload, a health endpoint that reports cache hit-rate and memory, and a metrics endpoint for observability.
A model is requested by name. Opta Local returns immediately as loading, then a worker thread materialises the MLX weights from a local cache, or fetches them on demand, and reports loaded without blocking the caller.
Requests stream back token by token using the standard server-sent-events protocol, terminated by [DONE], so output appears as it is generated.
When memory pressure crosses its configured threshold, the least-recently-used model is unloaded gracefully. In-flight requests finish first, to make room for the next one.
These are the concrete things Opta Local is built to do that a commodity runtime does not. Each one is a design target written into its specification.
This is the optimisation that matters most for everyday use, so it is worth seeing closely. The first token of a reply is the slow part. The model has to read the entire prompt before it can say anything. In a long conversation that prompt grows, and the cost is paid again on every turn.
MLX 0.31 and later ship prompt-prefix caching built in; Opta Local exposes it as a first-class feature. On each request it normalises the messages and hashes the prefix, everything up to the latest user turn. On a match, it reuses the cached state and only does fresh work for your newest message. On a miss, it runs the full prefill and stores the result under that prefix hash, with least-recently-used eviction. The published targets: a first-token time of five seconds or less on a 10k-token prefix against twenty-five seconds or more without the cache, and a cache hit-rate above seventy per cent across a realistic ten-turn coding session.
These figures are engineering targets and acceptance criteria from Opta Local's specification. They are measured against a benchmark, not a claim that Opta Local is the fastest engine in existence. The point is a repeatable improvement over running a model cold, on hardware you already have.
Because Opta's own agents operate the engine, Opta Local cannot be a black box. Every request emits a structured trace recording which model answered, how many tokens moved, whether the prefix cache was hit, the time to first token, and the throughput. That data flows to Opta's tracing system, where rolling cache hit-rates and latency can be watched honestly, by a person or by an agent.
Opta Local is a piece of software, but it is built in service of something larger. The engine exists to serve the Opta Local Mission: a goal, not a product.
"Local AI competent enough to be a viable everyday alternative to cloud AI for most people, most of the time, without subscription stacking or surveillance."
The Opta Local Mission, stated plainly
The everyday way of using AI today asks two things of you. It asks you to stack subscriptions, a separate paid plan for each tool, and it asks you to send your work to someone else's computer to be processed. For a great deal of ordinary use, neither is necessary.
A capable model running on a machine you own answers without a meter and without your prompts leaving the room. The reason that is not already most people's default is that local inference has been treated as a commodity: good enough to demo, not engineered to compete. Opta Local is Opta's attempt to close exactly that gap. A local engine optimised hard enough, and operable enough, to be a genuine option for most people, most of the time.
Cloud AI keeps its place for the hardest, largest tasks. The Opta Local Mission is not to replace it everywhere. It is to make sure local is a real choice for everything in between.