Running local models in Onyx Code

Start from your hardware, not from a leaderboard

The most common way to have a bad time with local models is to download the biggest one you have heard of. A model has to fit in memory alongside your editor and everything else you have open, and a model that fits comfortably will feel faster and more useful than a larger one that swaps.

Onyx Code detects your hardware during setup and marks the models that will run well on it, so the usual answer is to take the recommendation. If you would rather choose yourself, the rough shape is:

Quantisation is the other half of the question. A quantised model trades a little accuracy for a lot of memory. Q4 is the usual default and is what most people should take, Q5 costs a little more memory for a little more quality, and Q8 is worth it only if you have memory to spare. The model browser explains this inline rather than assuming you know it.

Downloading a model from inside the editor

There is no separate runtime to install. Onyx Code bundles the inference engine, so a model is the only thing you download, and you download it from inside the editor.

1. Open the Onyx icon in the activity bar and choose the Models view, or run Onyx: Browse Models from the Command Palette.
2. On the Browse Models tab, take something tagged Recommended for your hardware, or search HuggingFace directly from the same box.
3. Results can be sorted by Popular, Name, VRAM or Context. Sorting by VRAM is the fastest way to find the largest model your machine can actually hold.
4. A download can be paused and resumed from the Downloads section, and picks up where it stopped. If a network proxy sends the whole file again on resume, Onyx Code starts the file over, so the model always arrives intact.
5. Downloaded models land in ~/.onyxcode/models/ and appear on the Models tab with their size, parameter count and download date.

Activating a model card switches the chat backend to it, and the active one carries an Active badge. Models of 3B parameters or fewer are marked Light model, which is a useful signal when you are choosing a draft model rather than a chat model.

Using a GGUF file you already have

Any GGUF on disk works. If you have already built a collection through another tool, you do not have to download it again.

Point Onyx Code at the directory holding them and they appear alongside anything downloaded in-app:

{
  // Where Onyx looks for GGUF models
  "onyxcode.ai.modelsPath": "~/.onyxcode/models/"
}

Architecture matters more than filename. A GGUF built for an architecture the bundled engine does not support will not load, and embedding models are filtered separately from chat models because the two are not interchangeable.

The engine is llama.cpp. It ships inside each Onyx Code release and, from version 2.3.0, can also update itself between releases (see engine updates below). If a model fails with “unknown model architecture”, it is newer than the engine you have: Onyx Code says so and offers the newer engine in one click when one has been published. Until then, connect an Ollama, LM Studio or llama.cpp server you already keep current and run the model there.

The context window fits your machine.The bundled engine sizes a model’s context window to your free memory, up to the size the model was trained for, so a model with a very long window (a million tokens) starts on an ordinary laptop instead of failing, and a smaller machine keeps the memory it needs. The chat input shows the window the model got.

Which processor runs the model

Local models run on the GPU wherever Onyx Code ships a GPU-accelerated engine for that platform, and on the CPU otherwise. You do not install a driver toolkit, and there is no external server to run.

The engine reports its own capability rather than guessing from the hardware, so the setup wizard’s Your Hardware step tells you which of GPU (Metal), GPU (Vulkan) or CPU you actually have. That is the fastest way to check what your install is doing.

You can override the choice three ways:

• Command Palette: Onyx: Select AI Processor (GPU / CPU).
• In chat: /processor shows the current setting, and /processor gpu, /processor cpu or /processor auto changes it.
• In settings: onyxcode.ai.inferenceBackend.

Auto is the right answer almost always. On a build with no GPU engine, choosing GPU does nothing, because there is nothing to offload to. Changing the processor restarts the local servers for you.

Keeping the engine current between releases

New model families need a newer engine before they can load. From version 2.3.0, Onyx Code keeps its llama.cpp engine current on its own, between app releases, so a new model family does not have to wait for the next version of the app.

• What happens: once a day Onyx Code checks for a newer engine. When one is published it downloads it, checks every file against its published checksum, and tests it on your current chat model before switching. The switch waits until no chat or Onyx Tab request is running, and a notification tells you which new models it can run.
• If a new engine misbehaves: an engine that fails to start twice after an update is rolled back to the previous one automatically and is not offered again. The engine that ships inside Onyx Code is never modified and is always the fallback.
• When a model will not load:if the engine does not know a model’s architecture, Onyx Code tells you and offers the newer engine with an Update Engine button, then loads the model.
• Where to see it: the Engine section at the bottom of the Models tab shows the engine version and processor (for example llama.cpp v0.4.1 · Metal), when it last changed, Check Now, and Update when a newer engine is available. From the Command Palette: Onyx: Check for Engine Updates.

Choose how it behaves with onyxcode.ai.engineUpdates: auto (the default) downloads, tests and switches on its own, notify shows the update and waits for you to click Update, and off never checks (Check Now still works). Engine updates never run in Air-Gap Mode, and each check, download, install and rollback is recorded in the AI Activity Log.

Four jobs, and why one model does not do all of them

Onyx Code runs different models for different jobs, because a model tuned to complete the middle of a line is not the model you want answering a question about your architecture.

• Chat. The model that answers questions and writes code in the panel. This is the one worth spending your memory budget on.
• Autocomplete. A fill-in-the-middle model, which is a different training objective from chat. Smaller and much faster, because it runs as you type.
• Onyx Tab. Next-edit prediction, which suggests where your following edit goes rather than finishing the current line.
• Embeddings. Turns your codebase into vectors so chat can retrieve the right files. The bundled default is a code-tuned embedder that runs on any machine with 4 GB of RAM.

The Models tab separates these rather than presenting one list, so it is clear which slot you are filling when you activate something.

Running one prompt across several models

Choosing between two models by reading benchmarks is guesswork, because the question that matters is which one is better on your code. Model comparison is a Pro feature.

/compare write a debounce hook in TypeScript

Pick two to five installed models and Onyx Code assembles the context once and gives every model the identical prompt, so the model is the only variable. Each result arrives collapsed, with approximate tokens, wall-clock seconds and tokens per second, so you can compare speed as well as answer.

Local models run one at a time, since only one is resident in memory, so a comparison loads each in turn and takes longer than a normal chat. The upside is that it works on any machine that can run a single model. Your active model goes first to save a reload, and it is restored when the run finishes. Any result can be kept into the chat, or promoted to your active model.

Pointing at Ollama, LM Studio or another server

If you already run an inference server, Onyx Code will use it instead of the bundled engine. Connecting an external server is a Pro feature; the bundled engine, local models and bring-your-own cloud keys all work on Free.

Go to the Models tab and choose Connect to External Server, then point it at any OpenAI-compatible endpoint, such as http://localhost:11434 for Ollama or http://localhost:1234 for LM Studio.

Nothing is probed in the background. The endpoint is contacted when you open that section or press Connect, and at no other time, which matters if you are being asked to account for every outbound connection the editor makes.

Only you choose the server.The server and the active model live in your own user settings, set from the AI Settings panel; a project’s .vscode/settings.jsoncannot set them, so a repository you open cannot point chat at a server of its choosing. An external model’s card shows Disconnect as soon as you activate it.

The context window shown is the one the server runs with.The chat input reads it from the server: a llama.cpp server’s own size, LM Studio’s loaded context and vLLM’s limit. For Ollama it is at most 32K, the most Onyx Code asks Ollama to hold, so a long chat is summarized before Ollama would cut it.

{
  // Bundled engine ports
  "onyxcode.ai.port": 12345,
  "onyxcode.ai.embeddingPort": 12346,
  // "auto" / "gpu" / "cpu"
  "onyxcode.ai.inferenceBackend": "auto",
  // Advanced: GPU offload layers, or "auto"
  "onyxcode.ai.gpuLayers": "auto"
}