Run a local LLM in a VS Code editor

Three ways to run a local LLM in a VS Code-style editor

All three work. They differ in how much of your evening they cost and how much of the editor ends up local.

• An extension plus a runtime you install. Continue.dev or similar, pointed at Ollama or LM Studio. Maximum control, and you own the configuration: which model for chat, which for fill-in-the-middle, which quantisation fits your GPU, and keeping all of it current.
• VS Code’s own bring-your-own-model support. Add a local provider and use it in chat with no GitHub account and no connection. Worth knowing, and worth knowing the limit: inline completions and semantic codebase search still require an account and connectivity.
• An editor that ships the runtime. Onyx Code bundles llama.cpp, detects your hardware, and installs a model that fits. Least control, least setup, and every AI surface is local rather than just chat.

If route one sounds like a good evening, take it, and the comparison with Continue.dev is a fair account of what you get. The rest of this page is about route three.

How the bundled route works

• The engine ships inside the app. No separate server to install, start or keep running, and no port to remember.
• The model is chosen for your machine. The first-run wizard reads your RAM and GPU and recommends models that fit. When the recommendation is a large one, a small starter model installs in the foreground so chat works within about two minutes while the bigger download continues behind it.
• GPU acceleration is configured, not left to you. Metal on Apple Silicon, Vulkan on NVIDIA, AMD and Intel, with an in-app Auto, GPU or CPU switch when you want to override it.
• Roles are separated. A dedicated fill-in-the-middle model does autocomplete, a next-edit model predicts your following edit, and a chat model answers questions. Running one model for all three is the usual reason a hand-built stack feels slow.
• Your own GGUF files work. Drop one in and use it. The model browser can sort by VRAM so you can see what your card will actually hold.
• Codebase search is local too. The workspace index is built on your machine with a local embedding model, so asking about a file you have not opened does not involve a network call.

Getting to a first local completion

• Download and install. macOS, Windows and Linux builds are signed, and macOS is also available through a Homebrew tap and Windows through winget.
• Let the wizard detect your hardware. It reports what it found and recommends models that fit rather than offering you a list to guess from.
• Install the recommended model. One click. A starter model is used first if the recommendation is large, so you are not watching a progress bar to find out whether any of this works.
• Open a project and type. Autocomplete starts immediately. Ask chat a question about the codebase once the index has built.
• Optional: bring your own GGUF. Add a file you already have, or point at an external server on the Pro tier.

The documentation covers model management, the processor switch, index behaviour and every setting mentioned here.

What hardware you actually need

The honest answer is that it depends on which model you run, which is exactly why the app sizes the recommendation to the machine instead of publishing a minimum spec and hoping.

• Apple Silicon. Unified memory is the constraint that matters, and Metal acceleration is used automatically. Mid-range machines run useful code models comfortably.
• Discrete GPU on Windows or Linux. Vulkan covers NVIDIA, AMD and Intel, so there is no vendor-specific install step. Available VRAM decides which model you should hold, which is why the model browser can sort by it.
• No usable GPU. CPU inference works, with smaller models and slower chat. Autocomplete remains the most usable surface on modest hardware, which is fortunate, since it is the one most people use most.
• Intel Macs. Inference is CPU-only by design on that platform, because the GPU path produces unreliable output there. Better a slower correct answer than a fast wrong one.

Questions

Can I run a GGUF model inside VS Code?

Not in VS Code itself without a runtime. Either install an extension and a runtime such as Ollama or LM Studio, or use a VS Code-family editor that bundles one. Onyx Code bundles llama.cpp and accepts GGUF files directly.

Do I need Ollama?

Not with Onyx Code. If you already run Ollama, LM Studio or a remote inference server, connecting to it is supported on the Pro tier, so an existing setup is not wasted.

Which local model is best for coding?

The one that fits your hardware with room for context. That is why the recommendation is hardware-aware rather than a fixed name in a blog post: a model that swaps to disk is worse than a smaller model that does not.

Is local autocomplete fast enough to use all day?

Yes, on a dedicated fill-in-the-middle model with GPU acceleration and speculative decoding. It is often quicker than a cloud round trip, because the round trip is the slow part.

Will my VS Code extensions work?

Most of them, through the Open VSX registry. Microsoft’s marketplace is not licensed for forks, so this applies to every VS Code fork. The first-run wizard imports your settings and tells you by name which extensions are missing, and VSIX installation covers the rest.