Loading theme preference
Ilia Bukin
← Back to blog

Magnitude: an open-source inference server that picks your local LLM for you

Sep 6, 2026

Running a coding agent on local models usually means making a pile of choices before you ever write code: which model, which quantization, which runtime, and then how to point your existing agent at all of it. Nothing tells you which quant fits your GPU or how fast it will actually run. Magnitude, which is sitting on GitHub Trending today, is an open source inference server built to remove most of that guesswork. It profiles your machine, recommends models that fit, then downloads, tunes, and runs them, and it is designed to slot in behind whichever coding agent you already use.1

Where it fits

Magnitude is not another agent. It calls itself an inference server, and its pitch is that it works with the agent you already have instead of replacing it. The supported harnesses are Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi, and Cline, with a built-in harness included if you do not want to bring your own.2

Profiling instead of guessing

The core idea is that your agent should not have to guess what it can run. Magnitude profiles your chip, memory, and bandwidth, recommends models that fit your hardware with an estimate of tokens per second, and handles the tuning. The README points to speculative decoding and concurrency as things it configures for your machine. It also loads models on demand and unloads them once your agent goes idle or memory fills up, so a long session does not pin weights it is not using.3 There is no fixed hardware minimum; more memory just lets it recommend larger models.4

Setup

You install it as a global npm package and run a setup command: npm i -g @magnitudedev/cli, then magnitude setup. Setup profiles your hardware, lets you choose from the recommended models, downloads your selection, and writes the harness configuration that connects your agent to it. The CLI on npm is at version 0.0.11, so this is early software.5

There is also a prompt-first path: you can ask your agent to set up local models with the Magnitude CLI, and the agent walks through the onboarding and switches itself over.6 It supports macOS and Linux, with Windows via WSL.7

Private and offline

Models, prompts, and files stay on your machine. There are no token costs or API keys, and once Magnitude and a model are downloaded it can run fully offline. You are not limited to the built-in catalog either: you can download compatible GGUF models from Hugging Face and use them.8 The project is Apache-2.0 licensed.9

Worth a look?

The repo has drawn real attention over the last week: 3.3k stars, with over 600 of them coming in today according to the trending page.10 For anyone who has been caught between running local models and keeping a favorite agent, the appeal is straightforward: the recurring failure of local-model setups is rarely that a model does not exist, and more often that managing runtimes, API formats, and agent config quietly becomes a job of its own. A server that sits behind the agent you already use takes on exactly that part.

  1. Magnitude README: "It profiles your machine, recommends the models that fit, then downloads, tunes, and runs them." https://github.com/magnitudedev/magnitude
  2. The README and docs list the same set. https://docs.magnitude.dev/introduction.md
  3. README feature list: "Knows your hardware: profiles your chip, memory, and bandwidth," "Tuned end to end: speculative decoding, concurrency, all set for your machine," and "Models on demand: loaded on request, unloaded when idle or memory fills." https://github.com/magnitudedev/magnitude
  4. README FAQ, "What hardware do I need?": "There's no fixed minimum." https://github.com/magnitudedev/magnitude
  5. README install section; the npm registry shows @magnitudedev/cli on 0.0.11 with an Apache-2.0 license. https://registry.npmjs.org/@magnitudedev/cli/latest
  6. README, "Set up with your agent." https://github.com/magnitudedev/magnitude
  7. README: "Magnitude supports macOS and Linux. Windows is supported through WSL." https://github.com/magnitudedev/magnitude
  8. README FAQ: "Can it run completely offline? Yes. Once Magnitude and a model are downloaded, no internet connection needed," and "Can I use models outside the catalog? Yes. You can download compatible GGUF models from Hugging Face and use them in Magnitude." https://github.com/magnitudedev/magnitude
  9. README: "Magnitude is licensed under the Apache License 2.0." https://github.com/magnitudedev/magnitude
  10. The repo page shows 3.3k stars; GitHub Trending (daily) lists 674 stars today. https://github.com/trending?since=daily