How to run AI locally: setups, models and which combinations work
Benjamin Spinola, Founder
Published Updated 34 minutes read
Artificial intelligence (AI) does not have to run on a provider's servers: open models can be downloaded and run on a computer that you own. This guide explains how to run AI locally as of October 2026: what running a model involves and what has to run, three hardware setups, the open models by size and which of them run well on which machine, followed by the first steps, the assessment of results and the costs. The page about local AI explains what local AI is and when it is worth the cost.
Inference and training: what running AI locally means
Inference: using a finished model to answer
A language model is a large file of numbers, called parameters or weights, that its maker calculated in advance from a vast amount of text. Using that finished file to answer a request is called inference, which is what running AI locally almost always means.
Inference has two phases, both measured in tokens per second, a token being a word or part of a word. Reading the input, called prompt processing, depends mainly on computing power. Writing the answer, called generation, depends mainly on how quickly the machine moves the weights from memory to the processor, which is called memory bandwidth.
Training and fine-tuning: changing what a model knows
Training calculates the weights in the first place and needs vastly more computation: Meta reports 1.46 million hours on H100 graphics cards for Llama 3.1 8B, about 167 years on one card (our arithmetic). Fine-tuning adjusts an existing model with your own examples so that it follows a format, a style or a task more closely.
Changing every weight needs at least 16 bytes of memory per parameter for the weights and the figures that training keeps about each of them, according to Microsoft's ZeRO paper of 2019 on training large models, which is about 128 gigabytes (GB) for 8 billion parameters before the working data of training. A single machine therefore usually trains only a small add-on, known as low-rank adaptation (LoRA), often on a copy of the model compressed to 4 bits per weight, which is called quantised low-rank adaptation (QLoRA); for 8 billion parameters, the fine-tuning tool Unsloth gives 6 GB of graphics memory as the minimum for QLoRA.
What a local machine can and cannot do
- A local machine can run open models whose files and working memory fit in its memory and whose format its software supports, and it can fine-tune models with LoRA or QLoRA.
- It cannot train a capable model from scratch: the computation that Meta reports for its largest Llama 3 model would take about 1,200 years even at the theoretical peak of NVIDIA's DGX Spark, a desktop machine described below (our arithmetic).
- It does not match the leading hosted models on every task: on BrowseComp, a test of finding hard-to-locate facts on the web, OpenAI reports 91.5 per cent for GPT-6 Astra, whereas the best open models that fit on one machine scored 64 to 68 per cent on the Steel leaderboard of 4 September 2026, as our comparison with ChatGPT sets out.
What has to run
A working setup has three layers: the model files, the program that runs them and the applications through which people use them. Most problems arise where the layers connect, so check your own tools early.
The model files
A model is distributed as files that contain its weights, downloaded from Hugging Face, the main public website for models, or from the catalogues of Ollama and LM Studio, usually in several builds that store each weight in a different number of bits. Storing weights in fewer bits, called quantisation, makes a model smaller and faster but can cost accuracy.
The size of a build can be estimated in advance: the parameters in billions, multiplied by the bits per weight and divided by eight, give gigabytes. Qwen3.8-27B, with 27.78 billion parameters, needs 55.6 GB at 16 bits, and its 4-bit builds exceed the 13.9 GB of the formula because some parts stay at higher precision: on 26 October 2026, they ranged from 14.3 GB at Unsloth to 18 GB at Ollama, and Inferact's build in NVFP4, NVIDIA's 4-bit format, was 26.4 GB.
Ollama and LM Studio also list cloud models, which run on the provider's servers; on 26 October 2026, they offered GLM-5.3-Flash and DeepSeek-V4.1-Flash only in this form.
The inference runtime (Ollama, LM Studio, llama.cpp, vLLM)
The inference runtime loads the model, performs the calculation and offers an address for requests, usually in the format of OpenAI's application programming interface (API), so that tools written for OpenAI can use a local model instead.
- Ollama is the simplest start: it installs as a desktop application on macOS and Windows or with one command on Linux, and its library listed 260 model families on 26 October 2026.
- LM Studio is a desktop application that estimates a model's memory needs before loading it; it has been free for use at work since 8 July 2025, but it is not open source.
- llama.cpp is the open-source engine beneath many desktop tools, and its own server, llama-server, processes several requests together.
- vLLM serves one model to many users, above all on NVIDIA graphics cards, and publishes ready-made software packages, known as containers, for NVIDIA's Grace Blackwell systems, including the DGX Spark, but it needs more setup.
The software around the model: memory, web search and tools
A model on its own remembers nothing between conversations and can neither search nor act. Those functions come from the software around it.
- Chat applications such as Open WebUI, AnythingLLM and Jan give local models a chat window; Open WebUI adds memory across conversations and search through uploaded documents, whereas Jan listed memory as coming soon on 26 October 2026.
- Web search is never local, because the query goes to a search service; Ollama's web search, for example, is a cloud service that requires an Ollama account.
- Tools let a model read files, query databases or run programs, usually through the Model Context Protocol (MCP). Coding agents, programs with which a model plans and carries out several steps of programming work, such as OpenCode, Cline and Aider, connect to a local model through the same OpenAI-compatible address as any other application.
Three setups for running AI locally
The three setups differ above all in memory, which decides which models fit, and in memory bandwidth, which decides how fast a model writes.
Setup 1: a regular computer
A regular computer is the laptop or desktop that you already own; in the Steam Hardware Survey of September 2026, most computers had 16 or 32 GB of memory. Macs with Apple silicon share one pool of memory between processor and graphics, for example 16 to 32 GB at 153 gigabytes per second (GB/s) in a MacBook Air with M5. A desktop graphics card such as an RTX 5060 Ti with 16 GB or an RTX 5090 with 32 GB is much faster, but only for models that fit in its own memory.
Setup 2: a dedicated high-end machine
A dedicated high-end machine is bought for AI and serves one person or several. Shared-memory machines hold large models at moderate speed: machines with AMD's Ryzen AI Max+ 395, such as the Framework Desktop, have up to 128 GB at 256 GB/s, and the Mac Studio with M5 Ultra up to 512 GB at 1.2 terabytes per second (TB/s), although Apple listed the 512 GB option as coming in late October on 26 October 2026.
Graphics workstations hold less memory per card but are much faster: NVIDIA's RTX PRO 6000 Blackwell has 96 GB at 1,792 GB/s, and four of its Max-Q editions, which are limited to a lower power draw, fit in one workstation.
Setup 3: an NVIDIA DGX Spark
The NVIDIA DGX Spark is a small desktop computer built around the GB10 chip, which combines a 20-core Arm processor and a Blackwell graphics processor with shared memory, 128 GB at 273 GB/s in the original version; it runs DGX OS, NVIDIA's version of Ubuntu Linux. Seven other makers sell machines with the same chip, six of them have offered a 64 GB version since 23 October 2026, and NVIDIA states that one 128 GB machine handles models of up to 200 billion parameters.
Its strength is memory, and its weakness is bandwidth: in LMSYS's test of October 2025, which used Ollama, gpt-oss-20b wrote 49.7 tokens per second on a DGX Spark and 205 on an RTX 5090, while llama.cpp reached 83.43 with the same model on the DGX Spark in February 2026, so figures from different software are not directly comparable. It therefore suits models of about 65 to 100 GB, which no consumer graphics card holds; the section on costs describes how to test one before buying.
The three setups side by side
| Setup | Memory | Memory bandwidth | Price | Power draw | Users served at once |
|---|---|---|---|---|---|
| Setup 1: a regular computer | Most have 16 or 32 GB of system memory and 8 or 16 GB on the graphics card. | It ranges from 153 GB/s on a MacBook Air with M5 to 1,792 GB/s on an RTX 5090. | A MacBook Air with M5 starts at $1,299. On 22 October 2026, the lowest United States price of an RTX 5060 Ti with 16 GB was $789, and in August 2026 the median price of an RTX 5090 at the retailer Newegg was about $4,700. | NVIDIA rates the RTX 5060 Ti at 180 watts (W) and the RTX 5090 at 575 W. | It is suited to one person; we found no measurement with several users. |
| Setup 2: a dedicated high-end machine | Shared-memory machines have up to 128 GB with a Ryzen AI Max+ 395 or an M5 Max and 96, 256 or 512 GB with an M5 Ultra, and a workstation has 96 GB per graphics card. | It ranges from 256 GB/s on a Ryzen AI Max+ 395 to 1,792 GB/s on an RTX PRO 6000. | A Framework Desktop with 128 GB is listed at $3,449 but out of stock, a Mac Studio with M5 Ultra and 96 GB starts at $5,499, and one RTX PRO 6000 card alone was listed at $16,000 on 12 August 2026 while out of stock. | Signal65 measured 196 W at the power socket for AMD's Ryzen AI Halo, a small desktop machine with a Ryzen AI Max+ processor, running gpt-oss-120b; Apple rates the Mac Studio at up to 480 W, and NVIDIA rates the RTX PRO 6000 at up to 600 W, or 300 W as a Max-Q edition. | It serves one person or a small team, and a graphics workstation serves more people at once. |
| Setup 3: an NVIDIA DGX Spark | It has 128 GB, or 64 GB in the smaller version. | The 128 GB version moves 273 GB/s; NVIDIA has not stated the bandwidth of the 64 GB version. | NVIDIA's own Founders Edition with 128 GB was reported at $6,950, the 64 GB version starts at $4,999, and Dell lists its 128 GB machine with the same chip at $6,593. | ServeTheHome measured 60 to 90 W in many language-model workloads, and the power supply is rated at 240 W. | vLLM calls it a target for one user or a few requests processed together and limited its own test to four requests at once. |
Open-weight models by size
Open-weight models are models whose makers publish the trained files for anyone to download. In a dense model, every parameter works on every token; in a mixture-of-experts model, the parameters are split into groups called experts, of which only a few work on each token, so the whole model must fit in memory but writes almost as fast as a much smaller one. A name such as Qwen3.5-35B-A3B means 35 billion parameters in total, of which 3 billion are active. Sizes below are 4-bit downloads from Ollama, LM Studio or Hugging Face, read on 26 October 2026.
Small models (up to about 10 billion parameters)
Small models fit on almost any current computer but are the least capable class, so they suit short, well-defined tasks such as summarising, drafting, classifying text and answering questions about images.
- Gemma 4 E2B and E4B from Google (31 March 2026) accept text, images and audio and download at 4.6 and 6.6 GB.
- Qwen3.5-4B and Qwen3.5-9B (27 February 2026) download at 3.3 and 6.6 GB and cover 201 languages.
- Ministral 3 8B from Mistral AI (2 December 2025) downloads at 6.0 GB, and Phi-4-mini from Microsoft at 2.5 GB.
- Nemotron 3.5 Bolt E4B from NVIDIA (23 October 2026) has 8 billion parameters, 4 billion of them effective, and downloads at 7.9 GB.
Medium models (about 12 to 40 billion parameters)
Medium models are the largest that fit on a graphics card with 24 or 32 GB, and their makers describe them as suited to coding, agent work and longer documents.
- Qwen3.8-27B (14 August 2026) is a dense model that Qwen describes as strong in coding and agent tasks; it downloads at 18 GB.
- Gemma 4 comes as 12B with audio input (8.0 GB), as the mixture-of-experts 26B A4B (18 GB) and as the dense 31B (20 GB).
- gpt-oss-20b from OpenAI (5 August 2025) has 3.6 billion active parameters and is published at 13.8 GB, so that it runs within 16 GB of memory.
- Muse Glimmer-30B from Meta (August 2026) is designed for coding agents on one consumer graphics card and downloads at 18 GB.
Large mixture-of-experts models (about 80 to 250 billion parameters)
Large mixture-of-experts models need about 65 to 120 GB at 4 bits: builds of up to about 100 GB suit a 128 GB machine, builds of up to about 80 GB a workstation with one 96 GB card, and larger builds need two cards. With 5 to 12 billion active parameters, they write at a usable speed there.
- gpt-oss-120b from OpenAI has 117 billion parameters, 5.1 billion of them active, and downloads at 65 GB.
- Mistral Small 4 (16 March 2026) has 119 billion parameters despite its name; neither Ollama's library nor LM Studio's catalogue listed it on 26 October 2026, and Unsloth's build on Hugging Face is 73.8 GB.
- Nemotron 3.5 Super VL from NVIDIA (15 October 2026, 120 billion parameters) reads text, images and video, and its NVFP4 build is 75.0 GB.
- Qwen3.8-Flash-Next (August 2026) stores 176 billion parameters, of which 51 billion form a lookup table of word sequences, an n-gram embedding table, that can be read from the drive as needed. Unsloth states 96 to 114 GB of memory for its 4-bit builds, counting system and graphics memory together, and 75 GB for its 1-bit build; NVIDIA's NVFP4 build is 132.7 GB.
- Mistral Medium 3.5 (80 GB at 4 bits) is dense, so all of its weights are read for every token, and by the calculation in the next section a DGX Spark writes at most about 3.4 tokens per second with it.
The largest open models, and why they do not fit on one desktop machine
The largest open models come closest to the leading hosted ones: on 28 September 2026, Artificial Analysis rated GLM-5.3-Flash above GPT-6 Astra on GDPval-AA, a rating of finished professional work (1,641 against 1,542), but below it on its Intelligence Index, a combined score of many tests (42 against 53), as our AI model benchmarks page sets out. None of them fits at 4 bits in 128 GB, and the largest exceed even the 512 GB of the biggest Mac Studio.
Each way around the limit has a cost: linking several machines multiplies the price, builds with 1 to 3 bits per weight can cost accuracy, and reading part of a model from the drive slows the first answer, as our measurement in the next section shows.
- GLM-5.3-Flash from Z.ai (26 August 2026, 320 billion parameters) is 156.8 GB in its smallest 4-bit build; Unsloth's builds with fewer bits, for example 108.7 GB at 2 bits, fit in 128 GB but run only with Unsloth's own software, because the llama.cpp change that they need was closed without being merged on 1 October 2026.
- DeepSeek-V4.1-Flash (10 September 2026, updated on 17 October 2026 as DeepSeek-V4.1-Flash-1017) has official weights of 510.3 GB; Unsloth states that its 1-bit build of the update works with 128 to 145 GB of memory, but only because a large table of the model is read from the solid-state drive.
- Muse Spark Open from Meta needs at least 700 GB at 4 bits, according to Meta, and the official weights of Kimi K3 from Moonshot AI are 1,560.9 GB.
Context length: why long documents need extra memory
The context window is the amount of text a model considers at once. For every token in it, the model keeps a working memory, the key-value cache, whose size per token is 2 multiplied by the number of layers, the number of key-value heads, the size of each head and the bytes per number; the model's configuration file lists the middle three figures. By our arithmetic, Llama 3.1 8B needs 4.19 GB for 32,000 tokens and 16.78 GB for 128,000 tokens at 16 bits; the second figure is more than three times its 4-bit model of 4.9 GB in Ollama.
Storing the cache at 8 bits halves it with a very small loss of precision, according to Ollama, whose documentation sets the default context at only 4,000 tokens below 24 gibibytes (GiB, slightly larger units than gigabytes) of graphics memory and recommends at least 64,000 tokens for agents and coding tools.
Beyond chat: speech, image and search-index models
- For speech recognition, Whisper large-v3-turbo from OpenAI (99 languages) is the established choice, while Cohere Transcribe (14 languages) had a lower average word error rate, the share of words transcribed wrongly, on the Open ASR Leaderboard on 26 October 2026: 5.42 against 7.83 per cent.
- For image generation, FLUX.2 [klein] 4B from Black Forest Labs works in about 13 GB of graphics memory, and Z-Image-Turbo from Tongyi fits within 16 GB.
- For searching your own documents, embedding models turn text into numbers that capture its meaning; EmbeddingGemma 2 from Google (6 October 2026) downloads at 304 megabytes to 1.3 GB, depending on its size.
Licences for commercial use
An open-weight model is not automatically free for every commercial use; the summaries below are shortened, as of 26 October 2026, and the licence text decides.
- The Apache 2.0 and MIT licences, which permit commercial use with few conditions, cover Gemma 4, Qwen3.5, Qwen3.8-27B, gpt-oss, Mistral Small 4, GLM-5.3-Flash and DeepSeek-V4.1-Flash.
- The Qwen Community License of Qwen3.8-Flash-Next requires a separate licence for offering the model as a service and for products made mainly for AI-assisted coding or office work, while purely internal use is exempt.
- NVIDIA releases its Nemotron 3.5 models under the OpenMDW License 1.1, which it states permits commercial use, and its NVFP4 build of Qwen3.8-Flash-Next is governed by the NVIDIA Open Model License as well as the Qwen Community License, according to the model card read on 15 October 2026.
- Mistral Medium 3.5 grants commercial rights only to companies with monthly revenue of at most 20 million United States dollars.
Which models run well on which machine
| Machine | Small models, up to about 10 billion parameters | Medium models, about 12 to 40 billion parameters | Large mixture-of-experts models, about 80 to 250 billion parameters | The largest open models, about 280 billion parameters and more |
|---|---|---|---|---|
| A laptop with 16 GB of memory | Gemma 4 E4B and Qwen3.5-9B fit and suit drafting, summarising, translation and questions about images. It's FOSS measured about 3 tokens per second for an 8-billion-parameter model on an older laptop using only its processor, and by the calculation below a MacBook Air with M5 writes at most about 23 with a 6.6 GB model, an approximate limit because the parts for images and audio are not read for every token. | Gemma 4 12B fits and handles somewhat harder writing and image tasks, while gpt-oss-20b is built for 16 GB but leaves little room for the operating system and documents. Models with 27 to 31 billion parameters do not fit. | These models do not fit, because even gpt-oss-120b needs 65 GB. | These models do not fit: even Unsloth's 2-bit build of GLM-5.3-Flash is 108.7 GB. |
| A desktop with a 16 to 32 GB graphics card | These models run fast: Llama 2 7B at 4 bits (about 3.8 GB), the small test model of the llama.cpp scoreboards, wrote about 93 tokens per second on an RTX 5060 Ti and about 300 on an RTX 5090. Current small models of 6.6 GB write more slowly, roughly in proportion to their size, which still suits interactive chat. | A 16 GB card holds gpt-oss-20b, which the llama.cpp guide measured at 111.5 tokens per second on an RTX 5060 Ti and 282.5 on an RTX 5090. Models with 27 to 31 billion parameters need 24 or 32 GB, or part of the model in system memory, which Unsloth describes for Qwen3.8-27B on 16 GB cards and which slows it down. | These models exceed the card's memory, but part of a model can stay in system memory: the llama.cpp guide to gpt-oss reports about 30 tokens per second for gpt-oss-120b on an RTX 5090 with part of its experts in system memory, at the start of a conversation. | These models do not fit. |
| A machine with 128 GB of shared memory, such as a Mac Studio with M5 Max or a Ryzen AI Max+ 395 machine | Several small models fit side by side, for example for chat, speech and document search. The small test model wrote about 54 tokens per second on Ryzen AI Max+ 300 series graphics and about 120 on an M5 Max. | These models fit even at 16 bits, but dense ones write slowly: Signal65 measured the dense Qwen3.6-27B at 13 tokens per second on AMD's Ryzen AI Halo, against 63.6 for the mixture-of-experts Qwen3.6-35B-A3B, which therefore suits interactive work better. | gpt-oss-120b, Nemotron 3.5 Super VL and Mistral Small 4 fit. Signal65 measured gpt-oss-120b at 56.2 tokens per second on the Ryzen AI Halo, which read prompts at 809 tokens per second, 0.45 times the DGX Spark's rate in the same test; this suits coding and document work for one person. We found no published speed for the other two. | GLM-5.3-Flash fits only in Unsloth's builds below 4 bits, such as 108.7 GB at 2 bits, which run only with Unsloth's own software, and Unsloth states that its 1-bit build of DeepSeek-V4.1-Flash-1017 works on 128 GB machines when a large table is read from the drive. Both can lose accuracy at so few bits, and we found no published speed. |
| A graphics workstation with 96 GB or more of video memory | These models run about as fast as on an RTX 5090: the small test model wrote about 281 tokens per second on an RTX PRO 6000 Blackwell. | These models fit even at 16 bits, and LMSYS measured gpt-oss-20b at 215 tokens per second on an RTX PRO 6000 Blackwell. | gpt-oss-120b fits on one 96 GB card with about 30 GB to spare for the context and wrote 170.62 tokens per second on an RTX PRO 6000 Max-Q in the llama.cpp guide. Qwen3.8-Flash-Next does not fit on one 96 GB card at 4 bits, because Unsloth states 96 to 114 GB for those builds, so it needs two cards. | Four RTX PRO 6000 Max-Q cards with 384 GB hold NVIDIA's 4-bit build of GLM-5.3-Flash (204.4 GB) and, by our comparison with the 305 GB that Unsloth states for its 4-bit build, DeepSeek-V4.1-Flash-1017 with a large table read from the drive; we found no published speed. |
| An NVIDIA DGX Spark with 128 GB of shared memory | These models write more slowly than on a graphics card: the small test model reached about 57 tokens per second, against about 93 on an RTX 5060 Ti. For small models alone, a graphics card is the better purchase. | These models fit even at 16 bits. The llama.cpp project measured gpt-oss-20b at 83.43 tokens per second and Qwen3-Coder-30B-A3B at 61.06, fast enough for interactive coding, and Signal65 measured the dense Qwen3.6-27B at 11.9. | This is the class the machine is built for: in the llama.cpp benchmark, gpt-oss-120b wrote 58.72 tokens per second and read prompts at 2,443.91 tokens per second, and in vLLM's test of 1 June 2026, NVIDIA's NVFP4 build of Nemotron 3 Super, the predecessor of Nemotron 3.5 Super VL, wrote 22.7 to 23.7. In our single test on a Lenovo machine with the same chip, NVIDIA's NVFP4 build of Qwen3.8-Flash-Next, with part of it read from the drive, wrote 16.5 with reasoning and multi-token prediction switched off and read new text at only about 100, as described below the table. | As on the other 128 GB machines, GLM-5.3-Flash and DeepSeek-V4.1-Flash-1017 fit only in Unsloth's builds below 4 bits, with the limits described there. NVIDIA states that two linked machines handle up to 400 billion parameters, which we have not measured. |
How we filled the matrix
Each cell answers whether 4-bit builds of the typical models fit with room for the operating system and a document, and how fast the combination ran in a published measurement or in our own, according to the sources read on 26 October 2026 and listed in the note.
Where a cell gives an estimate, it is a labelled calculation: a model reads its active weights from memory once for every token it writes, so the memory bandwidth divided by their size is an upper limit for the generation speed. Measured figures depend on the software, the settings and the length of the conversation: on a DGX Spark, gpt-oss-120b wrote 58.72 tokens per second at the start and 42.76 after 32,768 tokens in the llama.cpp benchmark, and 57.8 in Signal65's test.
What we measured ourselves on a GB10 machine
Our lab runs open models on 128 GB machines with NVIDIA's GB10 chip, the chip of the DGX Spark, made by ASUS and Lenovo. Each figure below comes from a single dated run.
- On 12 October 2026, NVIDIA's NVFP4 build of Qwen3.8-Flash-Next, served with vLLM 0.30.0 on the Lenovo machine, wrote 16.5 tokens per second over a 600-token answer, with the first token after 1.19 seconds, in a single test with two features switched off: reasoning, in which the model writes out intermediate steps before answering, and multi-token prediction, which proposes several tokens at once.
- This build, at 132.7 GB, runs on one 128 GB machine only because its n-gram embedding table is read from the drive as needed. On 14 October 2026, text the machine had not read before was processed at only about 100 tokens per second, most likely for this reason, so a prompt of 16,000 tokens waited 148 seconds for the first token, against 7.3 seconds once the text had been read.
- In our AI accounting test, which uses one task from GDPval, OpenAI's set of real professional tasks, an earlier build of Qwen3.8-Flash-Next scored 80 of 95 points on 21 September 2026, with 46 of 264 invoice-level balances wrong, and Qwen3.8-27B without reasoning scored 39, 35 and 9 points on 15 September 2026 in three attempts stopped at 30 minutes. The reviewers knew which model had produced each workbook, the review is not official GDPval grading, and one task is not a ranking.
Running your first model, step by step
On a regular computer: install Ollama or LM Studio and choose a model that fits
- Find out how much memory the computer has and whether it has a graphics card with its own memory; on a Mac, both share one pool.
- Install Ollama from ollama.com or LM Studio from lmstudio.ai; on a Mac, LM Studio requires Apple silicon.
- Choose a model whose download is clearly smaller than the free memory, such as Gemma 4 E4B with 16 GB; the command ollama run gemma4:e4b downloads and starts it.
- Raise the context length towards that of your longest real document, in Ollama's settings or with its OLLAMA_CONTEXT_LENGTH variable, because its default below 24 GiB of graphics memory is about 4,000 tokens; every token of context needs memory, so raise it only as far as the memory allows.
- Run one of your own tasks and note the wait for the first word, the writing speed and whether the answer is right.
On a dedicated high-end machine: serve one model to a team
- Choose one model for the team and leave memory for several conversations, each of which needs its own key-value cache.
- Choose a runtime that processes several requests together, such as vLLM, llama-server or LM Studio's background service llmster.
- Open the runtime to the network deliberately: Ollama listens only on the machine itself until its OLLAMA_HOST setting is changed and has no access keys of its own, so access must be controlled in front of it, for example by a firewall rule or by a web server that checks a key before passing requests on, known as a reverse proxy.
- Give the team a chat application such as Open WebUI, and test with several people sending requests at the same time.
On a DGX Spark
- Update DGX OS first; the update of October 2026 brings version 7.6.0 with NVIDIA's CUDA Toolkit 13.0.3, the software through which programs use the graphics processor.
- Follow one of NVIDIA's playbooks at build.nvidia.com/spark; NVIDIA estimates 15 to 30 minutes for setting up vLLM.
- Prefer builds in NVIDIA's NVFP4 format, which use the chip's 4-bit arithmetic.
- Keep a large model running rather than reloading it: on our GB10 machine, starting vLLM with the 132.7 GB NVFP4 build of Qwen3.8-Flash-Next took about 15 minutes on 12 October 2026, including a preparation step that runs only on the first start.
Connecting your own tools
Tools written for OpenAI can use a local model once the base address and the model name are changed; where a tool requires an access key, any value works with Ollama.
- Ollama answers at http://localhost:11434/v1 and LM Studio at http://localhost:1234/v1.
- llama-server answers at http://127.0.0.1:9931/v1 in builds since 9 October 2026 and at port 8080 in older builds and documentation, and vLLM uses port 8000.
- Claude Code uses a local model through Ollama when ANTHROPIC_BASE_URL is set to http://localhost:11434 and ANTHROPIC_AUTH_TOKEN to ollama, and Ollama recommends a context of at least 64,000 tokens for it.
When it is too slow, does not fit or forgets the start of a document
- If the model does not load or the machine becomes extremely slow, the model does not fit, so choose fewer bits, a smaller model, a shorter context or an 8-bit key-value cache.
- If writing is slow, check that the whole model is in graphics memory, or choose a mixture-of-experts model.
- If the first word is slow with long documents, reading the input is the bottleneck, so keep the same document at the start of the prompt for the runtime to reuse.
- If the model forgets the start of a document, the context window is too small, so raise the context length within what the model and the memory allow.
Judging the results on real work
A demonstration almost always looks convincing; only your own work provides a meaningful test. Decide what counts as a good answer before you see any answers, so that the assessment does not depend on how persuasive they sound.
Use a representative task from the past week, at its real length and with its real complexity, because a short, carefully crafted prompt proves nothing about a working day.
Write down first what a good result looks like: what has to be correct, what must not be invented and what has to be present. Ten to twenty examples for which you already know the right answer are more informative than a long conversation.
Count the work that follows the answer. If someone has to check and correct every output, that time is part of the cost, and a model that needs less checking can be the better choice even when it appears weaker.
Record the exact setup that you tested: the model and its version, the build and its number of bits, the serving software and the settings, including the context length. Change one thing at a time; otherwise, you will not know which change helped.
What running AI locally costs
Buying a machine versus paying for a subscription
ChatGPT Plus costs $20 and Claude Pro $20 per person per month in the United States, as of 26 October 2026, whereas a machine is paid for once and charges nothing per request. Four parts decide which is cheaper.
- The purchase price has risen during 2026 because memory chips are scarce: the market research firm TrendForce reports that the prices agreed with large buyers for conventional dynamic random-access memory, the main memory of computers, rose by 93 to 98 per cent from the previous quarter in the first quarter of 2026. The DGX Spark Founders Edition, launched at $3,999 in October 2025, was reported at $6,950 on 2 October 2026.
- Electricity depends on the draw: a draw of 100 W, 24 hours a day, uses 72 kilowatt-hours (kWh) in 30 days, and a draw of 240 W, the rating of the DGX Spark's power supply, uses 172.8 kWh, to be multiplied by the price per kilowatt-hour on your own bill.
- Staff time to install, update and repair the machine is a cost that a business case must include.
- Quality is a cost as well, because an open model that needs more checking costs time; the comparison with ChatGPT compares the published evidence task by task.
Finding out before you buy
Buying a machine in order to answer the question is the expensive approach. If the answer is negative, you are left with both the answer and the hardware.
Running your own work on a prepared machine of the same kind answers the same question for the cost of a few hours, provided that the hardware, the model and the serving software are the ones that you would buy, so ask for the exact specification of all three. Our test drive offers such a test on request through the contact form: we run a machine with the GB10 chip of the DGX Spark and an open model, and you connect your own tools to it.
If you would rather not answer this question by buying hardware first, our local AI test drive provides a machine with the NVIDIA GB10 chip of the DGX Spark and an open model, so that you can connect your own tools, run your own workload and judge the result before buying equipment. The test drive is offered on request through the contact form. If you are weighing a local setup against ChatGPT or another AI chatbot, read private-local.ai vs ChatGPT.
Questions about running AI locally
Can I find out how to run AI locally without buying hardware first?
Yes, for most of the question: a small model on your present computer shows how the software works, and the table in this guide shows which models fit which machine. Only running your own work on a machine of the kind you are considering shows whether a larger model does it well enough.
Can I run ChatGPT or Claude on my own machine?
No. Their model files are not published, so no local copy exists to install, and OpenAI's open-weight gpt-oss models are separate from the models that ChatGPT uses. Whether an open model does your task well enough shows only when the same real work runs through both.
How much memory do I need to run a model locally?
Multiply the parameters in billions by the bits per weight and divide by eight to obtain gigabytes, then add room for the context and the operating system. As of October 2026, the 4-bit builds of Qwen3.8-27B, with 27.78 billion parameters, range from 14.3 to 18 GB, while Inferact's build in NVFP4, NVIDIA's 4-bit format, is 26.4 GB.
Is a local setup private?
A system that runs on hardware you control keeps the model's work inside your own network, which is the main reason why companies want one. It does not make everything secure by itself, because backups, logs, the tools around the model and access to the machine all still matter. Ollama's web search and cloud models, for example, are online services that send what you enter to Ollama's servers.
Can I run AI locally on a laptop?
Yes. A laptop with 16 GB of memory holds models of up to about 10 billion parameters, which suit drafting, summarising and questions about images, but the speed depends on the memory: It's FOSS measured about 3 tokens per second for an 8-billion-parameter model on an older laptop using only its processor (15 May 2026). The leading hosted models remain stronger for difficult work.
Which open model fits in 16 GB of memory?
As of October 2026, the 4-bit builds of Gemma 4 E4B (6.6 GB), Qwen3.5-9B (6.6 GB), Ministral 3 8B (6.0 GB) and Gemma 4 12B (8.0 GB) fit with room for a document. gpt-oss-20b (13.8 GB) is designed for 16 GB but leaves little room when that is all the memory.
What can an NVIDIA DGX Spark run?
One DGX Spark, with 128 GB of shared memory, runs the small and medium open models named in this guide and mixture-of-experts models of about 120 billion parameters at 4 bits, such as gpt-oss-120b at 58.72 tokens per second (llama.cpp, February 2026). The largest open models run only in builds below 4 bits that need special software or read part of the model from the drive, or not at all.
Can I fine-tune a model on my own machine?
Yes, with methods that train only a small add-on, known as low-rank adaptation (LoRA), or its variant for models compressed to 4 bits, quantised low-rank adaptation (QLoRA); NVIDIA states that a DGX Spark can fine-tune models of up to 70 billion parameters. Training a capable model from scratch needs far more computation than one machine provides.
Find out whether your work suits a local setup
Choose a setup, run a representative task and judge whether the result is good enough. If the result shows that you should not buy a machine, the test has answered the question just as well.
Request an endpoint