VRAM capacity decides which models you can run. GPU memory bandwidth decides how fast they run. And storage, PCIe, and system RAM decide how quickly you can load and swap them. VRAM capacity drives much of the conversation, and that’s fair, since the spec limits whether a model fits at all. Zoom out to the entire inference process, from loading the model out of storage to seeing the answer stream to your screen, and capacity is just one factor in a longer chain. And that’s all assuming that you’re running an AI model that fits neatly into the VRAM of one graphics card. The moment that you deploy system memory offloading or a multi-GPU setup to accommodate an oversized model, the rest of your system becomes even more important in determining the responsiveness of your AI.
That’s why your AI PC can’t just be VRAM on a stick. In this guide, we’ll walk through the full journey from loading a model to getting an answer to your prompt, flagging which part of your PC is most likely to be the bottleneck at each stage. Then we’ll offer practical build advice for the strategies people use when they can’t fully offload a model’s weights and context window into VRAM.
Key takeaways
- Local AI speed is a pipeline problem. Each stage of inference puts pressure on a different part of your PC, and the slowest stage of the pipeline determines the speed of the experience.
- VRAM capacity decides whether a model fits at all. GPU memory bandwidth decides how fast the conversation feels, and bandwidth is the spec that many people underestimate.
- Decode, the token-by-token generation phase, is bandwidth-bound. When you choose a card, a wider memory bus almost always beats more raw FLOPS.
- A daily AI assistant runs comfortably on a single card in a regular PC. Bigger ambitions currently require one of two options: offload to system RAM, or split the model across two GPUs.
How a local AI answer is built: the seven steps
To truly understand what hardware specs matter for local AI, we need to walk down the full road that your prompts travel. Loading the model, moving it into the GPU, reading your prompt, processing it, and generating the reply all happen in sequence. Let’s take a closer look at each step, examining the performance demands of each one as we go.
1. Cold load: reading the model from storage
When you launch your AI application, it opens the model weight file on your SSD, a file that can run tens of gigabytes for a mid-sized model. The operating system uses memory mapping to expose that file to the program, and the engine begins pulling in the weights as it needs them. Here, your storage drive earns its keep, because the speed of this first read determines how quickly the application becomes usable.
For this opening step, what matters most is NVMe read bandwidth. File sizes for AI models start at a couple gigabytes and scale up quickly, so loading a model is much more like firing up a AAA game than loading up a spreadsheet. A PCIe 4.0 or 5.0 NVMe drive reading at 7,000+ MB/s pulls a 30 GB weight file into system memory in roughly 4 seconds. An older, slower SSD won’t be nearly so light on its feet.
To be clear, subsequent prompts don’t require your AI application to re-load the model from scratch. The speed of your storage drive primarily plays a role in how long it takes to go from loading the application to having a conversation. Once you reach that point, the speed of your storage device is no longer the limiting factor. If your setup involves dynamic switching between different models for different tasks, that’ll keep your storage drive busier during the process, but that’s more of a power-user strategy than a standard local AI move.
One tip for anyone who likes to run AI on the same machine that they use for everyday computing: you might be best off using a dedicated storage drive just for AI. It’s the same advice that we offer creative professionals working with large project files, and for similar reasons. It’s a best practice for preserving the lifespan of your drives, it limits the possibility of thermal throttling, and it streamlines backup routines. On top of all that, if you keep your models on a drive separate from your system drive, you’ll help make sure that your system drive doesn’t get hung up with too many read/write instructions, slowing down your desktop PC experience. Call this a meaningful quality-of-life upgrade for a multitasking AI rig. It’s not essential, but it’s cheap insurance.
2. Bus transfer: moving weights into VRAM
Once the weights are in system memory, the engine streams them across the motherboard’s PCIe lanes into GPU memory. The GPU allocates space for the weight tensors and builds the workspace buffers it needs to run. If the model fits, this step completes and the GPU is ready to work. If it doesn’t fit, the engine falls back to leaving some layers in system RAM, which brings us back to the offloading tradeoff.
PCIe bandwidth is the main limiting factor here. PCIe 5.0 x16 offers roughly twice the throughput of PCIe 4.0 x16, and that difference shows up directly in model load times. A PCIe 5.0 motherboard paired with a CPU that actually supplies those lanes makes a real difference, especially for larger models or frequent model swaps.
3. Tokenization: breaking your prompt into pieces
The engine takes your raw text and breaks it into smaller units called tokens. Depending on the model, a token might be a whole word, a syllable, or a single character. The tokenizer maps each piece to an ID number from the model’s fixed vocabulary. That’s it. The output of this step is a small list of integers.
The good news for you is that basically any functional PC in 2026 can handle this step without breaking a sweat. A modern CPU finishes this in a fraction of a second, and the workload is so light that it never shows up as a bottleneck.
4. Embedding: turning IDs into vectors
The engine then takes those token IDs and converts each one into a dense vector, a long list of floating-point numbers that places the token in a high-dimensional space where related concepts sit close together. This is what lets the model do the mathematics of “meaning.”
The token IDs travel from system memory to the GPU over the same PCIe lanes as step 2, and the resulting vectors live in VRAM, so technically PCIe bandwidth and VRAM are the specs to watch for here. But in practice, the data involved is so small compared to the model weights (kilobytes rather than gigabytes) that this step is very unlikely to be bottlenecked by any aspect of your hardware.
5. Prefill: processing your whole prompt at once
The model passes the embedded vectors through all of its transformer layers. A mechanism called self-attention calculates how strongly each token relates to every other token in your prompt. As it goes, the engine stores the intermediate attention states in a region of memory called the KV cache. This cache is what lets the model generate the next token quickly without re-reading the whole prompt.
GPU compute and VRAM capacity are your primary concerns for prefill. This is the compute-bound step of the whole pipeline, so a GPU with more raw FLOPS finishes it faster. That’s the step that determines your time to first token: how quickly the AI starts typing. VRAM capacity matters here too, because the KV cache grows with prompt length, and a large cache is one reason long-context workloads eat VRAM fast.
6. Decode: generating the answer, one token at a time
Now the model starts producing output. For each new token, it reads the model weights and the KV cache from GPU memory, computes a probability distribution over the entire vocabulary, and picks the next token. It repeats this once per token, hundreds or thousands of times for a typical answer.
This step dominates the feel of a local AI experience, and it’s bottlenecked by a spec that many people underestimate. Decode is not compute-bound. It’s memory-bandwidth-bound. The speed of this phase scales almost directly with how fast the GPU can move its own weights out of VRAM. A wide memory bus (256-bit or 512-bit) paired with fast GDDR7 can separate a 20-token-per-second experience from a 60-token-per-second one, and it’s the difference between a response that streams fluidly and one that visibly stutters.
What matters here is GPU memory bandwidth, not GPU compute. Two GPUs can have the same FLOPS and very different decode speeds, and the difference is almost always memory bandwidth. If you’re choosing between a faster card with a narrower bus and a slightly slower card with a wider bus, the wider bus almost always wins for local LLM inference.
7. Sampling, detokenization, and output streaming
The engine takes the probability distribution from step 6 and applies your sampling settings. A low temperature sharpens the distribution so the model strongly favors the highest-probability token, while a higher temperature flattens it so the model is more willing to pick less likely tokens. It picks the winner, converts that token ID back into readable text, and sends it to the screen. Then it appends the new token to the running context, extends the KV cache by one entry, and goes back to step 6. It repeats until the model produces a designated stop token.
What hardware specs matter most here? Almost nothing, actually. This loop does involve CPU work (the sampling math, the text conversion) and a small GPU-to-CPU transfer for the probability distribution, but the total data involved is tiny and the operations are cheap. Any modern CPU handles it without any strain.
What this all means for building a PC for local AI
With the whole local AI inference roadmap in mind, it’s clear that VRAM capacity is vital but hardly the only factor to consider as you build an AI PC. VRAM capacity decides whether a model fits at all, but other specs play important roles along the way. They determine things like how quickly the model loads and how rapidly it produces new tokens. How much do those things matter? That depends on what you want your AI PC to do. Here’s what to weigh beyond the VRAM figure, by setup.
Scenario #1: You want a smart, responsive local AI assistant
This is a very common use case for people getting started with local AI. They want to be able to load an AI model on their primary PC, and have it available and ready for questions, research, tracking their schedule, and any other unexpected support tasks that might come up during the day. In this world, your AI acts like an overqualified but always enthusiastic personal assistant, working to keep your virtual desk clear so that you can focus on the things that you do best.
A couple observations about running this kind of a setup. First, responsiveness is very important. You don’t want to wait forever for your model to load. Similarly, you’ll get frustrated quickly if you can’t have a reasonable conversation with your local AI, and that only happens if it can write sentences faster than you can read them. A smooth tokens-per-second rate is crucial here. Second, you need your AI to offer respectable reasoning capabilities. It may not need to engage in long-range planning or exhaustive data analysis, but it’ll need to keep up with you. Finally, you’ll be best served with a setup that’s convenient. Odds are, you’ll prefer a fairly normal PC experience: one card, one set of drivers, one computer to keep cool.
All this means that you’ll be best off in this scenario with a well-rounded modern machine. Start with a current-gen PCIe 5.0 motherboard. AMD and Intel are both reasonable choices. You won’t need to go overboard with your CPU. A mainstream chip will do just fine, like an AMD Ryzen 5, AMD Ryzen 7, Intel Core Ultra 5, or Intel Core Ultra 7 processor. We recommend two PCIe 4.0 SSDs for this PC, one as your system drive and one as your dedicated AI drive.
Of course, your graphics card is a very important part of this setup. Get one fast card with enough VRAM for your target model. A 16GB PCIe 5.0 graphics card is the mainstream choice. You should aim for a setup that allows for the whole model to stay on the card. That’ll mean that every step of the pipeline that touches the GPU, from the first fast prefill to the token-by-token decode you’re reading, runs at the card’s own speed. Nothing has to cross the slower path into system memory in the middle of a reply. Opting for a card with 32GB VRAM will open up more possibilities for the model that you want to run, but will require a larger up-front investment.
As for system memory: ordinarily we’d recommend a fairly large pool of DDR5 for a machine that does this kind of multitasking. But memory prices are inflated right now, so you might be best off getting the minimum today with plans to upgrade later. Treat this suggestion as a floor, not target: for your local AI PC, match system RAM to VRAM, and then round up. A 16GB card calls for 32GB of system RAM, which is enough to keep the bus transfer step running smoothly while leaving overhead for the operating system and your other apps. Don’t worry too much about chasing high system RAM speed here, because the model never has to live in system RAM.
Scenario #2: You want an AI capable of tackling complex tasks that require extensive reasoning
For other people, what’s most attractive about AI is intelligence. They have big ambitions for what they want their local AI PC to accomplish, so they’re looking for ways to bring the power of the latest high-parameter models into their home.
But such users have to reckon with a tough truth: those kinds of models are too big to live entirely in the VRAM of a desktop graphics card. Qwen3.8 27B, the compact end of this class, fits into a 32GB graphics card rather comfortably, but most models in this class require more elbow room.
You have two options in this scenario. (More exactly, two options that don’t require shelling out for a subscription to a cloud-based frontier model.) The first option is to rely on system RAM offloading. Here, you’ll be building a PC with gobs of system memory. We won’t pull any punches here: the performance cost of this strategy is steep. Decode, the step that sets your tokens per second, will now partly run over system memory, which is slower than a graphics card’s bus, so the model will feel significantly slower than it would if it fully ran on your card. It all comes down to bandwidth. With DDR5, you’re looking at upward of 80GB/s bandwidth with a fast, expensive kit. A high-end GPU provides 1,000+ GB/s of bandwidth. A full order of magnitude and more separates those performance figures.
But that’s the cost. If you’re willing to accept it, the move is to minimize the performance loss as much as possible, and that’s where your money should go. Invest in a modern PCIe 5.0 motherboard, get a reasonably powerful CPU and a speedy SSD, and don’t be shy about getting the fastest DDR5 your budget will permit. Get a motherboard with robust optimization features. Even with this high-end hardware, you’ll likely run this AI PC in the background, assigning it some tough projects and then turning your attention elsewhere while it works. A good card matters here, but you don’t need to invest in the biggest one. Since part of the model runs in system memory, you’re no longer paying for maximum VRAM. You can spend less on the card and more on memory.
The second option retains the responsive operation that comes from running a local AI model entirely inside VRAM, but allows for higher-capacity models. You can build a multi-GPU rig. It’s been a while since such setups were relevant in the gaming space, but multi-GPU PCs are alive and kicking in the AI era. The apps that people use for running local AI models actually have a couple options for taking advantage of the resources of two GPUs, so if your ambitions are high and you have the budget for multiple high-end graphics cards, this is your best bet.
Here’s what you’ll need. Your first concern is getting a motherboard that can handle the workload. Get a board that’s not only equipped with two PCIe 5.0 x16 slots, but can run them in an x8/x8 configuration. There should be sufficient space between the two slots so that you can install two graphics cards. Selecting relatively slim cards, like the 2.5-slot ProArt GeForce RTX 5090, can help make sure that you have the clearance to install both in one system. You will need considerable system RAM, as well. System RAM capacity must equal or exceed the total model file size to map and transfer weights into GPU VRAM during application launch. Don’t neglect your power supply. This system will draw substantial power under load. Finally, plan for exceptional cooling. High-airflow cases with carefully planned airflow pathways will help prevent thermal throttling, whether that’s for your graphics cards, CPU, motherboard VRMs, system memory, or storage drives.
Scenario #3: You want an AI that automates a wide range of simple, short tasks
Let’s explore one final scenario for a local AI PC. Lots of folks are exploring this space because they want to use AI for a wide variety of things around their home, and they don’t want to pay for this functionality by the token. We’re talking here about all kinds of everyday tasks: smart home integrations, calorie tracking, organizing the files in your NAS, maintaining your personal calendar, sorting an email inbox, tracking stock prices, and much more.
Here’s some good news. You can do all of this on very mainstream PC hardware. There are some models out there today that are both quite powerful and very efficient. We recommend 16GB graphics cards for most people running local AI models, but honestly an 8GB graphics card isn’t completely off the table here. You’ll find some lightweight models in the 7B-to-9B parameter tier that will fit into 8GB VRAM with a reasonable context window, provided that you run the models at 4-bit quantization (Q4_K_M). Qwen3.5 4B would fit very comfortably with a context window large enough for agentic AI platforms.
For such a PC, we’ll recommend a modern graphics card and a dedicated SSD for AI, but otherwise you can safely stick with wallet-friendly hardware. Opting for a last-gen motherboard that supports DDR4 memory isn’t entirely out of the question. Such a move will limit your PCIe bandwidth to PCIe 4.0 speeds, which may slow down certain stages of AI inference. However, this strategy will help you dedicate more budget to your graphics card, which may well have more of an impact on the performance of your AI PC.
Frequently asked questions
How much VRAM do I need for local AI?
It depends on the models you want to run. Compact models fit inside 8 to 16 GB, which covers most current-generation graphics cards. Qwen3.8 27B sits comfortably in a 32 GB card. If you’re after the newest high-parameter models, you’ll need to look beyond a single card, either toward system RAM offloading or a multi-GPU build.
Is 16 GB of VRAM enough?
For compact models, yes, and it’s a fine place to start. For mid-sized models with a large context window, you’ll feel the limits, because the KV cache grows with prompt length and eats VRAM fast. If your target is a model like Qwen3.8 27B with room for long contexts, 32 GB is the comfortable line.
Does system RAM matter for local AI?
Not very much in the standard setup. The model lives in VRAM, so RAM speed and capacity take a back seat. The exception is offloading, where part of the model runs in system memory. In that scenario, RAM is the bottleneck, and you should invest in fast DDR5, a PCIe 5.0 platform, and as much capacity as your budget allows.
Why is my local AI slow?
Two usual suspects. First, decode is memory-bandwidth-bound, so a card with a narrow memory bus will dribble out tokens even if its other specs look strong. Second, part of the model may have spilled into system RAM, which adds a significant speed penalty depending on your context window and model size.
What’s the difference between time to first token and tokens per second?
They come from two different steps. Time to first token is set by prefill, the compute-bound phase where the model reads your whole prompt, so raw FLOPS matter. Tokens per second is set by decode, the bandwidth-bound phase where the model writes its answer one token at a time, so memory bus width matters.
Do I need a dedicated AI PC?
Not for a daily assistant. A local AI assistant runs comfortably on a regular PC with a capable single GPU. One meaningful upgrade is a dedicated NVMe drive for your models, so AI traffic never competes with your system drive. A fully dedicated AI PC makes more sense when you’re running long background workloads alongside everything else.
Does a multi-GPU setup actually help with local AI?
Yes, and the software side supports it properly. You’ll want a motherboard with two PCIe 5.0 x16 slots that run in x8/x8 configuration, enough clearance for two cards (slimmer cards like the ProArt GeForce RTX 5090 help), system RAM that equals or exceeds the total model file size, and a power supply with real headroom.
How much storage do I need?
Model files run tens of gigabytes for mid-sized models and scale up quickly from there, so plan for at least a terabyte of NVMe space if you intend to keep several models on hand. Speed matters too: a PCIe 4.0 or 5.0 drive reading at 7,000+ MB/s loads a 30 GB model in about four seconds.
Should I buy a lot of RAM now, given memory prices?
Not necessarily, but you’ll need enough RAM for bus transfer, along with overhead for anything else you’re doing with your PC. For smooth operation, make sure that your system has more system RAM than VRAM. For instance, if you have a 16GB graphics card, you should have at least 32GB of system RAM. Memory is the one component you can grow in place later without changing anything else, so buying the minimum now and upgrading when prices or your workload demand it is the sensible play.