Guide · Local AI

How to build a home AI server for local LLMs (2026)

By Tiny · how I pick

Updated October 2026. Picks are based on published specs and results that owners report, not my own benchmarks. I point out the trade-offs and keep prices out of the picks, because they change every week.

As an Amazon Associate I earn from qualifying purchases. Buying through these links supports the site at no extra cost to you. Full disclosure

A home AI server is a regular computer with a decent graphics card, running AI models on your own hardware like a private ChatGPT. One decision matters more than the rest: how much VRAM the graphics card has, because that decides which models you can run.

The short version:

Below are three builds by budget, each part explained, and what it costs to leave running.

Words you'll see in this guide
  • VRAM: the memory on your graphics card. AI models load into it, so more VRAM means you can run bigger models.
  • Parameters (7B, 14B, 70B): a rough measure of a model's size. "8B" means 8 billion. Bigger models are usually smarter and need more memory.
  • Quantization: shrinking a model by storing its numbers less precisely, so it fits in less memory with a small drop in quality. Think of saving a photo as a smaller JPEG. "4-bit" (Q4) is the common setting.
  • NVMe SSD: a small, fast storage drive that plugs straight into the motherboard. Much quicker than a hard drive or SD card.
  • PSU: the power supply, the box inside the case that turns wall power into power every part can use.
  • Docker: a tool that runs each app in its own sealed box, so apps don't break each other or your system.

See the full glossary

The spec that decides everything: VRAM

A local model only runs fast if it fits in the graphics card's memory. The model itself and the conversation it's holding both have to sit in VRAM. Once they spill over into system RAM, everything slows down by 5 to 20 times. Next to that, the card's raw speed barely matters. Your goal is the most usable VRAM you can afford.

VRAM rule of thumb. At the usual 4-bit setting, plan on about 0.6 GB of VRAM per billion parameters, plus a couple of GB for the conversation.
  • 8 GB: 7B and 8B models
  • 12 GB: most 13B and 14B models
  • 16 GB: the same, with real headroom
  • 24 GB: 30B-class models
  • About 40 GB: a 70B model, usually two 24 GB cards
Not sure what a card can handle? Put it into the Server Calculator.

Three builds, by budget

Pick the build that matches the models you'll actually run. Each one can grow later: start small, and swap in a bigger graphics card if the hobby sticks.

Graphics card prices jumped this year. As of October 2026, the RTX 5060 Ti and used RTX 3090 cost far more than they did last winter, so check current prices before picking a tier. My GPU guide has the details.

1
Budget AI server tower
Budget: first taste

Used tower + used RTX 3060 12GB

The cheapest way in that I'd actually recommend. Take a used desktop or a PC you already own and add a used RTX 3060 12GB. It runs 7B and 8B models comfortably and most 13B and 14B models at 4-bit. That covers a private chat assistant, help with writing, and help with code. Add a cheap NVMe SSD and 16 to 32 GB of RAM.

This is the build for finding out whether you like running AI at home before you spend real money.

Check price on Amazon
2
Mid-range AI server
Sweet spot

RTX 5060 Ti 16GB or used RTX 3090 24GB, 32GB RAM

The build I'd point most people toward. Pick one of these cards:

Pair it with 32 GB of system RAM and a 1 TB NVMe drive. That runs 14B models with long conversations and gets close to 30B-class models. You won't outgrow it in a year.

Check price on Amazon
3
High-end AI server for 70B models
Serious: 70B models

One or two used RTX 3090 24GB, 64GB RAM

For the big models. One used RTX 3090 runs 30B-class models by itself. Two of them pool to 48 GB, which is enough for a 70B model. Plan on 64 GB of system RAM, a strong power supply, and a case with good airflow.

It's more than you need for a chat assistant. It makes sense if you want the largest open models running at home.

Check price on Amazon

Also worth considering

A low-power option with no separate graphics card. Good for leaving on all the time.

The parts, one at a time

Apart from the graphics card, an AI server is an ordinary PC, so don't overspend on the rest. Here they are, most important first:

The software that makes it a private ChatGPT

The usual beginner setup in 2026 is two free programs:

Both run in Docker, work from any device on your home network, and keep your chats on your own hardware. Follow my step-by-step guide: run your own ChatGPT with Ollama and Open WebUI. If you'd rather use a desktop app, see Ollama vs LM Studio.

What it costs to run

An AI server only pulls serious power while it's writing an answer. The rest of the time it idles. A single-card build draws roughly 80 to 100 watts at idle and 400 to 500 watts while working. Leave it on around the clock with a few hours of use a day, and expect roughly $15 to $25 a month in electricity at the 2026 US average of about $0.18 per kWh. Heavy, all-day use costs more. A Mac does the same job for a few dollars a month.

Plug in your own electricity rate and hours with the home server power cost calculator before you decide to leave it running all day.

Check whether you need one at all. If you only use AI now and then, a $20 a month cloud subscription or renting a GPU by the hour costs less than building anything. Building your own makes sense when privacy matters to you, you use AI enough that a subscription adds up, or you want to own your setup. I compare the two in local AI vs Claude and ChatGPT.

What I'd do

Aim for the sweet-spot build if the card prices work for you: 16 to 24 GB of VRAM, 32 GB of RAM, and an NVMe SSD. If they don't right now, start with the budget build and upgrade the card later. Either way, you'll have a private AI server for a one-time cost instead of a monthly bill.

Your next step: run your current PC through the Server Calculator. It might already be closer than you think. Want to see how far this can go? Read inside PewDiePie's 424GB home AI server.

Quick answers

How much does it cost to build a home AI server?

A budget build around a used tower and an RTX 3060 12GB can start in the hundreds of dollars and runs 7B to 14B models. A build with a 16 to 24GB graphics card runs up to 30B-class models. A 70B-capable build with two 24GB cards costs several thousand dollars in 2026, now that used RTX 3090s sell for around $1,200 each on eBay. Prices move, so treat these as ballparks.

What is the most important part of an AI server?

The graphics card, and specifically its VRAM. A model only runs fast when it fits in graphics memory. When it spills into system RAM, it slows to a crawl. Buy the most VRAM you can justify and keep the rest of the build modest.

How much VRAM do I need to run local AI?

At the usual 4-bit setting, plan on roughly 0.6 GB of VRAM per billion parameters plus a couple of GB extra. So 8 GB runs a 7B to 8B model, 12 GB covers most 13B to 14B models, 16 GB adds room for longer chats, and 24 GB fits a 30B-class model. A 70B model needs about 40 GB, usually two 24 GB cards.

Can I build a home AI server on a Mac instead?

Yes. Apple Silicon Macs share their memory between the processor and graphics, so a Mac mini or Mac Studio with lots of memory can run models a similar-priced PC without a big graphics card can't. Macs also use far less power. The trade-off is a higher upfront cost and slightly less software support than NVIDIA cards.

Related guides

Affiliate disclosure: Some links on this page are affiliate links. If you buy through them, I may earn a small commission at no extra cost to you. As an Amazon Associate I earn from qualifying purchases. No manufacturer paid for placement, and nothing here is sponsored.