Building a
Self-Hosted AI Ecosystem

Why Self-Host in 2026

Cloud AI is convenient. It's also opaque about data handling, expensive at heavy usage, and entirely dependent on someone else's infrastructure and policies. A well-built local stack gives you something different: models that run on your hardware, answer to no one, and keep every conversation off someone else's servers.

The economics shifted in 2025–2026. Quantized open-weight models running on consumer hardware now match or exceed GPT-4-class quality on most everyday tasks. The hardware is widely available. The tooling — llama.cpp, Open WebUI, Whisper, Piper — is mature and actively developed.

Ecosystem Overview

This isn't a single-machine setup — it's a distributed stack that grew to cover different use cases. Each piece solves a specific problem and they work together through standard HTTP APIs.

Primary GPU Server

The workhorse of the stack. A desktop machine with a 24GB NVIDIA GPU running llama.cpp as a system service. It handles primary chat inference, the voice backend, embeddings, and document Q&A simultaneously.

// Hardware

// LLM inference — llama.cpp

llama.cpp runs as a systemd service, exposing an OpenAI-compatible HTTP API internally and (via reverse proxy) externally with API key authentication.

cmake .. -DGGML_CUDA=ON
cmake --build . --config Release -j$(nproc)

# Key flags for 24GB VRAM + 26B MoE model
llama-server \
  --model model.gguf \
  --port 8001 \
  --n-gpu-layers 999 \
  --ctx-size 65536 \
  --jinja \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  --metrics

// Current model: Gemma 4 26B

// VRAM management on 24GB

With a 26B model, Whisper, and an embedding server all running, the 24GB card stays near capacity. Key strategies that make it work:

• KV cache quantization (-ctk/-ctv q8_0) trades a small quality loss for major VRAM savings
• Flash attention further reduces memory at long context lengths
• Parallel slots set to 1 to maintain headroom
• Fallback: reduce --ctx-size to 32768 if out-of-memory errors appear

// Model recommendations (2026, 24GB VRAM)

AMD Unified Memory Workstation

A fundamentally different approach. The AMD Ryzen AI Max+ integrates an iGPU that can access the entire system memory pool — 128GB — eliminating the VRAM ceiling that constrains discrete GPU setups.

// What this enables

Models that would require a multi-GPU server run on a single consumer device. Qwen 3.5 122B at Q4_K_M (~70GB) loads with room to spare. 128K context windows are practical. At idle, the whole system draws about 5W.

// ROCm setup

# Install ROCm — no DKMS needed on Ubuntu 26.04
sudo apt install rocm

# Build llama.cpp with HIP backend
cmake .. \
  -DGGML_HIP=ON \
  -DGGML_RPC=ON \
  -DAMDGPU_TARGETS=gfx1151
cmake --build . -j$(nproc)

# Required in systemd service file:
Environment="HSA_OVERRIDE_GFX_VERSION=11.5.1"

// Performance

// Whisper on AMD — boot sequencing fix

When the LLM and Whisper services start simultaneously at boot, they race for ROCm initialization and Whisper loses, falling back to CPU. Fix by sequencing the services:

# whisper-server.service [Unit] section:
After=network.target llama-server.service

# [Service] section:
ExecStartPre=/bin/sleep 15
# Without this, whisper runs on CPU after every reboot.

Web Interface — Open WebUI

Open WebUI is the primary chat interface. It runs in Docker, connects to llama.cpp's OpenAI-compatible API, and handles multiple users with separate conversation histories.

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "8080:8080"
    volumes:
      - /data/open-webui:/app/backend/data
    restart: always

docker compose up -d
# Update: docker compose pull && docker compose down && docker compose up -d

// Key features in use

• Integrated web search — connected to a self-hosted SearXNG instance. No external tracking.
• Model switching — swap between any model the llama.cpp server exposes, per conversation.
• Thinking mode — Gemma 4 extended reasoning, toggle per-conversation.
• Document upload — chat with PDFs and text files directly in the interface.
• User accounts — multiple users with separate histories and settings.
• OpenAI-compatible API — use Open WebUI itself as an API endpoint for external tools.

Voice Backend — Transcription & Task Extraction

A custom FastAPI service wrapping Whisper and the LLM into a multi-purpose voice processing API. One service handles transcription, AI task extraction from voice memos, the voice assistant endpoint, and a web dashboard.

// Endpoints

// Intent routing — avoiding unnecessary LLM calls

// Performance

Whisper hallucination filtering rejects transcripts with too few ASCII alpha characters before they reach the LLM — preventing garbage responses from silence or non-speech audio.

Supported audio formats: .wav .mp3 .m4a .ogg .flac .webm

Kitchen Voice Assistant — "Hey Helix"

A Raspberry Pi 4 on the kitchen counter running a fully local voice assistant. No Alexa. No Google Home. Wake word detection, speech recognition, AI responses, text-to-speech, and timer management — all self-hosted.

// Hardware

// Software stack

// Timer system

Timers are managed entirely on the Pi without server round-trips:

• Multiple simultaneous timers, numbered (reuses lowest available slot)
• On completion: TTS announcement + looping alarm until dismissed
• Dismiss: "Hey Helix, stop" (or cancel/quiet/shut up/dismiss)
• Cancel specific: "Hey Helix, cancel timer 2"
• Wake word during TTS immediately interrupts speech

// ALSA — name-based config is critical

Linux assigns audio device numbers dynamically at boot. A USB speakerphone that's card 2 today may be card 3 after adding a different USB device. Configure by device name, not number, for boot stability:

# ~/.asoundrc — use name, not card number
pcm.!default {
    type hw
    card Plus    # stable across reboots
    device 0
}

Hardware Voice Recorder

A second Raspberry Pi built as a dedicated voice recorder for capturing thoughts, tasks, and meeting notes. Press a button → record → release → automatically transcribed and task-extracted within seconds.

// Hardware

// Flow

Button held   → recording starts (LED solid)
Button release → recording stops
               → audio chunked + sent to /process endpoint
               → Whisper transcription + LLM task extraction
               → saved to dashboard database
               → LED off

Audio is written in chunks to handle long recordings without memory issues. Recovery scripts handle failed uploads and split oversized recordings into smaller segments automatically.

Document Intelligence — RAG & Private Search

A custom FastAPI + LlamaIndex service handles document Q&A across multiple independent collections. Query your own documents in natural language — manuals, PDFs, notes, research — with semantic search and LLM synthesis.

// RAG API

POST /upload/{collection}  # Upload documents
POST /query                # Natural language query
GET  /indexes              # List collections
POST /rebuild/{collection} # Rebuild after adding files
GET  /health               # Health check

// Private web search — SearXNG

A self-hosted SearXNG instance aggregates results from DuckDuckGo, Brave, Startpage, and Wikipedia. No tracking. Used by both the voice assistant and Open WebUI. Exposes a simple JSON API.

Automation, Bots & Workflow

A dedicated Proxmox VM runs supporting services — workflow automation, document RAG, and container management — isolated from the primary inference server.

// Telegram bots

Networking & Reverse Proxy

Internal ports are never exposed directly. All external traffic enters through Nginx Proxy Manager, which handles SSL termination for every domain. Nginx on the primary server sits behind NPM and handles internal routing and API key enforcement.

// Traffic flow

Internet → NPM (SSL termination, all domains)
         │
         ├── yourdomain.com      → server :80   → Nginx → static / Open WebUI
         ├── api.yourdomain.com  → server :8002  → Nginx (API key check) → LLM
         └── voice.yourdomain.com → server :8005 → voice backend

// NPM — the entry point

Nginx Proxy Manager handles all SSL certificates via Let's Encrypt and routes every public domain. The GUI makes adding proxy hosts and managing certs straightforward. All subdomains managed here — no separate Certbot needed.

# Extended timeouts required for voice endpoint (long audio uploads)
proxy_read_timeout 3600;
proxy_connect_timeout 3600;
proxy_send_timeout 3600;
# Without these: 504 Gateway Timeout on recordings over 60 seconds

// Nginx — API key enforcement

Nginx sits behind NPM on the server, handling API key validation before proxying requests to llama.cpp. NPM forwards requests; Nginx checks the key. Ensure NPM passes custom headers through unchanged.

# nginx.conf — API key enforcement via map directive
map $http_x_api_key $api_key_valid {
    default 0;
    "your-secret-key" 1;
}

# In server block for LLM port:
if ($api_key_valid = 0) { return 401; }

# Test the full chain:
curl -v https://api.yourdomain.com/v1/models \
  -H "X-API-Key: your-key" 2>&1 | grep -E "HTTP|401|200"

Security Practices

• No direct port exposure. All external access goes through the reverse proxy.
• API key authentication on all external LLM endpoints. Keys in Nginx config, not application code.
• Separate dashboard auth — session cookie with its own password, 24-hour expiry.
• Firewall on all servers — default deny, explicit allowlist only.
• Hardware firewall at the router as an additional layer.
• SSL/TLS everywhere for external services. Certbot auto-renews.
• Monitor logs. Web scanners hitting random endpoints are constant background noise — they should get 401 or 404, never 200.

Backup Strategy

BorgBackup provides local backups with deduplication, AES-256 encryption, and mountable archives.

sudo borg init --encryption=repokey /path/to/repo
sudo borg create /path/to/repo::$(date +%Y-%m-%d) /
sudo borg list /path/to/repo
sudo borg mount /path/to/repo::archive-name /mnt/restore

System Architecture

  ┌─────────────┐    ┌──────────────┐    ┌──────────────┐
  │   Internet  │    │  Kitchen Pi  │    │  Recorder Pi │
  └──────┬──────┘    │  Hey Helix   │    │  GPIO button │
         │           │  TTS · Timers│    │  audio memo  │
         │           └──────┬───────┘    └──────┬───────┘
         │                  │                   │
         └──────────┬────────┘                   │
                    └───────────────┬────────────┘
                                    │
               ┌────────────────────▼────────────────────┐
               │              Reverse Proxy               │
               │       NPM — SSL termination              │
               │       Nginx — API key auth               │
               └──┬──────────────┬─────────────┬──────────┘
                  │              │             │
       ┌──────────▼──┐      ┌────▼───┐    ┌───▼─────────────┐    ┌──────────────────────────────┐
       │  Open WebUI │      │  LLM   │    │  Voice Backend  │    │       Automation VM          │
       │  Web·Search │      │  API   │    │  Whisper · STT  │    │  OpenClaw · Hermes agent     │
       └─────────────┘      └────┬───┘    │  Dashboard      │    │  AnythingLLM · n8n           │
                                 │        └────────┬─────────┘    └──────────────┬───────────────┘
                                 │                 │                             │
                                 └─────────────────┼─────────────────────────────┘
                                                   │
                                                   ▼
               ┌───────────────────────────────────────────────────────────────┐
               │                       Inference Layer                         │
               │                                                               │
               │   ┌─────────────────────────┐   ┌─────────────────────────┐  │
               │   │       GPU Server        │   │    AMD Workstation      │  │
               │   │   Gemma 4 26B           │   │   Qwen 3.5 122B         │  │
               │   │   65K context           │   │   128K context          │  │
               │   │   fast chat · voice     │   │   long context          │  │
               │   │   embeddings · RAG      │   │   agents · reasoning    │  │
               │   └────────────┬────────────┘   └─────────────────────────┘  │
               └────────────────┼───────────────────────────────────────────────┘
                                │
                   ┌────────────┴────────────┐
                   │                         │
              ┌────▼─────┐             ┌─────▼───┐
              │Embeddings│             │ RAG API │
              │  nomic   │             │  Docs   │
              └──────────┘             └─────────┘
      

Lessons Learned