Local AI Inference and Operations Architecture

A private, distributed AI research environment built around local inference, private connectivity, and AI-assisted infrastructure workflows.

Architecture Overview

Key Design Principle

Local AI is not only a model problem — it is a systems problem. Reliable operation depends on the interaction between hardware, drivers, containers, model servers, routing layers, orchestration, observability, security boundaries, and recovery workflows.

Private Connectivity and Lab Isolation

Tailscale Mesh

Trusted private access across all lab devices without public internet exposure.

Services Protected

Open WebUI, LiteLLM, OpenPlaud, Whisper, and monitoring remain off the public internet by default.

Trust Boundary

Experimental nodes like the Razer Blade 18 are reachable without being treated as public services.

Spark3 — Application and Control Services

Core Services

  • Open WebUI — local AI interaction
  • LiteLLM — model routing and OpenAI-compatible access
  • OpenPlaud + Whisper — recording ingestion and transcription
  • Prometheus + Grafana — observability
  • PostgreSQL — application data

Local inference problems often surface above the model server layer: API compatibility, prompt format, token handling, and downstream application expectations.

Spark1 and Spark2 — Local Inference Nodes

Serving Patterns Tested

vLLM OpenAI-compatible serving, Ray-based distributed inference, Gemma, GPT-OSS 20B, and Qwen-family models.

Failure Modes Observed

HTTP 200 with empty output, reasoning-token budget exhaustion, distributed instability under large prompts, and hallucination requiring route removal.

Key Lesson

A route can pass a smoke test and still fail under real agent or application traffic. Compatibility, stability, and correctness matter as much as raw throughput.

Razer Blade 18 — Experimental Inference Node

Configuration

  • RTX 5090 Laptop GPU — 24 GB VRAM
  • Local Ollama and OpenAI-compatible endpoints
  • Open WebUI, containerized tooling
  • Dedicated external model storage
  • Security testing workspace

Orchestration, Transcription, and Security

Apple Silicon Orchestration

OpenClaw-based coordination, workflow automation, memory and checkpointing, and human-in-the-loop validation — intentionally separate from model-serving.

OpenPlaud + Whisper

PLAUD device capture → OpenPlaud on Spark3 → local Whisper STT → LLM summarization. Validate actual runtime behavior — GPU labels alone are not sufficient.

Security Research Layer

Containerized tooling behind explicit control boundaries. Scope-gated workers, adversary-emulation experiments, and authorized testing — logged and reviewed, not exposed publicly.

AI Host Nanobot Concept

Small, scoped sidecars near infrastructure components — not large LLM planning logic embedded everywhere.

Design Principles

  • Least privilege and clear scope per agent
  • eBPF as sensors and enforcement points — not planning logic
  • Controlled escalation to orchestration layer
  • Sidecar or systemd-agent model preferred

Operational Lessons Learned

01

Systems, Not Benchmarks

Local AI reliability depends on the full stack: drivers, runtimes, containers, routing, and observability.

02

Test with Real Traffic

Smoke tests are insufficient. Routes must be validated under realistic agent and application workloads.

03

Correctness Over Throughput

Hallucination and instability disqualify a model from default routing regardless of speed.

04

Security and Human Oversight

Security boundaries and human-in-the-loop validation must be designed in from the start — not added later.

Architectural Direction

The emerging model is a coordinated local AI operations environment — not a monolithic system.

Private Mesh

Trusted access foundation

Spark Cluster

Application services and inference

Razer Node

Experimental staging

Apple Silicon

Durable orchestration

Scope Gates

Security and promotion criteria