Local AI Inference and Operations Architecture
A private, distributed AI research environment built around local inference, private connectivity, and AI-assisted infrastructure workflows.
Architecture Overview
Key Design Principle
Local AI is not only a model problem — it is a systems problem. Reliable operation depends on the interaction between hardware, drivers, containers, model servers, routing layers, orchestration, observability, security boundaries, and recovery workflows.

All nodes connect through a private Tailscale mesh, keeping experimental services off the public internet.
Private Connectivity and Lab Isolation
Tailscale Mesh
Trusted private access across all lab devices without public internet exposure.
Services Protected
Open WebUI, LiteLLM, OpenPlaud, Whisper, and monitoring remain off the public internet by default.
Trust Boundary
Experimental nodes like the Razer Blade 18 are reachable without being treated as public services.
Spark3 — Application and Control Services
Core Services
  • Open WebUI — local AI interaction
  • LiteLLM — model routing and OpenAI-compatible access
  • OpenPlaud + Whisper — recording ingestion and transcription
  • Prometheus + Grafana — observability
  • PostgreSQL — application data
Local inference problems often surface above the model server layer: API compatibility, prompt format, token handling, and downstream application expectations.
Spark1 and Spark2 — Local Inference Nodes
Serving Patterns Tested
vLLM OpenAI-compatible serving, Ray-based distributed inference, Gemma, GPT-OSS 20B, and Qwen-family models.
Failure Modes Observed
HTTP 200 with empty output, reasoning-token budget exhaustion, distributed instability under large prompts, and hallucination requiring route removal.
Key Lesson
A route can pass a smoke test and still fail under real agent or application traffic. Compatibility, stability, and correctness matter as much as raw throughput.
Razer Blade 18 — Experimental Inference Node
Configuration
  • RTX 5090 Laptop GPU — 24 GB VRAM
  • Local Ollama and OpenAI-compatible endpoints
  • Open WebUI, containerized tooling
  • Dedicated external model storage
  • Security testing workspace

Apparent model slowness can be a systems issue. A stale container with broken NVML state caused CPU-bound inference — restarting the container restored GPU placement and decode performance.
Orchestration, Transcription, and Security
Apple Silicon Orchestration
OpenClaw-based coordination, workflow automation, memory and checkpointing, and human-in-the-loop validation — intentionally separate from model-serving.
OpenPlaud + Whisper
PLAUD device capture → OpenPlaud on Spark3 → local Whisper STT → LLM summarization. Validate actual runtime behavior — GPU labels alone are not sufficient.
Security Research Layer
Containerized tooling behind explicit control boundaries. Scope-gated workers, adversary-emulation experiments, and authorized testing — logged and reviewed, not exposed publicly.
AI Host Nanobot Concept
Small, scoped sidecars near infrastructure components — not large LLM planning logic embedded everywhere.
Design Principles
  • Least privilege and clear scope per agent
  • eBPF as sensors and enforcement points — not planning logic
  • Controlled escalation to orchestration layer
  • Sidecar or systemd-agent model preferred

The goal is operational awareness and triage assistance, not autonomous privileged action.
Operational Lessons Learned
01
Systems, Not Benchmarks
Local AI reliability depends on the full stack: drivers, runtimes, containers, routing, and observability.
02
Test with Real Traffic
Smoke tests are insufficient. Routes must be validated under realistic agent and application workloads.
03
Correctness Over Throughput
Hallucination and instability disqualify a model from default routing regardless of speed.
04
Security and Human Oversight
Security boundaries and human-in-the-loop validation must be designed in from the start — not added later.
Architectural Direction
The emerging model is a coordinated local AI operations environment — not a monolithic system.
Private Mesh
Trusted access foundation
Spark Cluster
Application services and inference
Razer Node
Experimental staging
Apple Silicon
Durable orchestration
Scope Gates
Security and promotion criteria

The key question is no longer whether a model can run locally — it is whether a local AI system can operate reliably, safely, and usefully across real infrastructure.