My small homelab is slowly turning into an AIOps / local LLM lab

My small homelab is slowly turning into an AIOps / local LLM lab

I've been building this mostly from small/refurbished hardware, with the goal of keeping each machine's role simple rather than building one big server.

Current setup:

• Cerebro — Lenovo ThinkCentre M900 Tiny, i5-6500T / 8 GB Dedicated kubeadm Kubernetes control plane.
• Atlas — Dell OptiPlex 7070 Micro, i3-8100 / 16 GB General-purpose Kubernetes worker, with a 1 TB SATA SSD for persistent data.
• Maestro — Dell OptiPlex 7070 Micro, i5-9500T / 32 GB Dedicated coding-agent host. Runs my Git/dev tooling, remote coding agents and test suites independently from Kubernetes.
• Hercules — Raspberry Pi 3B / 1 GB Kept deliberately outside Kubernetes and runs Pi-hole, Unbound, Tailscale and a few Prometheus exporters. The idea is that household DNS should not depend on the cluster being healthy.

Everything is now going into a small 10-inch 8U RackMate T1 Plus. I went with boring ventilated shelves instead of custom mounts. Current plan is patch panel + switch at the top, then Hercules, Maestro, Cerebro and Atlas, leaving about 2.5U free for expansion.

The next addition is Titan, which is where things get more interesting.

I'm currently planning a used Lenovo ThinkStation P520 with 32 GB RAM and 2× RTX 3060 12 GB, giving me 24 GB aggregate VRAM. The first goal isn't to build a giant AI cluster, but a simple standalone local LLM worker that Maestro can call over an OpenAI-compatible API.

Likely first model: Qwen 3.8 27B, probably using EXL3/llama.cpp depending on what works best with the two 3060s.

The workloads I'm targeting are intentionally bounded:

• repository scouting
• test generation / validation
• failure and log triage
• simple fixes
• bounded refactors
• lightweight code review

Hard architecture/reasoning and final security/correctness review would still go to stronger cloud models.

Longer term, Titan becomes my playground for:

• llama.cpp vs vLLM vs SGLang
• GPU telemetry + LGTM observability
• NVIDIA GPU Operator / Kubernetes
• KServe
• inference benchmarking
• CUDA profiling
• multi-GPU behaviour
• eventually distributed inference / NCCL / RDMA experiments

I'm deliberately starting with one GPU node rather than immediately building a cluster. If the workloads eventually justify it, I'd like to experiment with a future DGX Spark/GB10-class successor or an AMD unified-memory system and get into proper distributed inference.

The whole lab is also managed increasingly through Ansible. One small thing I've found surprisingly useful is enabling bounded persistent systemd journals on every host, so failures survive reboots and power cuts.

It's definitely overkill for the actual services I need at home, but that's increasingly the point: the lab is becoming less about hosting things and more about learning Kubernetes, automation, observability and local/distributed AI systems.

Any obvious architectural mistakes before I add the GPU box?

I'm attaching some pictures of the process.