Production-Grade Bare Metal Kubernetes

I decided to give myself a challenge: set up a Kubernetes cluster with absolutely zero dependencies on the outside internet.

This is the current state of the cluster, as reported by the kubelet:

Background Context

This article assumes a few external dependencies are already in place outside the cluster. Specifically: a Git server, DNS, and S3-compatible object storage are all expected to exist and be reachable before any of the steps here begin.

This is a deliberate architectural choice. Hosting your source-of-truth Git repository or your DNS inside the very cluster that bootstraps from them creates a circular dependency that complicates recovery significantly. Keeping these concerns external makes the cluster itself more disposable and easier to rebuild from scratch if needed.

Requirements

A few core principles guide every decision in this setup:

Full autonomy. The cluster has no external authorities. No cloud-init, no external certificate authorities, no external identity providers. The only exception is IP routing, where the physical network is an unavoidable dependency. Every other trust decision is owned internally.
Monitoring is not optional. Any service introduced to the cluster ships with monitoring. This is a hard requirement, not an afterthought. If something can't be observed, it doesn't belong here.
GitOps by default. Everything that can be managed through GitOps, is. The Git repository is the source of truth. Manual intervention is the exception, not the workflow.

Pre-Setup

Before any node boots, the groundwork needs to be laid at the OS and configuration level.

Operating System

Talos Linux is the obvious choice for a Kubernetes-focused bare-metal setup. The attack surface is minimal and the operational model is clean. I created two ISOs via the Image Factory: one for the control plane nodes, and one for worker nodes.

Talos exposes a kernel parameter, talos.config, which points to a URL that the node fetches on first boot to configure itself. Here that URL points to the S3 bucket established as an external dependency. When a node powers on it immediately pulls its configuration and bootstraps without any manual intervention.

The images are also configured to pull the extensions for btrfs and iscsi-tools. These aren't needed at this stage, but are required later for PVC provisioning. Setting them here means nodes come up ready without needing a reconfiguration pass later, as with Talos 1.13 it requires a full reinstall.

Initial Node Configuration

The configuration hosted in S3 covers everything a node needs to join the cluster correctly:

certSANs, the hostnames the cluster will be addressed by. These resolve through the internal nameservers rather than any public DNS authority, using the .kube TLD which does not exist publicly. The cluster is not resolvable from outside by design.
nameservers, pointing to the internally controlled DNS, which is what makes the .kube hostnames meaningful. No external resolver is in the chain. Hostnames following the convention of <nodename>.<nodepool>.<cluster>.infra.testlab.kube automatically resolve.
Cilium prerequisites, required to run Cilium as the CNI and kube-proxy replacement are baked in from the start, so the cluster is ready to receive Cilium immediately after the initial setup phase.

Initial Setup

With nodes booted and configured, the next step is getting the cluster into a state where GitOps can take over.

Node Hostnames

The first thing Terraform handles is assigning meaningful hostnames to each node. Talos generates random hostnames by default, which are functionally fine but create unnecessary cognitive overhead when managing a multi-node cluster. Distinguishing talos-abc123 from talos-def456 in logs or during troubleshooting is friction that compounds over time. Semantic hostnames make the cluster's topology legible at a glance.

Bootstrapping

With hostnames in place, Helmfile is used to bring up two things: Cilium and ArgoCD.

Cilium is deployed as the CNI and kube-proxy replacement, using the configuration options already baked into the node config from the pre-setup phase. This is the networking foundation everything else depends on.

ArgoCD comes up alongside it, but only barely. It's the minimum viable deployment needed to get GitOps operational. Once ArgoCD is up and pointed at the Git repository, it becomes responsible for deploying and managing itself going forward, along with everything else in the cluster.

After this phase, Helmfile is done. Everything from this point is GitOps.

Secondary Setup

With ArgoCD running and GitOps in control, the first order of business is laying down the CRDs that everything else will depend on. These go in before any application-layer deployments because ArgoCD will need them to be present when it starts reconciling the rest of the cluster.

CRDs

Two sets of CRDs are installed first: Prometheus and Gateway API. Nearly every subsequent deployment references one or both, so installing these up front means ArgoCD can reconcile everything else without hitting missing CRD errors.

Harbor

Harbor is deployed as an internal container registry, backed by the same S3 bucket used for node configuration. Its primary role here is as a pull-through cache for upstream registries.

This is a practical necessity. Docker's anonymous limit of 10 pulls per hour means a cluster pulling images directly from Docker Hub will exhaust its allowance quickly. With multiple nodes pulling the same images, that ceiling is hit faster than expected.

With Harbor in place, images are pulled through once and cached internally. Subsequent pulls never leave the cluster's network boundary, which also satisfies the autonomy requirement. The cluster's ability to schedule workloads is no longer contingent on Docker Hub availability or rate limit state.

Promoting ArgoCD and Cilium

Both ArgoCD and Cilium are then handed back to ArgoCD as Application manifests, overwriting the minimal bootstrap deployments with their full configurations. ArgoCD gets HA to handle the number of deployments ahead of it; the Mimir deployment itself takes ~2GB of RAM when pulling the helm chart. Cilium gets its advanced features, like Hubble, enabled. Both are now self-managed through GitOps going forward.

Networking

Running on bare-metal means none of the networking primitives that cloud deployments take for granted are available. There is no AWS Load Balancer Controller, no ELB, and no automatic ingress provisioning.

MetalLB fills the load balancer gap, advertising service IPs directly over L2 depending. On top of that, Gateway API ingress resources are configured to handle inbound traffic routing into the cluster.

For HTTPS, cert-manager is deployed to handle certificate provisioning and renewal. With no external CA in the chain, it operates entirely against the internal PKI, consistent with the full-autonomy requirement established at the outset.

Storage

Without EBS or EFS, Longhorn is the storage layer responsible for provisioning PVCs across the cluster. It uses the iSCSI and btrfs extensions set during pre-setup, and handles replication, snapshotting, and volume scheduling across nodes.

Longhorn's default StorageClass is replicated, which is appropriate for general workloads but not everything. Two additional StorageClasses are defined to cover more specific needs:

Best-effort local: single replica, waitForFirstConsumer binding. The scheduler uses the pod's placement to influence where the volume lands, colocating storage and compute where possible. Used primarily for database pods where local I/O performance matters but strict locality isn't a hard requirement.
Strict-local: single replica, pinned to a specific node. For workloads like Kafka brokers where the volume cannot move, this provides a cleaner alternative to local-path-provisioner while staying within the Longhorn management surface.

Supporting Infrastructure

The final piece of making the cluster "ready" for observability workloads was the infrastructure needed to support it - namely Kafka and Postgres. Both of these, as stateful services, have operational lifecycles that require maintenance.

Kafka

Kafka is deployed via the Strimzi operator. Kafka brokers run on dedicated nodes with no competing workloads, giving them the CPU, memory, and I/O headroom they need without having to fight for resources.

PostgreSQL

Postgres is deployed via the CloudNativePG operator, and this one is configured more deliberately. The cluster runs five replicas rather than the typical three. The priority is availability over resource efficiency; with five replicas, the cluster can tolerate multiple simultaneous failures or a long node rebuild without requiring any human intervention to maintain quorum. For a single-human setup, that margin is worth the overhead.

Observability

The LGTM stack (Loki, Grafana, Mimir, and Tempo) was chosen primarily for low cognitive overhead. I already have articles about logs, metrics, and traces, so I won't go into detail about those here.

node-exporter and kube-state-metrics are deployed alongside it for deeper visibility. Hardware-level metrics and Kubernetes object state respectively, covering the layer below what application instrumentation typically reaches.

Grafana Alloy handles scraping, pulling metrics from ServiceMonitors and PodMonitors, node-exporter, kube-state-metrics, the kubelet, and the API server. Everything flows into Mimir.

In short, the Grafana stack was selected because of its microservice deployment method and low cognitive overhead, because I was already familiar with the architecture.

Conclusion

The result is a fully self-contained Kubernetes cluster that owns its entire operational surface. No external authorities, no cloud primitives, no implicit dependencies.

From the OS boot process through to application observability, every layer is explicit, auditable, and managed through Git.

The tradeoffs are real: what a cloud provider gives you for free here has to be built and maintained deliberately. But that's also the point: understanding what sits beneath the abstractions, and being able to own it, is what separates a cluster that's operated from one that's merely running.