spinstate notes

The Same Pattern, Two Platforms: From Proxmox to Azure AVM

2026-07-11T00:00:00Z

Roles All the Way Down showed how Terraform tier and role fields feed an Ansible inventory, which feeds k3s cluster formation. That post ended at the homelab: bare-metal Proxmox, three nodes, a cluster that assembles itself from a single playbook run.

This post picks up the same architectural pattern and applies it to Azure using Azure Verified Modules (AVM). The hook is simple: the interface between Terraform and Ansible I designed on my homelab is the same interface I deploy at work. The pattern transfers. Only the providers change.

The pattern, recapped

The homelab pattern has three moving parts:

Terraform declares infrastructure with structured fields. Not just names — tier and role fields that carry intent.
Ansible consumes a derived inventory. The inventory is generated from Terraform outputs at apply time. It doesn't exist independently.
The contract between layers is explicit and machine-readable. One source of truth for what machines exist and what they are. Ansible reads from it; it doesn't maintain a parallel copy.

That's the whole pattern. It's substrate-independent. It works on Proxmox because the contract is the thing that matters, not the hypervisor underneath.

Why AVM changes things

Most Azure infrastructure-as-code content still talks about CARML (the now-deprecated module library) or raw ARM/Bicep templates written from scratch. AVM replaced both. It splits into two tiers:

Resource Modules (avm-res-*): one module per Azure resource type — virtual machines, virtual networks, storage accounts. Each ships with Well-Architected Framework defaults baked in: availability zones, Entra ID authentication, RBAC, private endpoints where applicable.
Pattern Modules (avm-ptn-*): opinionated compositions that solve a complete problem — landing zones, hub-and-spoke networking, policy-driven governance.

The shift is this: you don't write the module. You compose it. Microsoft maintains the module, keeps it aligned with WAF guidance, and you consume it. This is the Azure equivalent of writing a Terraform module for Proxmox VMs — except someone else maintains the module and it already encodes hundreds of hours of hard-won Azure opinion.

On the homelab, I wrote my own VM module because there was no one to write it for me. On Azure, AVM gives me that module for free, and it's better than what I'd write alone.

The bridge: Terraform AVM to Ansible

This is the interesting technical bit.

The AVM resource module for virtual machines (avm-res-compute-virtualmachine) provisions a VM with WAF defaults. What it does not do is configure the OS. It makes the machine exist. Making the machine what it's supposed to be is still your problem.

On the homelab, Ansible handles all of it because Proxmox doesn't have cloud-init in the way Azure does. On Azure, you split the work between cloud-init and Ansible — but the Terraform-to-Ansible bridge is identical.

The pattern: declare VMs with structured fields in Terraform, render those fields into an Ansible inventory at apply time.

module "vm" {
  source  = "Azure/avm-res-compute-virtualmachine/azurerm"
  version = "~> 0.18"

  name                = "vm-app-01"
  resource_group_name = module.rg.name
  location            = module.rg.location
  zone                = 1

  os_type             = "linux"
  sku_name            = "Standard_D2s_v5"
  image_reference    = local.ubuntu_image

  admin_username = local.admin_user
  ssh_public_key = local.ssh_pubkey

  tags = {
    tier = "app"
    role = "webserver"
  }
}

The tier and role tags are the same fields from the homelab. They're not Azure tags for billing or policy — they're the vocabulary the next layer consumes. The VM gets provisioned with WAF defaults (zone-pinned, Entra ID auth, RBAC). Then Terraform renders the inventory:

resource "local_file" "ansible_inventory" {
  filename = "${path.module}/../ansible/inventory.yml"
  content = templatefile("inventory.yml.tftpl", {
    hosts = [
      for vm in module.vm : {
        name        = vm.name
        ip_address  = vm.private_ip_address
        tier        = vm.tags.tier
        role        = vm.tags.role
      }
    ]
  })
}

Same template, same derived inventory, same single source of truth. The provider changed from bpg/proxmox to hashicorp/azurerm and the module changed from my hand-written VM module to AVM. The contract between layers didn't change at all.

Azure offers a dynamic inventory plugin (azure_rm) that queries the Azure API at playbook runtime instead of reading a static file. It's the right tool when your fleet is large and churns frequently. But for a platform where you control the Terraform and the Ansible in the same pipeline, derived inventory from Terraform state gives you one source of truth and a reproducible build. The homelab taught me that. It transfers.

cloud-init vs Ansible: when each wins

This is where Azure genuinely differs from the homelab, and the split is worth being explicit about.

cloud-init handles first-boot basics:

Package installation
User creation and SSH key injection
Hostname and network config
Any one-shot bootstrap that doesn't need to run again

Ansible handles everything after that:

Domain join (where applicable)
Monitoring agent configuration
Application stack install and config
Anything that might need to re-run or drift-correct

On the homelab, Ansible does both because Proxmox doesn't have cloud-init in the same way. On Azure, cloud-init takes the first-boot work, and Ansible picks up once the machine is up and addressable. The split keeps cloud-init small (first-boot should be fast and idempotent) and keeps Ansible authoritative for ongoing configuration state.

The mistake is treating them as competitors. They're not. cloud-init is the chassis; Ansible is the role. Same conceptual split as Terraform (chassis) and Ansible (role) on the homelab — just with cloud-init inserted as a first-boot shim.

The landing zone as platform

The AVM pattern module for landing zones (avm-ptn-alz) deploys the governance layer: management group hierarchy, policy definitions and assignments, RBAC role assignments, and the connectivity foundation. This is the Azure Landing Zone (ALZ) as code.

This is the governance layer. On the homelab, my governance layer is VLANs and deny-by-default inter-VLAN routing — the network defines what can talk to what, and everything else is least-privilege on top. On Azure, Azure Policy definitions are the equivalent: they define what can't exist, not just what can.

Policy assignments are the Azure equivalent of firewall rules. A policy that denies public IP on storage accounts is the same shape as a firewall rule that denies a VLAN from reaching the internet. Both define the boundary of the allowed. Both are enforced before the workload runs, not after.

The landing zone is the platform the product teams build on. On the homelab, that platform is the VLAN and routing fabric plus the base Ansible role every machine gets. On Azure, it's the management group hierarchy, policy assignments, and RBAC that every subscription inherits. Same concept, different substrate.

What I'd do differently

Some things translate cleanly. Some don't.

Transfers cleanly:

The Terraform-to-Ansible inventory bridge. Identical pattern, different provider. This was the biggest surprise — I expected to redesign the contract for Azure and didn't.
Role and tier as structured tags feeding downstream consumers. AVM modules support tags natively, so the vocabulary travels.
The chassis/role split. Terraform makes the machine exist; Ansible makes it what it's supposed to be. cloud-init slots in as a first-boot shim. The three-layer model holds.

Doesn't transfer:

Proxmox's vm_id as a stable identifier. Azure resources have lifecycle-independent resource IDs, so the integer IDs I used on the homelab are unnecessary. Let the provider manage identity.
Linked clones from a template. Azure uses marketplace images or custom images in a compute gallery. The template-clone pattern doesn't exist; image references replace it.
Manual IP assignment in Terraform locals. Azure allocates private IPs dynamically by default and lets you override. On the homelab I had to manage the whole IP range in code. On Azure, let the platform allocate and export what it allocated into the inventory.

Where AVM's opinions saved me from mistakes I'd already made:

Availability zones. On the homelab I have one node, so HA isn't an option. AVM VM modules default to zone-pinned deployments, which I would have had to design from scratch with raw Terraform.
Entra ID-based RBAC instead of SSH key management. AVM VM modules default to Entra ID SSH login, which is more defensible than the static-key approach on my homelab.
Private endpoints for PaaS. AVM resource modules default to private endpoints where the resource supports them. I had to build that discipline manually on the homelab; AVM hands it to you.

The pattern I designed in the basement is the pattern I deploy at work. That's not a coincidence. Platform engineering isn't about the substrate — it's about the contracts between layers. Get those right on bare metal and they hold up in the cloud.

The takeaway

The homelab and the Azure platform run on different providers, different modules, and different substrates. The thing that's the same is the contract: Terraform declares intent with structured fields, Ansible consumes a derived inventory, and the layers compose because their interface is explicit.

AVM didn't change that. It made the Terraform layer cheaper to maintain by handing me well-architected modules I'd otherwise write myself. The bridge to Ansible is the same bridge.

If you're moving from self-hosted IaC to cloud IaC and wondering whether your patterns survive the move: the ones worth keeping will. The ones that don't were probably substrate-specific all along, and AVM will tell you which is which by handing you a better default than the one you wrote.

Where have you seen a homelab pattern survive the move to a cloud platform? Let me know in the comments.

Cost as a Design Constraint: FinOps on a £0 Budget

2026-07-10T00:00:00Z

When people hear "£0 budget," they usually think "hobby project." But the Cycle Kirklees platform runs production workloads for real users on a self-hosted stack that costs nothing per month to operate. That constraint wasn't a limitation. It was the best design pressure I've ever worked under.

The constraint is the architecture

Most FinOps conversations start with a cloud bill and work backwards: "How do we reduce this?" The £0 budget inverts the question. You start with nothing and ask: "What can we build that costs nothing to run?"

That sounds restrictive, but it forces clarity about what actually matters. When you can't spin up a managed service to solve a problem, you have to understand the problem. When you can't pay for a monitoring tier, you have to know exactly what you need to monitor and why. The constraint strips away the luxury of over-provisioning.

Where the £0 maps to cloud cost patterns

What I did	Cloud equivalent	FinOps parallel
k3s on bare metal instead of AKS	Azure Kubernetes Service	Compute right-sizing: using hardware you already own instead of paying per-node
GitLab CE self-hosted instead of GitHub Actions	Managed CI/CD with per-minute billing	CI minute optimisation: no metering anxiety, no pipeline rationing
Prometheus/Grafana/Loki self-hosted instead of Azure Monitor	Managed observability with ingestion-based pricing	Observability cost control: you own the retention window, not a pricing tier
Traefik + Cloudflare free tier instead of Azure Front Door	Application Gateway / Front Door	Edge routing without per-rule billing
Self-hosted Umami instead of Google Analytics	Managed analytics with per-hit pricing	First-party analytics with no per-event cost

The pattern is the same in every row: the cost governance decision was made at architecture time, not at invoice time. That's what good FinOps looks like: not chasing savings after the bill arrives, but designing the system so the bill never arrives.

We also used Azure to back up critical membership data for fractions of a penny per day. We'd be surprised if we get billed before 2028—unless the platform grows beyond my wildest imagination.

The honest accounting is that £0 isn't free. I pay in time: mine, on evenings and weekends. The Proxmox host consumes electricity. The DNS records cost nothing because Cloudflare's free tier covers them, but that's a deliberate choice, not a gift.

The FinOps framing matters here because platform engineering teams face the same trade-off at scale. "Should we run this ourselves or pay someone to run it?" is the fundamental platform cost question. My £0 budget forced me to answer it honestly for every single component, and the answer was never "self-host because it's cheaper." It was "self-host because the cost of the managed alternative doesn't justify the cognitive load it removes, given the volume we're running."

That's the same calculation a platform team makes when deciding whether to offer a managed database as a golden path or let teams self-serve. The difference is that most teams have enough budget to avoid the question. I didn't, so I had to answer it.

What I'd do differently with budget

If I had a cloud budget for this platform, here's where I'd spend it:

AKS for the Kubernetes workloads. The operational overhead of patching k3s nodes and managing etcd backups is real. AKS removes that. The cost is justified at any scale where downtime costs more than the node fee.
Azure Monitor instead of self-hosted Prometheus. The self-hosted stack works, but Azure Monitor's correlation across services and infrastructure is genuinely better for multi-workload platforms. The ingestion pricing stings at high volume, but the operational simplicity wins.
Managed CDN with WAF. Cloudflare's free tier is excellent, but the WAF rules and bot management in the Pro tier would have saved me hours of hand-tuning rate limits and access controls.

The point isn't that self-hosting is always right. It's that the £0 constraint taught me exactly where the break-even points are, and that's the most useful thing a platform engineer can know when advising on cloud spend.

The takeaway for platform teams

FinOps isn't about cutting costs. It's about making cost-visible decisions at architecture time. When you build with a £0 budget, every decision is cost-visible by definition: there is no hidden spend to discover later. That discipline transfers directly to cloud platform work: if you can design a production platform that costs nothing, you can design one that costs the right amount.

The constraint was the best teacher. Not because it forced me to be cheap, but because it forced me to be deliberate.

Where's the break-even point you've found between self-hosting and managed services? Share it in the comments.

Service Sovereignty in an Age of Withheld Intelligence

2026-06-29T00:00:00Z

The decisions to withhold two frontier models from general consumer access have brought a long-simmering issue into sharper focus: service sovereignty. For many, this will feel like a sudden shift. It is not. It is the continuation of a tension that has been building for years across the technology landscape, one that anyone who has worked inside an enterprise IT function will recognise immediately.

Modern software consumption has trended relentlessly toward convenience. Cloud platforms, managed services, and API-driven ecosystems promised speed, scale, and reduced operational overhead. The pitch was compelling, and organisations bought into it fast, often faster than their thinking could keep up. The true costs of that decision, loss of control, vendor lock-in, price-hikes and exposure to the strategic choices of third parties, were deferred, underestimated, or simply brushed aside in the rush to modernise.

This is the enduring challenge of technology adoption: innovation and systems thinking are not natural allies. Moving quickly delivers competitive advantage; but the systems underpinning critical capabilities require deliberate design, clear ownership, and a clear-eyed view of failure modes. When those systems are externalised, so too is a portion of your agency.

In enterprise IT, this tension predates the cloud. It showed up in every outsourcing contract, every SaaS migration, every decision to trade operational control for someone else's SLA. The questions were always there: what happens when a critical service changes its terms, degrades its performance, or simply disappears? For a long time, those questions stayed theoretical. Increasingly, they are becoming operational realities, especially in a fast moving domain like genAI.

What is unique now is the expanding influence of geopolitical and state-level forces. Access to advanced technology is no longer governed by market dynamics alone. National strategies, export controls, and policy decisions are actively shaping who can use what, and under what conditions. The whims of states, once considered ideological allies, are now a material factor in business continuity planning, and very few organisations have modelled that risk or understand how to react.

This is where service sovereignty matters. At its core, it is the ability to maintain meaningful control over the tools and systems you depend on. It does not necessarily mean rejecting cloud services or external providers. It means understanding and mitigating the risks of dependency, knowing which capabilities are load-bearing, which can be replicated, and which represent single points of failure you cannot tolerate.

Self-hosting and self-provisioning with open source software are not new ideas. They predate the cloud by decades. What has changed is the urgency of treating them as deliberate strategic choices rather than a niche preference for those who distrust convenience. The tooling has matured. The models exist. The barrier is not technical, it is organisational will and the discipline to think in systems rather than just solutions.

The obvious objection is that most organisations are not in a position to run their own AI infrastructure. The compute requirements are significant, the expertise is scarce, and the operational overhead is not trivial. That objection is fair, but it is becoming less decisive by the month. Open-weights models are closing the gap with proprietary frontier systems faster than most anticipated. The capability threshold at which a self-hosted model becomes genuinely useful for enterprise workloads has already been crossed for a wide range of applications. The question is shifting from whether open-weights models are good enough to whether your organisation has the architectural foundations to take advantage of them when the time comes.

Sovereignty comes with real costs. Running your own infrastructure introduces complexity, demands expertise, and requires ongoing investment. None of that is insurmountable. With genuine systems thinking, deliberate architecture, and a willingness to sit with trade-offs rather than optimise them away, organisations can build hybrid approaches that preserve meaningful control without abandoning the benefits of external services.

The key is intentionality. Innovation without systems planning creates hidden fragility that only reveals itself at the worst moment. Every dependency is a decision about trust, resilience, and autonomy, whether you made it consciously or not.

The question is no longer whether to think about this. It is which capabilities are too critical to leave in someone else's hands, and whether you have the architecture and the will to do anything about it.

Which capability in your stack is too critical to leave in someone else's hands? I'd like to hear your view in the comments.

Observing What You've Built: Secrets, Alerting, and Blast Radius

2026-06-18T00:00:00Z

Apologies for the radio silence since the last post. I started a new job recently, which has eaten into the time I'd normally spend on this kind of writing, and the rest of my spare hours have gone into Cycle Kirklees, the cycling campaign group I volunteer for. More on that platform soon, for now back to the self-hosted platform.

When you've built infrastructure on your own hardware, in your own home office, the stakes feel different. There's no "we'll fix it Monday": the pipes are yours to maintain. And there's no SRE team to page when something breaks at 2am.

This is part 4 in a platform engineering series. The previous posts covered CI/CD infrastructure, declarative VM provisioning, and Kubernetes cluster automation. But once you have a working cluster and pipelines that deploy code, you hit two problems that feel like afterthoughts but absolutely aren't:

How do you keep secrets secret? (Even from yourself, when you're tired and just want to ship something.)
How do you know what's happening when your infrastructure is running 26 services across multiple VMs?

And underneath both: how do you expose capabilities to other people without exposing the entire system?

The Secret You Meant to Revoke

Here's the thing about hardcoding a secret in your pipeline, with a mental note to "revoke it later": you won't remember. You'll intend to. You'll think "I'll just test this locally first, then rotate the credential before I commit." Then you push at 11pm because the thing works and you're tired. Six months later, that secret is still live.

This only matters if anyone ever sees your repository. My GitLab instance isn't on the public internet, so the blast radius is limited to: anyone who compromises my self-hosted platform, anyone on my LAN with physical access, anyone who gains shell access to a Proxmox VM. All real threats, none of them zero-probability.

But the lesson isn't "you're safe, so never mind." The lesson is: don't rely on future-you to remember operational discipline. Structure the system so that the easy path and the safe path are the same path.

GitLab CI/CD Variables: The Parallel to Azure Key Vault

In my day job, we use Azure Key Vault to manage secrets across infrastructure and applications. The pattern is straightforward:

Secrets live in a centralised, encrypted store.
Access is gated by identity and RBAC.
Audit trails log every read and write.
Applications request secrets at runtime; they're never committed to code.
Secret rotation happens in one place; all consumers pick up the new value immediately.

GitLab's CI/CD variable system is the self-hosted equivalent. Not as featureful as Key Vault (no audit trail, no separate identity layer), but it solves the core problem: keep credentials out of your repository.

Here's how it works in practice.

Variable Scoping: Environment, Project, and Group

GitLab variables can be scoped at three levels:

Project-level variables apply to all CI/CD jobs in a project. Useful for values that don't change by environment.

# Available in all jobs in this project
REGISTRY_USERNAME: my_docker_registry_user

Environment-level variables apply only to jobs that explicitly deploy to that environment. This is where the separation of concerns lives.

# Only available when deploying to staging
STAGING_DATABASE_URL: postgres://staging-db:5432/app

# Only available when deploying to production
PRODUCTION_DATABASE_URL: postgres://prod-db:5432/app

Group-level variables (if you have multiple projects) are inherited by all projects in the group. Useful for shared credentials like container registry tokens.

In my Cycle Kirklees setup:

The project has a shared CI/CD token (read-only) for pulling from my internal registry.
The staging environment has credentials for pushing to the staging Docker registry and deploying to the k3s staging namespace.
The production environment has separate credentials for the production registry and k3s namespace, and it's locked behind a manual gate: no auto-deploy to prod.

stages:
  - build
  - test
  - deploy_staging
  - deploy_production

build:
  stage: build
  script:
    - docker build -t app:$CI_COMMIT_SHA .
    - docker push $REGISTRY/$DOCKER_NAMESPACE/app:$CI_COMMIT_SHA

deploy_to_staging:
  stage: deploy_staging
  environment:
    name: staging
    kubernetes:
      namespace: cyclekirklees-staging
  script:
    - helm upgrade --install cyclekirklees ./helm
      --namespace cyclekirklees-staging
      --set image.tag=$CI_COMMIT_SHA
  only:
    - main

deploy_to_production:
  stage: deploy_production
  environment:
    name: production
    kubernetes:
      namespace: cyclekirklees-prod
  script:
    - helm upgrade --install cyclekirklees ./helm
      --namespace cyclekirklees-prod
      --set image.tag=$CI_COMMIT_SHA
  when: manual  # Explicit human gate
  only:
    - main

When the deploy_to_staging job runs, GitLab injects the staging environment variables. When someone manually triggers deploy_to_production, the production variables are injected instead. The same pipeline, different secrets.

Why This Matters: Blast Radius

The philosophy here is simple: if something goes wrong, limit the damage.

If a staging credential leaks (or is accidentally logged), only staging is affected. The attacker, or the tired sysadmin, can't accidentally deploy to production. That requires a separate credential, manually gated.

In my setup:

Staging auto-deploys from main. If I push a commit, the staging environment is updated automatically. This is low-risk; staging can be broken.
Production requires manual approval. Even if CI passes, even if staging works, production won't update until I explicitly click "Deploy" in the GitLab UI.
Production credentials are environment-scoped. They're never used in the staging pipeline.

This is "least privilege" in practice: each role gets the minimum credentials it needs, and no more.

Protecting the Protected Branch

GitLab also lets you lock down which branches people can merge into, and require sign-off before changes go live. Here's how that works for Cycle Kirklees.

The main branch is what actually gets deployed to production, so it's protected: nobody can merge into it without approval from a Maintainer. Only committee members (and me) have that level of access.

Committee members add and edit website content through Sveltia, a CMS that isn't exposed to the open internet. Sveltia needs to push changes directly to a branch to save content, so I gave it its own branch, called content, that isn't locked down the same way main is.

That sounds like a hole: why leave a branch unprotected? Because changes on the content branch still have to go through an approval before they reach main. So a committee member can publish content updates any time, without waiting on me, but they can't merge straight to production. If something goes wrong, the damage is limited to draft content sitting on a branch, not a live deploy.

Everything is a trade-off. The point is: the trade-off is visible and justified.

Observability: Seeing What You Can't See

You have a cluster. Workloads are running. But:

Is memory pressure building on a worker node?
Which service is causing the most load?
When did that pod start crashing, and why?
How much network bandwidth is the database pod consuming?

Without observability, these questions have one answer: SSH into a node, run commands, hope you ask the right questions. With observability, you have a single pane of glass.

I use Prometheus and Grafana. Prometheus scrapes metrics from k3s nodes and workloads. Grafana visualises them and provides alerting.

Prometheus: Scraping the Cluster

Kubernetes exposes metrics on every node and pod via the /metrics endpoint. Prometheus polls those endpoints, stores time-series data, and makes it queryable.

The k3s setup includes the Prometheus Operator, which watches for ServiceMonitor and PrometheusRule resources and automatically wires them into the Prometheus configuration. No static config files; infrastructure as code.

A simple ServiceMonitor to scrape the Cycle Kirklees application:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cyclekirklees-app
  namespace: cyclekirklees-prod
spec:
  selector:
    matchLabels:
      app: cyclekirklees
  endpoints:
    - port: metrics
      interval: 30s

This tells Prometheus: "Every 30 seconds, scrape the metrics port on any service labelled app: cyclekirklees."

The Cycle Kirklees application (built with Node.js and Express) exports metrics via prom-client:

const prometheus = require('prom-client');

const httpRequestDuration = new prometheus.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.1, 0.5, 1, 2, 5]
});

app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    httpRequestDuration
      .labels(req.method, req.route?.path || 'unknown', res.statusCode)
      .observe(duration);
  });
  next();
});

app.get('/metrics', (req, res) => {
  res.set('Content-Type', prometheus.register.contentType);
  res.end(prometheus.register.metrics());
});

Now Prometheus knows: request latency, error rates, how many requests per route. That's data.

Grafana: Seeing the Data

Grafana queries Prometheus and renders dashboards. A simple dashboard for Cycle Kirklees might show:

Request latency (p50, p95, p99)
Error rate (5xx responses per minute)
Pod restart count
Memory and CPU usage
Network I/O

When you look at a latency spike, you can cross-reference it with an error rate spike. If they align, you know the service is struggling under load. If error rate spikes without latency, you know the issue is application logic, not resources.

Example Grafana dashboard JSON (snippet):

{
  "panels": [
    {
      "title": "Request Latency (p95)",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))"
        }
      ]
    },
    {
      "title": "Error Rate",
      "targets": [
        {
          "expr": "rate(http_requests_total{status_code=~\"5..\"}[1m])"
        }
      ]
    },
    {
      "title": "Pod Restarts (24h)",
      "targets": [
        {
          "expr": "increase(kube_pod_container_status_restarts_total[24h])"
        }
      ]
    }
  ]
}

This is the "single pane of glass." When something feels off, you start here.

Alerting: The Right Message to the Right Person at the Right Time

Observability without alerting is a dashboard you check after something breaks. Alerting means you know before it affects users.

But alerting is easy to get wrong. Fire too many alerts, and you'll ignore them. Too few, and you'll miss the real problems. The goal is: signal, not noise.

In my setup, I use four alert rules, each routed to a different channel:

Alert 1: Pod Restart Spike → Telegram (Cycle Kirklees Team)

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: pod-restarts
spec:
  groups:
    - name: pod-restarts
      rules:
        - alert: PodRestartingTooOften
          expr: |
            increase(kube_pod_container_status_restarts_total{namespace="cyclekirklees-prod"}[15m]) > 3
          for: 5m
          annotations:
            summary: "Pod {{ $labels.pod }} restarted {{ $value }} times in 15 minutes"
            message: "High restart rate detected. Check pod logs for crashes."

When this fires, a Webhook sends a Telegram message to the Cycle Kirklees committee channel. The committee doesn't need to know why it restarted, just that it did, so they can tell me to fix it.

Why Telegram? It's a group channel. Committee members see the alert, they know something's up, they can ask questions before escalating to me.

Alert 2: High Error Rate → Email (Me)

- alert: HighErrorRate
  expr: |
    rate(http_requests_total{status_code=~"5.."}[5m]) > 0.05
  for: 2m
  annotations:
    summary: "{{ $labels.service }} error rate {{ $value | humanizePercentage }}"

Email goes to my personal account. It's slower than Telegram, but it's deliberate. A 5% error rate is bad, but it's not "wake me at 2am" bad. I'll see it in the morning and investigate.

Alert 3: Node Disk Space → Personal Email + AI Agent Alert

- alert: NodeDiskPressure
  expr: |
    (1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) > 0.85
  for: 10m
  annotations:
    summary: "Node {{ $labels.node }} is {{ $value | humanizePercentage }} full"

When disk usage hits 85%, two things happen:

Email alert goes to me.
A webhook triggers an n8n flow (my local agentic automation platform) that:
- Checks which pods are consuming the most space
- Identifies old logs that could be safely rotated
- Sends me an AI-generated summary: "Disk pressure on node-2. Postgres logs are 40GB. Consider rotating logs or adding storage."

The AI agent doesn't make decisions. It gathers context and presents options. But it's better than just "disk full," because it's already done half the troubleshooting.

Alert 4: Cluster API Unavailable → Telegram + Email + Heartbeat Monitor

- alert: KubernetesAPIDown
  expr: up{job="kubernetes-apiservers"} == 0
  for: 1m
  annotations:
    summary: "Kubernetes API is unreachable"

This is a "hard stop" alert. If the API is down, the cluster is down. It goes everywhere:

Telegram (committee, so they know the platform is offline)
Email (me, so I know to check the lab)
A heartbeat monitor (externally hosted) pings my self-hosted platform every 5 minutes. If it doesn't respond, that sends a backup alert to my phone.

The heartbeat monitor is paranoia, a belt-and-suspenders approach. But the idea is sound: if my platform is down, the alerts on my platform won't fire. So I need an external check that says "hey, your platform just went dark."

Why This Structure Works

Look at the four alerts. They all solve a real problem:

Pod restarts: The committee needs to know the service is flaky.
Error rate: I need to know to check logs.
Disk pressure: I need context, not just a binary "full/not full" signal.
API down: Everyone needs to know immediately.

Each alert goes to a different channel because each has a different audience and urgency. Telegram for "the platform is acting weird"; email for "investigate when you get a chance"; AI context for "here's what's probably wrong."

No alert goes to a channel it doesn't belong in. No Telegram spam about disk space. No "alert about alerts." Just: the right information, to the right person, at the right time.

Bringing It Together: Platform Principles

This is what a self-hosted platform looks like:

Secrets are scoped and gated. Different environments get different credentials. Production requires explicit approval.
Visibility is built in. Every service exports metrics. Grafana shows the picture. Alerting catches problems early.
Blast radius is limited. A mistake in staging doesn't touch production. A misconfigured alert doesn't spam everyone.
Operational discipline is structural, not aspirational. You don't promise to remember to rotate credentials. The system makes it easy to avoid hardcoding them in the first place. You don't hope you notice a problem. Alerts tell you.

For Cycle Kirklees, this means:

Committee members can edit content in Sveltia CMS without being able to deploy to production.
Staging automatically reflects the latest content changes, so they can preview before going live.
Production deployments require my explicit approval, so I'm in control of when changes ship.
If something goes wrong, I see it in Grafana or via Telegram before users complain.
If a node fills up, I get context-aware suggestions, not just a "full" alert.

This is how you keep hosting costs down (self-hosted hardware) whilst keeping the blast radius down (staged rollout, scoped secrets, alerting). And it's how you expose capabilities to other people, in this case a volunteer-run cycling advocacy organisation, without exposing the entire system to risk.

Closing the Loop

This started as a roadmap: declarative provisioning, configuration management, a cluster, and finally the two things that make a platform trustworthy rather than just functional. Secrets that don't leak. Problems that surface before someone complains about them.

None of this is finished in the sense that software is ever finished. Credentials still need rotating. Alert thresholds still need tuning as the platform grows. But the structure is in place, and that's the point of a platform: not that nothing goes wrong, but that when something does, the damage is contained and you find out about it on your terms.

That's where this series ends. The lab runs Cycle Kirklees, this portfolio, and a handful of internal tools on infrastructure I provisioned, configured, and now watch over, end to end, with no cloud safety net and no one else to call.

Resources

This post is part of my platform engineering portfolio. See more at spinstate.dev.

How do you scope secrets and alerts on a platform with no dedicated SRE team? I'd like to hear your approach in the comments.

Roles All the Way Down

2026-02-24T00:00:00Z

This is part 2 in a 3 part series. The last post covered how I brought Terraform into my self-hosted production environment. VMs went from click-ops to declared infrastructure. Cloned from a template, assigned a static IP, given a role name, done.

But the role name was still just a name. k3s_control in a Terraform module told anyone reading the code what the machine was supposed to be. It didn't do anything about it.

This post is about closing that gap: introducing a roles structure in Terraform that Ansible could consume, so that machines could be configured for what they were supposed to do. Specifically, to stand up a Kubernetes cluster.

The Design Decision

The previous post ended with a roadmap. Phase 3 was Ansible. Phase 4 was Kubernetes. But phases on paper don't tell you how the layers connect.

The connection needed to be designed, not improvised. Specifically: Ansible needs to know which machines exist and what groups they belong to so it can run the configuration in a deliberate fasion. If that information lives in one place in Terraform and a separate hosts.yml in Ansible, you will eventually have two sources of truth. They will drift apart, and drift is the enemy of platforms.

So the Terraform VM definitions got two new fields:

k3s-cp-01 = {
  vm_id      = 210
  ip_address = "192.168.2.50/24"
  tier       = "k3s"
  role       = "control"

  cores     = 4
  memory    = 8192
  disk_size = 20
}

k3s-wk-01 = {
  vm_id      = 211
  ip_address = "192.168.2.61/24"
  tier       = "k3s"
  role       = "workers"
}

k3s-wk-02 = {
  vm_id      = 212
  ip_address = "192.168.2.62/24"
  tier       = "k3s"
  role       = "workers"
}

So we have tier and role. No, not documentation. Not naming conventions. Structured fields with a specific purpose: to be consumed by the next layer, Ansible.

The Interface Between Layers

A Terraform template renders those fields directly into an Ansible inventory:

all:
  children:
%{ for tier in distinct([for h in hosts : h.tier]) ~}
    ${tier}_hosts:
      children:

%{   for role in distinct([for h in hosts : h.role if h.tier == tier]) ~}
        ${tier}_${role}:
          hosts:
%{     for name, host in hosts ~}
%{       if host.tier == tier && host.role == role ~}
            ${name}:
              ansible_host: ${replace(host.ip_address, "/24", "")}
              ansible_user: ubuntu
              ansible_python_interpreter: /usr/bin/python3

%{       endif ~}
%{     endfor ~}
%{   endfor ~}

%{ endfor ~}

tier = "k3s" and role = "control" produce the k3s_control Ansible group. tier = "k3s" and role = "workers" produce k3s_workers. The inventory is generated at terraform apply time. It doesn't exist independently. It's derived.

There is one source of truth for what machines exist and what they are. Ansible reads from it; it doesn't maintain a parallel copy. Adding a new VM to locals.tf automatically puts it in the right Ansible group.

This is the interface between layers, and it was the part that needed to exist before Ansible could do anything useful.

What Ansible Does With It

Once the inventory was sorted, Ansible could be structured around it. The playbook runs in sequence:

- name: Apply base configuration to all servers
  hosts: all
  become: true
  roles:
    - base_linux

- name: Install K3s control plane
  hosts: k3s_control
  become: true
  roles:
    - k3s_control_plane

- name: Join K3s workers
  hosts: k3s_workers
  become: true
  roles:
    - k3s_workers

Every node gets base_linux first. The platform baseline. Then the cluster roles run against their respective groups. The same groups that came from tier and role.

base_linux is the contract every machine must satisfy: a platform user created with a known SSH key and passwordless sudo, base utilities installed, SSH password auth disabled, hostname set. Whatever else a machine will become, it starts here.

k3s_control_plane is specific: disable swap (Kubernetes requires it), install k3s with the bundled Traefik disabled (there's already a Traefik instance at the network level, so no need to duplicate ingress inside the cluster), wait for the API to become ready, read the node token and set it as a host fact.

That last step matters. The workers need the token to join the cluster:

- name: Set join token fact
  set_fact:
    k3s_join_token: ""

- name: Install K3s worker node
  shell: |
    curl -sfL https://get.k3s.io | \
    K3S_URL=https://:6443 \
    K3S_TOKEN= \
    sh -

The worker role looks up the token from the control plane's host variables and uses it to join. No manual token copying. No shared secrets file. The playbook runs sequentially (control plane first, workers after) and the cluster forms.

Cluster formation is declared. You run the playbook; the cluster assembles itself.

Why the Roles Structure Matters

The original Terraform setup could provision a VM named k3s-control. It couldn't make that VM a control plane. Naming something and making it what it's named are different operations. The roles structure closes that gap.

The fields in locals.tf aren't just metadata. They're the vocabulary that Ansible operates on. When you add a new worker to the Terraform definition, it appears in the k3s_workers Ansible group automatically, and the next playbook run will configure it, join it to the cluster, and it will be available for workloads. The work is in declaring the machine. The configuration follows.

This is what it means for infrastructure to be composable: each layer produces outputs the next layer can consume, and the contract between them is explicit and machine-readable. Terraform's job is to make VMs exist. Ansible's job is to make them what they're supposed to be.

The roles structure was the thing that made one layer's output useful to the next.

Where Things Stand

Phase 1 ✓ Declarative VM provisioning with Terraform
Phase 2 ✓ Role abstraction and resource profiles
Phase 3 ✓ Ansible configuration management: baseline and cluster roles
Phase 4 ✓ Kubernetes via k3s: control plane and workers operational
Phase 5: Observability: Prometheus, Grafana, alerting

Observability is next. A cluster that can't be inspected has the same problem as infrastructure that can't be reproduced: the knowledge is implicit and the failure modes are invisible until they aren't.

How far do you take role abstraction before it becomes its own complexity problem? Let me know in the comments.

From Click-Ops to Roles: Terraforming My Proxmox Platform

2026-02-19T00:00:00Z

This is part 1 in a 3 part series.

Not every self-hosted platform is a sandpit.

Mine sits closer to a small data centre than an experiment bench. It runs things I depend on and things others depend on.

Which changes what good infrastructure management looks like.

This Host Runs Real Services

GitLab. Containers. Monitoring. My portfolio. Community infrastructure. Internal tooling. Backups.

When I rebuild something, it matters.

For a long time, provisioning a new VM meant:

Clone template
Set CPU and RAM
Attach storage
Configure networking
Inject SSH key
Start
Hope I did not miss a setting

It worked. It was not broken.

But it was fragile.

And fragile systems do not scale cognitively.

The Constraint Set

This is not a cloud environment.

Single Proxmox node
Thin LVM storage
pfSense upstream
Flat network with segmented services
Linked clones for efficiency

No managed control plane. No autoscaling groups. No cloud safety net.

Just deliberate engineering.

Those constraints forced clarity. Every design choice has weight when there is no abstraction layer hiding it.

The Problem Was Not Speed

I can provision a VM manually in five minutes.

The issue was repetition.

The real problem was this:

Provisioning lived in my head.

The template ID. The correct bridge. The RAM profile for Docker hosts vs GitLab. Which disk size I normally assign. Every one of those decisions was correct, but none of them were written down.

That knowledge was implicit.

Implicit knowledge is operational risk.

The Mental Shift: Machines vs Roles

The turning point was subtle.

I stopped thinking in terms of VMs and started thinking in roles.

Not:

VM 200, 2 cores, 4GB RAM

But:

docker_host gitlab k3s_control monitoring

Each role carries intent:

Resource profile
Storage expectations
Network assumptions
Future configuration layering

I was no longer provisioning machines.

I was declaring responsibilities.

Encoding Intent with Terraform

I refactored into a reusable module and began expressing infrastructure declaratively.

Instead of manually creating a Docker host, I now declare it:

module "docker_host" {
  source         = "./modules/vm"
  vm_id          = 200
  name           = "docker-host"
  template_id    = local.template_id
  cores          = 2
  memory         = 4096
  disk_size      = 40
  bridge         = "vmbr0"
  ip_address     = "192.168.2.50/24"
  dns_servers    = ["192.168.2.1"]
  ssh_public_key = local.ssh_key
}

module "gitlab" {
  source         = "./modules/vm"
  vm_id          = 201
  name           = "gitlab"
  template_id    = local.template_id
  cores          = 4
  memory         = 8192
  disk_size      = 80
  bridge         = "vmbr0"
  ip_address     = "192.168.2.51/24"
  dns_servers    = ["192.168.2.1"]
  ssh_public_key = local.ssh_key
}

Two roles. Same module. Different intent. The shape of the infrastructure is visible in the code itself.

Underneath, the module handles:

Linked clone from cloud-init template
Storage mapping
Network attachment
Deterministic configuration

If I destroy and reapply, the outcome is predictable.

That predictability is the real feature.

What Changed

Provisioning time dropped, but that was not the main win.

The real gains were:

Rebuild confidence
Reduced cognitive load
Clear separation between chassis and role
Faster experimentation

I can now:

Destroy a service VM without hesitation
Iterate on templates safely
Standardise resource profiles
Spin up test roles in seconds

The infrastructure is no longer memory dependent.

It is declared.

What Went Wrong

This was not frictionless.

Along the way I hit:

Provider quirks between Telmate and BPG
Cloud-init disk attachment issues
Permission errors on thin LVM volumes
Template lifecycle confusion

The provider choice alone cost me hours. I started with the Telmate provider because most tutorials reference it, but hit issues with cloud-init disk handling and inconsistent plan diffs. Switching to the BPG provider resolved most of those problems, but meant rewriting every resource block. If you are starting fresh, save yourself the detour and go straight to BPG.

Each issue forced me deeper into how Proxmox actually works:

How disks are mapped at the block layer
What happens during clone operations
Why cloud-init ISO generation can fail
Where Terraform state becomes authoritative

I now understand my hypervisor better than I did when everything was manual.

Automation did not abstract the system away.

It made me confront it properly.

The Platform Perspective

This is still a single node.

There is no HA. No distributed storage. No control plane redundancy.

But the thinking has shifted.

Infrastructure is now:

Declarative
Role-based
Rebuildable
Evolvable

The next phase is layering configuration management and eventually Kubernetes patterns on top.

But this step mattered.

Because platform engineering is not about scale first.

It is about repeatability first.

Why This Matters Beyond My Basement

Declarative infrastructure is not exclusive to AWS or Azure.

It is a pattern of thinking.

When infrastructure intent is encoded:

Failure becomes recoverable
Change becomes deliberate
Systems become testable

I did not automate VM creation.

I automated forgetting how to create VMs.

And that is the real shift.

The Roadmap

Phase 1: Declarative VM provisioning ✓
Phase 2: Role abstraction and resource profiling ✓

Still to come:

Phase 3: Configuration layering with Ansible: roles declared in Terraform drive Ansible inventory, turning declared intent into applied configuration
Phase 4: Kubernetes integration: a k3s cluster bootstrapped from Ansible, control plane and workers resolved from the same role structure defined in Terraform
Phase 5: Observability and policy enforcement: Prometheus/Grafana stack, resource budgets, drift detection

The foundation is solid.

The work continues.

The moment you can destroy and recreate without hesitation, your relationship with your platform changes.

Mine did.

What changed for you the first time you could rebuild your infrastructure from a declarative config? Tell me in the comments.

Self-Hosted GitLab CI/CD: Building My Own Deployment Platform

2026-01-28T00:00:00Z

When I decided to build my own CI/CD infrastructure, the easy path would have been GitHub Actions or GitLab.com's free tier. Instead, I went self-hosted. Here's why:

Enterprise relevance. Many organisations choose not put their code on the public internet. Financial services, healthcare, government, they all run internal GitLab instances. I wanted hands-on experience with the patterns I'd encounter professionally, not just the managed SaaS version.

No metering anxiety. I run commits 50+ times a day when iterating on a problem. With a self-hosted setup, I never think about CI minutes, runner limits, or surprise bills. The pipeline runs when I push. Every time.

Hardware I already own. I have an enterprise R730 server running Proxmox on my self-hosted platform. It has more headroom than I'll ever need. Why pay for cloud compute when the capacity is sitting in my home office?

Isolation for peace of mind. I experiment a lot. VMs get broken, rebuilt, wiped. By keeping GitLab on its own isolated VM, I know that even if I catastrophically break my other environments, my codebase survives. Belt-and-braces: nightly backups mean I'd never lose more than a day's work anyway.

Key takeaways

Mirror enterprise patterns with self-hosted GitLab.

Keep SSL termination at Traefik; run GitLab over HTTP internally.

Isolate runners to protect GitLab responsiveness.

Read error messages literally; they usually name the root cause.

The Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                         Proxmox Host (R730)                         │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────────────────┐  │
│  │  GitLab VM  │    │  Runner VM  │    │  Other VMs              │  │
│  │  (isolated) │    │   (DinD)    │    │ (experiments, can break)│  │
│  └──────┬──────┘    └──────┬──────┘    └─────────────────────────┘  │
│         │                  │                                        │
│         └────────┬─────────┘                                        │
│                  ▼                                                  │
│         ┌───────────────┐                                           │
│         │    Traefik    │  ← SSL termination + routing              │
│         └───────┬───────┘                                           │
│                 │                                                   │
└─────────────────┼───────────────────────────────────────────────────┘
                  ▼
          https://gitlab.example.internal

Tip: Scroll horizontally on smaller screens to see the full diagram.

Key Design Decisions

GitLab on a dedicated VM: Isolation from experimental workloads that might break.
Runners on a separate VM: Resource isolation; GitLab stays responsive during heavy builds.
Docker-in-Docker executor: Container-based builds with clean environments each run.
Traefik for SSL termination: Already part of my infrastructure; no need to duplicate certificate management.
Nightly VM backups: Never lose more than a day; peace of mind for experimentation.

The Troubleshooting Journey

Setting this up wasn't a smooth path. Here's where I went wrong, and what I learned.

Problem 1: The Let's Encrypt Trap

I started with GitLab's official installation for Ubuntu:

sudo EXTERNAL_URL="https://gitlab.example.internal" apt-get install gitlab-ce

After correcting my external URL and running gitlab-ctl reconfigure, I hit this:

FATAL: RuntimeError: letsencrypt_certificate[gitlab.example.internal] had an error
DNS problem: NXDOMAIN looking up A for gitlab.example.internal

The realisation: GitLab saw my HTTPS URL and automatically tried to provision a Let's Encrypt certificate. But this is local-only infrastructure. Let's Encrypt requires public DNS validation, which doesn't apply to my self-hosted platform.

Key learning: When you specify https:// in external_url, GitLab assumes you want it to handle SSL via Let's Encrypt unless you explicitly tell it otherwise.

Problem 2: Fighting My Own Infrastructure

My initial instinct was to make GitLab handle SSL directly. But I was missing the bigger picture: I already have a perfectly good SSL termination point, Traefik.

My self-hosted platform already includes:

Traefik reverse proxy managing all external traffic
Automated Let's Encrypt certificate provisioning
Wildcard certificates for *.spinstate.dev

Trying to make GitLab handle its own SSL was duplicating functionality and fighting against my existing architecture.

The Solution: Separation of Concerns

The correct approach separates responsibilities:

Traefik handles: HTTPS, SSL termination, certificate management, routing
GitLab handles: Application logic, Git operations, internal HTTP only

GitLab configuration (/etc/gitlab/gitlab.rb):

# External URL is what users see
external_url 'http://gitlab.example.internal'

# GitLab listens on plain HTTP internally
nginx['listen_port'] = 80
nginx['listen_https'] = false

# Disable Let's Encrypt completely
letsencrypt['enable'] = false

# Tell GitLab it's behind an HTTPS proxy
nginx['proxy_set_headers'] = {
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on"
}

Traefik configuration:

http:
  routers:
    gitlab:
      rule: "Host(`gitlab.example.internal`)"
      service: gitlab-service
      entryPoints:
        - websecure
      tls:
        certResolver: letsencrypt

  services:
    gitlab-service:
      loadBalancer:
        servers:
          - url: "http://X.X.X.X:80"  # HTTP to GitLab obsfucated internal

Problem 3: Protocol Mismatch

Even after the above, I had one more mistake. My initial Traefik config had:

- url: "https://X.X.X.X:80"  # WRONG

I was telling Traefik to connect via HTTPS to a service listening on HTTP. Small typo, confusing errors.

Fixed:

- url: "http://X.X.X.X:80"  # Correct

The Request Flow

User Browser
    ↓ HTTPS
gitlab.example.internal:443
    ↓
Traefik (SSL termination)
    ↓ HTTP
GitLab nginx (port 80)
    ↓
GitLab application

What This Enables

The payoff for this setup is simple but significant:

I just git push, and it's deployed.

No more SSH-ing into servers. No more remembering which script to run, or from which directory. I push to main, the pipeline runs, and my sites update. That's it.

Currently deploying through this pipeline:

spinstate.dev: my portfolio site
A community campaign website: pro bono project

Both deploy to nginx:alpine containers. Nothing fancy. Does the job.

The real value isn't the complexity of what I'm deploying. It's that I now understand the fundamentals of CI/CD from the infrastructure up, not just the YAML, as fun as that is.

Costs and Tradeoffs

Self-hosting buys control and realism, but it isn't free. You take on patching, monitoring, backups, storage growth, and the occasional weekend fix. And if your lab goes down, your pipelines do too. For me, the trade is worth it because the learning and autonomy outweigh the extra operational overhead.

Takeaways

Understand your infrastructure layers. Don't duplicate functionality. If you already have SSL termination, use it.
Read error messages carefully. The Let's Encrypt error explicitly mentioned DNS validation failure. That should have immediately signalled the mismatch with local-only infrastructure.
Match protocols correctly. When configuring reverse proxies, the upstream URL protocol must match what the backend actually listens on.
Use proxy headers appropriately. X-Forwarded-Proto and X-Forwarded-Ssl are crucial for applications to know they're behind an HTTPS proxy.
Consult the community. The GitLab forums had threads on Traefik integration. The problem had been solved before.

Resources

What's Next

This setup works, but it's not finished. In Part 2, I'll cover the security side, specifically, secrets management. My initial approach had gaps that I've since corrected, and I'll walk through:

What I got wrong with credential handling
GitLab CI/CD variables and environment scoping
The path toward proper secrets management

Building your own CI/CD teaches you things that clicking "Enable GitHub Actions" never will. The troubleshooting was frustrating in the moment, but every error message was a lesson in how these systems actually work.

This post is part of my platform engineering portfolio. See more at spinstate.dev.

What's the hardest lesson you've learned building your own CI/CD? Share it in the comments.

Designing Business Operations That Don't Rely on Heroics

2026-01-10T00:00:00Z

Root causes don't persist because people don't care. They persist because the system rewards restoration, not prevention.

If you have ever worked in an MSP-led IT environment, this pattern will feel familiar.

The same incidents recur. The same fixes are applied. Service is restored, everyone moves on, and nothing really changes.

For a long time, I thought this meant someone was not trying hard enough. In reality, it usually means the system is working exactly as designed.

This post is about why root causes so often go unaddressed in MSP environments, and how an IT Operations Manager can fix that without becoming a hero, a bottleneck, or a burnout case.

The myth of heroic IT

Most organisations do not explicitly ask for heroic behaviour. It emerges quietly.

An incident happens. Someone stays late to fix it. The business is grateful. Normal service resumes.

The next time it happens, the same person is expected to step in again. Over time, they become the safety net. The organisation feels stable, but only because someone is absorbing risk on its behalf.

Heroic IT feels productive. It feels necessary. It is also one of the most effective ways to prevent an organisation from learning.

Why MSP environments struggle with root causes

This is not about bad suppliers or lazy engineers. It is about incentives.

SLAs reward speed, not durability

Most MSP contracts measure things like response time, resolution time, and availability. Once service is restored and the ticket is closed, the work is considered complete.

A permanent fix that prevents future incidents is often slower, riskier, and outside scope.

From the MSP's point of view, stopping at restoration is commercially rational.

Root cause work sits in a grey zone

True root cause correction usually involves change. Configuration standardisation. Documentation. Architectural cleanup. Dependency mapping.

That work is neither pure support nor clearly defined project delivery.

If it is not explicitly funded or prioritised, it becomes optional. Optional work rarely survives a busy service desk.

Improvement work is invisible

Incident resolution is visible. A ticket closes. A dashboard turns green.

Preventative work is quiet. When it succeeds, nothing happens. No outage. No alert. No thank you.

In environments driven by urgency, invisible work loses.

Everyone assumes someone else owns prevention

The business assumes IT has it covered. IT assumes the MSP will fix it. The MSP assumes the issue is accepted as part of the environment.

Nobody is being negligent. Ownership is simply unclear.

The hero trap

This is where well meaning IT managers get stuck.

They know the root cause. They know how to fix it. They know the business will not fund it easily.

So they squeeze it in. Late nights. Quiet changes. Personal effort.

The system never feels the pain, so it never changes.

Over time, the manager becomes indispensable, exhausted, and frustrated. The organisation becomes dependent without realising it.

This is not leadership. It is risk concentration.

The shift from effort to structure

Solving this does not require more energy. It requires better design.

Separate restore from improve

Incident management exists to restore service. That is its job.

Problem management exists to remove causes. That must be treated as planned, deliberate work.

When improvement is expected to happen inside BAU support, it will always lose.

Fund prevention explicitly

Even a small allocation changes behaviour.

Five to ten percent of MSP capacity. A modest monthly remediation budget. A rolling backlog of known problems.

Once prevention is paid for, it becomes legitimate.

Make recurrence visible

Do not obsess over single incidents. Track patterns.

What breaks repeatedly. How often. With what impact.

Frequency multiplied by impact tells you where to focus. It also gives you an evidence base that does not rely on anecdotes or emotion.

Force trade offs into the open

When a permanent fix is declined, record it.

Document the risk. Note the expected recurrence. Capture who accepted it.

This is not about blame. It is about clarity.

Once risk is explicit, it stops living in someone's head.

Use governance instead of heroics

Service reviews should not just report what happened. They should decide what happens next.

Which recurring issue are we eliminating this month. Which risks are we consciously accepting. What work are we choosing not to do, and why.

Cadence beats effort every time.

What changes when you do this

Things get quieter.

Incidents reduce. Escalations become rarer. The MSP starts suggesting improvements rather than reacting to noise.

IT feels boring in the best possible way.

Boring IT is predictable, resilient, and trusted. It does not rely on individuals stepping in at the last minute. It improves because the system makes improvement inevitable.

The kind of IT worth running

The goal is not perfect systems.

It is systems that get better over time without burning people out.

If your IT operation only works because someone is being heroic, it is fragile by definition.

Design the incentives. Create the feedback loops. Make the trade offs visible.

Do that, and root causes stop being a personal burden and start becoming an organisational responsibility.

That is where calm, sustainable IT actually begins.

Have you worked somewhere that rewarded heroics over prevention? How did you shift the incentives? I'd like to hear about it in the comments.

Year-end notes: self-hosted platform resolutions and the backlog I am finally tackling

2025-12-28T00:00:00Z

Late December always brings clarity. I can see the gaps in my platform backlog, and I finally want to finish the work that turns the stack into a real platform, not just a collection of services.

Observability across the stack

Grafana, Loki, and Prometheus are next. I want full coverage across the stack so I can answer questions quickly and trace issues end to end. I am done with piecemeal logs and dashboards that only cover a single host.

Agentic workflows on real events

I am building automated AI workflows that trigger on events like journal updates and platform alerts. The goal is less noise, faster diagnosis, and a clean handoff when something needs a human.

Alerts and self-healing

Next year I want to go deeper on alert tuning and self-healing patterns. It is not enough to detect failures. The platform should recover when it can, and surface the right signal when it cannot.

Certifications to focus on

AZ-305
CKA

This is my new year reset. Smaller list, higher quality, real systems work. That is the goal.

What's the one platform backlog item you're finally going to tackle this year? Tell me in the comments.

CI/CD runners on GitLab CE: a small win worth celebrating

2025-12-10T00:00:00Z

December brought a quiet but meaningful win. I finally stood up CI/CD runners in GitLab CE (yeah, Gitea didn't scratch my itch!) and integrated them into my deployment flow. Releases now move through a predictable pipeline instead of being kicked off by a script I have to remember to run by hand.

I deliberately avoided an off-the-shelf hosted service. I wanted to understand the moving parts, own the configuration end to end, and build something that fits my platform rather than bending the platform to fit a tool. It took longer, but the result feels solid.

Pipelines run cleanly. Deployments are consistent. Failures surface early and clearly. When a build breaks, tests fail and I know about it immediately. When something unexpected happens further down the line, getting back to a known good state is fast and repeatable.

The day-to-day interaction is almost trivial now:

'git commit -m "commit note" && git push origin main'

Behind that simplicity sits a set of guardrails that remove manual steps without removing intent or control. Automation replaces repetition, not judgement, and that’s where the confidence comes from.

On paper, this is a small change. In practice, it’s a shift in how the platform behaves: less manual work, less reliance on ad-hoc scripting, and a clearer, calmer path from commit to release. Small wins like this compound, and this one felt worth celebrating.

What's a small automation win that changed how your platform behaves day to day? Let me know in the comments.