We build software products, and we keep their internal dev, QA, demo and proof-of-concept environments on Kubernetes: a multi-tenant .NET microservices platform, an ASP.NET Core and Angular web app, and a Node.js agent-orchestration service. All of them ran on Google Kubernetes Engine at roughly $200/month. Cloud prices for non-production environments that sit at 4% CPU are hard to justify. That had to change.

This is the full technical story of how we migrated everything to a mini PC we already owned in one day, cut our infrastructure bill to about $5/month in electricity, and kept the exact same GitOps workflow.

1. Why Did We Migrate Off GCP?

GKE cost ~$200/month against 4% CPU utilization for dev, QA and demo environments, while a mini PC we already owned sat unused — the math was too obvious to ignore.

GKE was costing us $190–200/month for that roughly 45-pod, three-workload cluster:

GCP resource Spec Monthly cost
GKE control plane Standard cluster ~$73
e2-standard-4 node 4 vCPU, 16 GB RAM ~$97
Persistent disks ~60 GB SSD ~$10
Network egress Variable ~$5–10
Total ~$200

For non-production environments, $2,400/year on infrastructure you can run on hardware you already own is hard to justify. We had a mini PC sitting unused. CPU utilization on GCP was 4%. The math was obvious.

The question was never "should we move?" but "can we do it without losing our deployment workflow?"

2. What we were running on GCP

Our GKE cluster (single-node, US region) ran a e2-standard-4 node hosting 40+ pods across three workloads:

  • Multi-tenant microservices platform — a suite of .NET microservices, an API gateway, an Angular frontend, Kafka, and ZooKeeper. Row-level security, tenant isolation, the works.
  • Web app — an ASP.NET Core API and an Angular UI.
  • Agent orchestration — a Node.js service that manages worker agents via WebSocket and REST endpoints.

Plus platform services: ArgoCD (App of Apps), nginx-ingress, cert-manager, External Secrets Operator syncing with GCP Secret Manager, Prometheus node-exporter. CI/CD via Azure Pipelines pushing images to GCP Artifact Registry and ArgoCD auto-syncing within three minutes. DNS through Cloudflare.

Not a toy setup: 17 container images, 5 namespaces, Kustomize overlays, sealed secrets, network policies, HPAs. The migration had to preserve all of it.

3. The hardware

The server that became our dev, QA and demo cluster:

  • A small-form-factor PC: 8-core Intel, plenty of RAM, NVMe
  • Network: Gigabit Ethernet
  • OS: Ubuntu LTS, bare metal

A second machine on the same LAN already ran PostgreSQL, Redis, and a few other internal services.

Total hardware cost for the migration: $0. Both machines were already powered on.

4. Why K3s and Cloudflare Tunnel Over the Obvious Alternatives?

Why K3s over Docker Compose

We already had 30+ Kubernetes manifests, Kustomize overlays, ArgoCD applications, and a working GitOps pipeline. Docker Compose would have meant rewriting every deployment, service, configmap, network policy, and ingress rule into a completely different format.

K3s is a CNCF-certified Kubernetes distribution that runs as a single binary. Our existing manifests work without modification. ArgoCD just needs a new overlay path. Control plane: ~500 MB RAM.

Why Cloudflare Tunnel over port forwarding

Cloudflare Tunnel creates an outbound-only encrypted connection from our server to Cloudflare's edge. The critical property is that the cluster exposes no inbound ports. No port forwarding rules. No dynamic DNS. No firewall holes for the cluster.

Traffic path: Internet → Cloudflare (TLS, WAF, DDoS) → Tunnel → nginx-ingress → pods.

We were already using Cloudflare for DNS on our product domains. The tunnel is free on Cloudflare's free plan.

Why bare metal over Proxmox

Our original plan was Proxmox VE with an Ubuntu VM. During execution, we realized Proxmox added complexity without benefit: the VM layer costs 1 GB of RAM for the hypervisor, and we did not need snapshots, live migration, or a web management UI badly enough to justify it. Ubuntu's autoinstall USB gave us the same zero-touch installation. Bare metal won.

5. The migration — step by step (1 day)

Phase 0: OS installation (30 min)

Created an autoinstall USB with Ubuntu LTS and a user-data file pre-configuring hostname, network, SSH keys, and packages. Plugged it into the server, booted from USB, walked away. Came back to a fully configured server.

Phase 1: K3s installation (15 min)

curl -sfL https://get.k3s.io | sh -

That is the entire installation. K3s was running within a minute. We installed Helm 3 and copied the kubeconfig to our dev workstation for remote kubectl access.

Phase 2: Platform services (45 min)

Installed via Helm charts:

Service Purpose
ArgoCD GitOps controller
nginx-ingress Ingress controller (ClusterIP)
Sealed Secrets Replaces GCP Secret Manager + ESO
node-exporter Prometheus metrics

Three PriorityClasses (prod-critical: 1000, platform-normal: 500, test-low: 100) so the pods in the prod namespace win if the node gets resource-constrained.

Key change from GCP: ESO does not exist on K3s, so we switched to Bitnami Sealed Secrets. We kubeseal secrets on the dev workstation using the controller's public cert; the encrypted SealedSecret YAML is safe to commit to git. The controller decrypts on the cluster into regular Kubernetes Secret resources.

Phase 3: Cloudflare Tunnel (10 min)

cloudflared tunnel create k3s-cluster

cloudflared tunnel route dns k3s-cluster example.com
cloudflared tunnel route dns k3s-cluster app.example.com
cloudflared tunnel route dns k3s-cluster api.example.com

Deployed cloudflared as a Kubernetes Deployment with the tunnel credentials mounted from a Secret and a ConfigMap defining the ingress rules. The tunnel established outbound QUIC connections to Cloudflare's edge. CLI-only — no browser OAuth flow needed.

Phase 4: Container images (20 min)

First attempt: docker save on Windows and docker load on Linux. This corrupted the images — OCI manifests showed up as 2.2 KB stubs instead of actual layers. Cross-platform save/load does not reliably preserve OCI format.

The fix was simple: pull directly from GCP Artifact Registry:

gcloud auth print-access-token | docker login -u oauth2accesstoken \
  --password-stdin REGION-docker.pkg.dev

for img in api frontend ...; do
  docker pull REGION-docker.pkg.dev/PROJECT/REPO/$img:latest
done

This also meant our existing Azure Pipelines CI/CD kept working unchanged — pipelines push to GCP Artifact Registry, K3s pulls from the same registry with an imagePullSecret.

Phase 5: GitOps overlays & ArgoCD deployment (90 min)

The bulk of the work: creating onprem overlays alongside the existing gcp overlays. 

gitops/
  app-a/
    base/               # Shared manifests
    overlays/
      gcp/              # Cloud-specific (ESO secrets)
      onprem/           # On-prem (sealed secrets, LAN hosts, Cloudflare)
  app-b/
    base/               # Cloud version (includes ESO)
    base-onprem/        # On-prem version (excludes ESO CRDs)
    overlays/
      onprem/
  apps/
    app-a-onprem.yaml   # ArgoCD Application
    app-b-onprem.yaml

The onprem overlay patches configmaps to point at the LAN database host (PostgreSQL and Redis), replaces ExternalSecret resources with SealedSecret, disables SSL redirect (Cloudflare handles TLS), and adds imagePullSecrets for the container registry.

The Kustomize images transformer remaps every image reference to GCP Artifact Registry with pinned tags. We created four ArgoCD Application resources, pointed them at the onprem overlay paths, enabled auto-sync with prune and self-heal, and pushed.

Five commits and 90 minutes later: 44 out of 44 application pods running across 5 namespaces (the full cluster count is in section 7).

Phase 6: DNS cutover (2 min)

With the tunnel live and all pods healthy, DNS cutover was a CNAME change in Cloudflare's dashboard. Traffic shifted from the GKE external IP to the tunnel endpoint. No downtime. Rollback is the same change in reverse — two minutes.

6. What Broke During the Migration (and How We Fixed It)?

inotify limit exhaustion

Symptom: .NET pods randomly failing to start. Cause: Linux default fs.inotify.max_user_instances=128. Each .NET process watches files for config reload. 36+ .NET containers blew through the limit. Fix: 

echo "fs.inotify.max_user_instances=1024" >> /etc/sysctl.conf
sysctl -p

Applied on the host, not inside containers.

Kafka OOMKilled in test environment

Symptom: Kafka in the test namespace stuck in CrashLoopBackOff. Cause: We set test resources lower than the prod namespace to save memory. Kafka's JVM heap does not scale with traffic — it needs the same memory whether it is processing 0 or 10,000 messages. Fix: 512Mi request, 1Gi limit minimum, regardless of environment. Stateful services get production-grade resources everywhere.

SSL redirect loop

Symptom: ERR_TOO_MANY_REDIRECTS on every domain. Cause: nginx-ingress redirected HTTP to HTTPS, but the tunnel route pointed at an http:// service URL. cloudflared sent plain HTTP, nginx answered with a redirect to HTTPS, and the browser came back through the tunnel again. Infinite loop. Fix: nginx.ingress.kubernetes.io/ssl-redirect: "false" on all ingresses. TLS terminates at Cloudflare, and the edge to cloudflared hop is already encrypted. We kept the zone on Full / Full (strict) rather than switching to Flexible. Cloudflare's docs say not to use Flexible to solve the redirect loop; see also their guide on HTTPS origins with Tunnel.

Docker save/load corrupts OCI images cross-platform

Symptom: Loaded images showed 2.2 KB manifests instead of multi-hundred-MB layers. Cause: docker save tar format is not guaranteed portable across platforms for OCI images. Fix: Skip save/load. Pull from the registry on the target machine.

Tailscale MagicDNS hijacks system DNS

Symptom: After installing Tailscale, apt update and helm repo add failed with DNS errors. Cause: Tailscale's MagicDNS rewrites /etc/resolv.conf to its own resolver. If your control plane does not serve general DNS, all queries fail. Fix: Override with systemd-resolved pointing at public resolvers (Cloudflare and Google).

ESO CRDs missing on K3s

Symptom: ArgoCD sync failed with no matches for kind "SecretStore". Cause: Our GCP overlay referenced External Secrets Operator CRDs that K3s does not have. Fix: Created base-onprem/ directories that exclude all ESO resources. The onprem overlay references base-onprem/ instead of base/. Kustomize does not let an overlay selectively exclude individual resources from a base, so the separate base directory was necessary.

Wrong database IP

Symptom: All pods up, every API returned 500. Cause: Connection strings pointed at a stale LAN address; the database host had a different IP. Fix: ping first, then update configmaps. A 10-second check would have saved 20 minutes.

7. What Did the Migration Actually Save?

Namespace Pods Purpose
platform 18 APIs + frontend + Kafka + ZooKeeper
test 16 Mirror of the platform namespace
product B 2 API + UI
product C 1 Launcher
logging 3 Loki + Grafana + Promtail
System 15 K3s, ArgoCD, nginx, sealed-secrets, cloudflared, node-exporter
Total ~55

Resource usage on the server:

  • CPU: ~5%
  • RAM: ~10%
  • Disk: ~3%

Headroom for 5–10× our current workload before we would need a second node.

Cost comparison

GCP GKE On-Prem K3s
Monthly cost ~$200 ~$5 (electricity)
Annual cost ~$2,400 ~$60
Hardware $0 (pay-as-you-go) $0 extra (already owned)
Annual savings ~$2,340

Security

  • No inbound ports on the cluster — Cloudflare Tunnel is outbound-only
  • Cloudflare WAF — free-tier DDoS protection, rate limiting, bot detection
  • Network policies — deny-by-default per namespace, explicit allow rules
  • Sealed Secrets — encrypted at rest in git, decrypted only inside the cluster

Deployment workflow (unchanged)

Developer pushes code
  → Azure Pipelines builds container image
  → Pushes to GCP Artifact Registry
  → Updates image tag in gitops repo
  → ArgoCD detects change (< 3 min)
  → Pulls new image on K3s
  → Rolling deployment

Developers do not need to know or care that the cluster moved. The workflow is identical.

8. What Would We Do Differently Next Time?

  1. Set ssl-redirect: "false" from the start. We wasted 30 minutes debugging the redirect loop. If your tunnel route uses an http:// service URL, turn off the ingress HTTP-to-HTTPS redirect up front and keep the zone on Full / Full (strict). Don't reach for Flexible to fix the loop.
  2. Never reduce resources for stateful services in test environments. Kafka, ZooKeeper, Redis, PostgreSQL need the same memory whether they handle test or heavier traffic. Save resources on stateless application pods instead.
  3. Verify every IP before writing configs. A 10-second ping would have saved 20 minutes of debugging.
  4. Use GCP Artifact Registry directly from K3s. Our original plan involved a local CNCF Distribution registry. Pulling directly from GCP turned out simpler and means our CI/CD needs zero changes.
  5. Set inotify limits in the autoinstall user-data. fs.inotify.max_user_instances=1024 belongs in OS provisioning, not a post-deployment hotfix.
  6. Skip Proxmox for single-purpose machines. Bare metal Ubuntu is simpler, uses less RAM, and the autoinstall USB provides the same hands-off experience.

9. Is this right for you?

Good for:

  • Dev, QA, staging and demo environments — a full cloud-stack replica running locally
  • Proof-of-concept work where $2,400/year of cloud spend is hard to justify
  • Hobby and side projects
  • Learning Kubernetes without worrying about cloud bills

Not good for:

  • High-availability requirements — single node, single point of failure (mitigated by the 2-minute DNS rollback to cloud)
  • Global traffic with latency requirements — your traffic routes through one physical location
  • Regulated industries — SOC 2, HIPAA, PCI-DSS auditors want cloud-provider compliance certifications
  • Bursty traffic — no auto-scaling beyond what the single node can handle

The key insight

K3s manifests are standard Kubernetes. Kustomize overlays let you maintain gcp/ and onprem/ side by side. ArgoCD does not care where the cluster is. You can move back to cloud in the time it takes to change a DNS record. This is not a one-way door.

The cloud is not always the answer. For dev, QA and proof-of-concept environments running lightweight workloads, K3s and Cloudflare Tunnel on a ~$300 mini PC we already owned provide the same Kubernetes experience as GKE at 2.5% of the cost. The migration took one day, preserved the entire GitOps workflow, and the rollback path is a DNS change.

If you are paying cloud bills for workloads that could run on hardware you already own, do the math.

Part 2 of this series covers the monitoring stack we built on the same hardware — Production-grade monitoring on a $5/month server.