We run three SaaS products on Kubernetes: a logistics platform with 15 .NET microservices, an AEO (Answer Engine Optimization) tool, and an AI orchestration system. All of them ran on Google Kubernetes Engine at roughly $200/month. We have zero revenue. That had to change.

This is the full technical story of how we migrated everything to a mini PC under the TV in one day, cut our infrastructure bill to about $5/month in electricity, and kept the exact same GitOps workflow.

1. Why Did We Migrate Off GCP?

GKE cost ~$200/month against 4% CPU utilization on a pre-revenue startup, while a mini PC we already owned sat unused — the math was too obvious to ignore.

GKE was costing us $190–200/month:

GCP resource Spec Monthly cost
GKE control plane Standard cluster ~$73
e2-standard-4 node 4 vCPU, 16 GB RAM ~$97
Persistent disks 62 GB SSD ~$10
Network egress Variable ~$5–10
Total ~$200

For a pre-revenue startup, $2,400/year on infrastructure you can run on hardware you already own is hard to justify. We had a mini PC sitting unused. CPU utilization on GCP was 4%. The math was obvious.

The question was never "should we move?" but "can we do it without losing our deployment workflow?"

2. What we were running on GCP

Our GKE cluster (squib-cluster, us-central1-a) ran a single e2-standard-4 node hosting 40+ pods across three products:

  • Squib365 (logistics SaaS) — 15 .NET microservices (auth, core, fleet, driver, order, dispatch, account, maintenance, messages, notifications, payroll, profile, security, transport, plus an API gateway), an Angular frontend, Kafka, and ZooKeeper. Row-level security, tenant isolation, the works.
  • Lunimrank (AEO platform) — An ASP.NET Core API and an Angular UI. Scans AI engines (OpenAI, Perplexity, Gemini) to score how visible a business is in AI-generated answers.
  • Nova AI (AI orchestration) — A Fastify/Node.js launcher that manages Claude Code worker agents via WebSocket and REST endpoints.

Plus platform services: ArgoCD (App of Apps), nginx-ingress, cert-manager, External Secrets Operator syncing with GCP Secret Manager, Prometheus node-exporter. CI/CD via Azure Pipelines pushing images to GCP Artifact Registry and ArgoCD auto-syncing within three minutes. DNS through Cloudflare.

Not a toy setup: 17 container images, 5 namespaces, Kustomize overlays, sealed secrets, network policies, HPAs. The migration had to preserve all of it.

3. The hardware

PC3 — the mini PC that became our production cluster:

  • CPU: Intel i5-12450H (8 cores, 12 threads)
  • RAM: 40 GB DDR4
  • Storage: 512 GB NVMe SSD
  • Network: 1 Gbps Ethernet
  • OS: Ubuntu 24.04.2 LTS, bare metal

PC2 — an existing machine on the same LAN running PostgreSQL (4 databases), Redis, Headscale (self-hosted Tailscale control plane), and a mail server.

Total hardware cost for the migration: $0. Both machines were already powered on.

4. Why K3s and Cloudflare Tunnel Over the Obvious Alternatives?

Why K3s over Docker Compose

We already had 30+ Kubernetes manifests, Kustomize overlays, ArgoCD applications, and a working GitOps pipeline. Docker Compose would have meant rewriting every deployment, service, configmap, network policy, and ingress rule into a completely different format.

K3s is a CNCF-certified Kubernetes distribution that runs as a single binary. Our existing manifests work without modification. ArgoCD just needs a new overlay path. Control plane: ~500 MB RAM.

Why Cloudflare Tunnel over port forwarding

Cloudflare Tunnel creates an outbound-only encrypted connection from our mini PC to Cloudflare's edge. The critical property is zero open inbound ports. No port forwarding rules. No dynamic DNS. No firewall holes. If someone scans our IP, they see nothing.

Traffic path: Internet → Cloudflare (TLS, WAF, DDoS) → Tunnel → nginx-ingress → pods.

We were already using Cloudflare for DNS on all three domains (squib365.com, lunimrank.com, blockb.ca). The tunnel is free on Cloudflare's free plan.

Why bare metal over Proxmox

Our original plan was Proxmox VE with an Ubuntu VM. During execution, we realized Proxmox added complexity without benefit: the VM layer costs 1 GB of RAM for the hypervisor, and we did not need snapshots, live migration, or a web management UI badly enough to justify it. Ubuntu's autoinstall USB gave us the same zero-touch installation. Bare metal won.

5. The migration — step by step (1 day)

Phase 0: OS installation (30 min)

Created an autoinstall USB with Ubuntu 24.04 and a user-data file pre-configuring hostname, network, SSH keys, and packages. Plugged it into PC3, booted from USB, walked away. Came back to a fully configured server.

Phase 1: K3s installation (15 min)

curl -sfL https://get.k3s.io | sh -

That is the entire installation. K3s v1.34.5 running within a minute. We installed Helm v3.20.1 and copied the kubeconfig to our dev workstation for remote kubectl access.

Phase 2: Platform services (45 min)

Installed via Helm charts:

Service Chart version Purpose
ArgoCD 7.8.3 GitOps controller
nginx-ingress 4.12.1 Ingress controller (ClusterIP)
Sealed Secrets 2.17.1 Replaces GCP Secret Manager + ESO
node-exporter 4.52.0 Prometheus metrics

Three PriorityClasses (prod-critical: 1000, platform-normal: 500, test-low: 100) so production pods win if the node gets resource-constrained.

Key change from GCP: ESO does not exist on K3s, so we switched to Bitnami Sealed Secrets. We kubeseal secrets on the dev workstation using the controller's public cert; the encrypted SealedSecret YAML is safe to commit to git. The controller decrypts on the cluster into regular Kubernetes Secret resources.

Phase 3: Cloudflare Tunnel (10 min)

cloudflared tunnel create on-prem-cluster

cloudflared tunnel route dns on-prem-cluster squib365.com
cloudflared tunnel route dns on-prem-cluster app.squib365.com
cloudflared tunnel route dns on-prem-cluster lunimrank.com
cloudflared tunnel route dns on-prem-cluster nova.squib365.com
cloudflared tunnel route dns on-prem-cluster logs.squib365.com

Deployed cloudflared as a Kubernetes Deployment with the tunnel credentials mounted from a Secret and a ConfigMap defining the ingress rules. The tunnel established 4 QUIC connections to Cloudflare's ORD (Chicago) edge. CLI-only — no browser OAuth flow needed.

Phase 4: Container images (20 min)

First attempt: docker save on Windows and docker load on Linux. This corrupted the images — OCI manifests showed up as 2.2 KB stubs instead of actual layers. Cross-platform save/load does not reliably preserve OCI format.

The fix was simple: pull directly from GCP Artifact Registry:

gcloud auth print-access-token | docker login -u oauth2accesstoken \
  --password-stdin us-central1-docker.pkg.dev

for img in gateway auth-api fleet-api driver-api order-api ...; do
  docker pull us-central1-docker.pkg.dev/squib-gcp/squib/$img:latest
done

This also meant our existing Azure Pipelines CI/CD kept working unchanged — pipelines push to GCP Artifact Registry, K3s pulls from the same registry with an imagePullSecret.

Phase 5: GitOps overlays & ArgoCD deployment (90 min)

The bulk of the work: creating onprem overlays alongside the existing gcp overlays. 

gitops/
  squib/
    base/               # Shared manifests
    overlays/
      gcp/              # GCP-specific (ESO secrets, GCP IPs)
      onprem/           # On-prem (sealed secrets, LAN IPs, Cloudflare)
  lunimrank/
    base/               # GCP version (includes ESO)
    base-onprem/        # On-prem version (excludes ESO CRDs)
    overlays/
      onprem/
  apps/
    squib-onprem.yaml   # ArgoCD Application
    lunimrank-onprem.yaml
    nova-onprem.yaml
    logging-onprem.yaml

The onprem overlay patches configmaps to point at PC2's LAN IP (192.168.0.36:5432 for PostgreSQL, 192.168.0.36:6379 for Redis), replaces ExternalSecret resources with SealedSecret, disables SSL redirect (Cloudflare handles TLS), and adds imagePullSecrets for GCP Artifact Registry.

The Kustomize images transformer remaps every image reference to GCP Artifact Registry with pinned tags. We created four ArgoCD Application resources (one per product), pointed them at the onprem overlay paths, enabled auto-sync with prune and self-heal, and pushed.

Five commits and 90 minutes later: 44 out of 44 application pods running across 5 namespaces.

Phase 6: DNS cutover (2 min)

With the tunnel live and all pods healthy, DNS cutover was a CNAME change in Cloudflare's dashboard. Traffic shifted from the GKE external IP to the tunnel endpoint. No downtime. Rollback is the same change in reverse — two minutes.

6. What Broke During the Migration (and How We Fixed It)?

inotify limit exhaustion

Symptom: .NET pods randomly failing to start. Cause: Linux default fs.inotify.max_user_instances=128. Each .NET process watches files for config reload. 36+ .NET containers blew through the limit. Fix: 

echo "fs.inotify.max_user_instances=1024" >> /etc/sysctl.conf
sysctl -p

Applied on the host, not inside containers.

Kafka OOMKilled in test environment

Symptom: Kafka in squib-test stuck in CrashLoopBackOff. Cause: We set test resources lower than production to save memory. Kafka's JVM heap does not scale with traffic — it needs the same memory whether it is processing 0 or 10,000 messages. Fix: 512Mi request, 1Gi limit minimum, regardless of environment. Stateful services get production-grade resources everywhere.

SSL redirect loop

Symptom: ERR_TOO_MANY_REDIRECTS on every domain. Cause: Cloudflare SSL mode was set to "Strict" but nginx-ingress behind the tunnel serves HTTP. Cloudflare sent HTTPS, nginx redirected to HTTPS, Cloudflare sent HTTPS again. Infinite loop. Fix: Cloudflare SSL = "Flexible", and nginx.ingress.kubernetes.io/ssl-redirect: "false" on all ingresses. TLS is handled entirely by Cloudflare.

Docker save/load corrupts OCI images cross-platform

Symptom: Loaded images showed 2.2 KB manifests instead of multi-hundred-MB layers. Cause: docker save tar format is not guaranteed portable across platforms for OCI images. Fix: Skip save/load. Pull from the registry on the target machine.

Tailscale MagicDNS hijacks system DNS

Symptom: After installing Tailscale, apt update and helm repo add failed with DNS errors. Cause: Tailscale's MagicDNS rewrites /etc/resolv.conf to its own resolver. If your control plane (Headscale here) does not serve general DNS, all queries fail. Fix: Override with systemd-resolved pointing at 1.1.1.1 and 8.8.8.8.

ESO CRDs missing on K3s

Symptom: ArgoCD sync failed with no matches for kind "SecretStore". Cause: Our GCP overlay referenced External Secrets Operator CRDs that K3s does not have. Fix: Created base-onprem/ directories that exclude all ESO resources. The onprem overlay references base-onprem/ instead of base/. Kustomize does not let an overlay selectively exclude individual resources from a base, so the separate base directory was necessary.

Wrong database IP

Symptom: All pods up, every API returned 500. Cause: Connection strings pointed at 192.168.0.106; PC2's actual IP was 192.168.0.36. Fix: ping first, then update configmaps. A 10-second check would have saved 20 minutes.

7. What Did the Migration Actually Save?

Namespace Pods Purpose
squib (prod) 18 15 APIs + frontend + Kafka + ZooKeeper
squib-test 16 Mirror of prod
lunimrank 2 API + UI
nova 1 Launcher
logging 3 Loki + Grafana + Promtail
System 15 K3s, ArgoCD, nginx, sealed-secrets, cloudflared, node-exporter
Total ~57

Resource usage on the 40 GB / 12-core mini PC:

  • CPU: 488m of 12 cores → 4%
  • RAM: 4.6 GB of 39 GB → 9%
  • Disk: 14 GB of 468 GB → 4%

Headroom for 5–10× our current workload before we would need a second node.

Cost comparison

GCP GKE On-Prem K3s
Monthly cost ~$200 ~$5 (electricity)
Annual cost ~$2,400 ~$60
Hardware $0 (pay-as-you-go) $0 (already owned)
Annual savings ~$2,340

Security

  • Zero open inbound ports — Cloudflare Tunnel is outbound-only
  • Cloudflare WAF — free-tier DDoS protection, rate limiting, bot detection
  • Network policies — deny-by-default per namespace, explicit allow rules
  • Sealed Secrets — encrypted at rest in git, decrypted only inside the cluster

Deployment workflow (unchanged)

Developer pushes code
  → Azure Pipelines builds container image
  → Pushes to GCP Artifact Registry
  → Updates image tag in gitops repo
  → ArgoCD detects change (< 3 min)
  → Pulls new image on K3s
  → Rolling deployment

Developers do not need to know or care that the cluster moved. The workflow is identical.

8. What Would We Do Differently Next Time?

  1. Start with Cloudflare SSL "Flexible" mode. We wasted 30 minutes debugging the redirect loop. If your origin serves HTTP behind a tunnel, start with Flexible; only move to Full/Strict if you add origin TLS.
  2. Never reduce resources for stateful services in test environments. Kafka, ZooKeeper, Redis, PostgreSQL need the same memory whether they handle test or production traffic. Save resources on stateless application pods instead.
  3. Verify every IP before writing configs. A 10-second ping would have saved 20 minutes of debugging.
  4. Use GCP Artifact Registry directly from K3s. Our original plan involved a local CNCF Distribution registry. Pulling directly from GCP turned out simpler and means our CI/CD needs zero changes.
  5. Set inotify limits in the autoinstall user-data. fs.inotify.max_user_instances=1024 belongs in OS provisioning, not a post-deployment hotfix.
  6. Skip Proxmox for single-purpose machines. Bare metal Ubuntu is simpler, uses less RAM, and the autoinstall USB provides the same hands-off experience.

9. Is this right for you?

Good for:

  • Pre-revenue startups — $2,400/year buys real runway with no income
  • Dev/staging environments — full cloud-stack replica running locally
  • Hobby projects, side projects that need to stay up
  • Learning Kubernetes without worrying about cloud bills

Not good for:

  • High-availability requirements — single node, single point of failure (mitigated by the 2-minute DNS rollback to cloud)
  • Global traffic with latency requirements — your traffic routes through one physical location
  • Regulated industries — SOC 2, HIPAA, PCI-DSS auditors want cloud-provider compliance certifications
  • Bursty traffic — no auto-scaling beyond what the single node can handle

The key insight

K3s manifests are standard Kubernetes. Kustomize overlays let you maintain gcp/ and onprem/ side by side. ArgoCD does not care where the cluster is. You can move back to cloud in the time it takes to change a DNS record. This is not a one-way door.

The cloud is not always the answer. For a pre-revenue startup running lightweight workloads, a $300 mini PC with K3s and Cloudflare Tunnel provides the same Kubernetes experience as GKE at 2.5% of the cost. The migration took one day, preserved the entire GitOps workflow, and the rollback path is a DNS change.

If you are paying cloud bills for workloads that could run on hardware you already own, do the math.

Part 2 of this series covers the monitoring stack we built on the same hardware — Production-grade monitoring on a $5/month server.