In the previous post, we migrated three SaaS products from Google Kubernetes Engine to a mini PC running K3s. We saved $2,340/year. We had 58 pods running, Cloudflare handling TLS, and ArgoCD deploying our code. But we had no monitoring. If a pod crashed at 3 AM, nobody would know until a user complained.

This is how we built a production-grade observability stack — metrics, logs, alerts, and deployment notifications — all running on that same $5/month mini PC, all alerting to Microsoft Teams.

1. Why Wasn't a Bash Script on Cron Enough?

Our initial monitoring was a bash script running on a 5-minute cron: 

#!/bin/bash
RUNNING=$(kubectl get pods -A --no-headers | grep Running | wc -l)
[ $RUNNING -lt 40 ] && alert "Only $RUNNING pods running"

DISK=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
[ $DISK -gt 80 ] && alert "Disk at ${DISK}%"

MEM=$(free | grep Mem | awk '{printf "%.0f", $3/$2*100}')
[ $MEM -gt 80 ] && alert "Memory at ${MEM}%"

It worked. But it had problems:

  • No historical data — "Was CPU high last Tuesday?" No idea.
  • No per-pod visibility — "Which pod is using all the memory?" No idea.
  • No log aggregation — Want to see errors from all 15 microservices? SSH in and check each one.
  • No deployment tracking — "Did that deploy succeed?" Check ArgoCD manually.
  • Does not scale — Adding a new app means editing the bash script.
  • 5-minute blind spots — A pod can crash, restart, and recover between checks. You would never know.

2. What Does the Full Monitoring Stack Look Like?

Four layers on one mini PC: Prometheus + Loki + ArgoCD for collection, Alertmanager with 35 rules for alerting, Microsoft Teams for notification, and Grafana for visualization — about 2 GB of RAM total. 

┌──────────────────────────────────────────────────────┐
│                  DATA COLLECTION                      │
│  Prometheus         Loki              ArgoCD          │
│  (metrics)          (logs)            (deploys)       │
│  15 scrape targets  4 namespaces      6 apps          │
│  30s intervals      all pod logs      sync events     │
├──────────────────────────────────────────────────────┤
│                  ALERT ENGINE                         │
│  Alertmanager (Prometheus)                            │
│  35 PrometheusRules (34 built-in + 1 custom)          │
│  Group by: [namespace, alertname]                     │
│  Severity routing: critical (1h), warning (4h)        │
├──────────────────────────────────────────────────────┤
│                  NOTIFICATION                         │
│  Microsoft Teams (Power Automate Workflow webhook)    │
│  Deployment alerts (ArgoCD → Teams)                   │
│  Infrastructure alerts (Prometheus → Teams)           │
├──────────────────────────────────────────────────────┤
│                  VISUALIZATION                        │
│  Grafana (logs.squib365.com)                          │
│  Datasources: Prometheus + Loki + Alertmanager        │
│  PromQL for metrics, LogQL for logs                   │
└──────────────────────────────────────────────────────┘

Components

Component Version RAM Purpose
Prometheus v0.89.0 ~1.5 GB Scrapes metrics every 30s
Alertmanager (bundled) ~64 MB Deduplicates, groups, routes alerts
kube-state-metrics (bundled) ~50 MB Pod count, restart count, phase as metrics
Prometheus Operator (bundled) ~50 MB Manages alert rules as CRDs
node-exporter 1.10.2 ~20 MB CPU, memory, disk, network from host
Grafana 11.5.2 ~128 MB Dashboards + ad-hoc queries
Loki 3.4.2 ~128 MB Logs from all pods (30-day retention)
Promtail 3.4.2 ~64 MB Ships pod logs to Loki (DaemonSet)
Total ~2 GB 5% of our 40 GB RAM

After installing all of this, our mini PC went from 13% to 15% memory usage.

3. How Long Does This Stack Take to Install?

About 10 minutes total: 2 minutes to unbind K3s's internal components from localhost, 5 minutes for one Helm install, and 2 minutes to wire Grafana to the new Prometheus datasource.

Step 1: Configure K3s metrics (2 min)

K3s binds its internal components (controller-manager, scheduler, proxy) to 127.0.0.1 by default. Prometheus cannot scrape them. Fix: 

# /etc/rancher/k3s/config.yaml
kube-controller-manager-arg:
  - bind-address=0.0.0.0
kube-scheduler-arg:
  - bind-address=0.0.0.0
kube-proxy-arg:
  - metrics-bind-address=0.0.0.0

Then sudo systemctl restart k3s. The only K3s-specific configuration needed.

Step 2: Install kube-prometheus-stack (5 min)

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  -f helm-values.yaml --wait --timeout 5m

Our helm-values.yaml disables the bundled Grafana (we already have one in the logging namespace), disables node-exporter (already installed separately), and enables all 34 default PrometheusRules: 

grafana:
  enabled: false
nodeExporter:
  enabled: false
prometheus:
  prometheusSpec:
    retention: 30d
    retentionSize: "40GB"
    storageSpec:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: 50Gi
    # Scrape ALL ServiceMonitors across ALL namespaces
    serviceMonitorSelectorNilUsesHelmValues: false

That last line is critical. Without it, Prometheus only scrapes ServiceMonitors in its own namespace. With it set, 15 scrape targets up, 34 PrometheusRules active, 0 configuration needed per app.

Step 3: Connect Grafana to Prometheus (2 min)

Our Grafana was already running at logs.squib365.com with Loki as its only datasource. We added Prometheus and Alertmanager via the API: 

curl -X POST "http://$GRAFANA_IP:3000/api/datasources" \
  -u admin:$GRAFANA_PASS \
  -H "Content-Type: application/json" \
  --data '{
    "name": "Prometheus",
    "type": "prometheus",
    "url": "http://kube-prometheus-kube-prome-prometheus.monitoring.svc.cluster.local:9090",
    "isDefault": true
  }'

Now Grafana can query both metrics (PromQL) and logs (LogQL) from the same dashboard.

4. How Do 35 Alert Rules Need Zero Per-App Configuration?

The 34 built-in kube-prometheus-stack rules fire automatically for every pod, namespace, and node — adding a 40th product gets monitored instantly, with no config file to touch.

The 34 built-in rules

The kube-prometheus-stack Helm chart ships with 34 PrometheusRule CRDs covering:

  • kubernetes-apps: KubePodCrashLooping, KubePodNotReady, KubeDeploymentReplicasMismatch
  • kubernetes-resources: KubeCPUOvercommit, KubeMemoryOvercommit
  • kubernetes-storage: KubePersistentVolumeFillingUp, KubePersistentVolumeErrors
  • node-exporter: NodeFilesystemSpaceFillingUp, NodeRAIDDegraded, NodeClockSkewDetected
  • prometheus: PrometheusTSDBCompactionsFailing, PrometheusRuleFailures

These rules fire automatically for every pod, every namespace, every node. When we add our 5th, 10th, or 40th product, it is monitored instantly. No configuration per app.

Our custom rules

We added 8 rules specific to our platform:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: blockb-platform-alerts
  namespace: monitoring
spec:
  groups:
    - name: blockb-pod-health
      rules:
        - alert: PodCrashLooping
          expr: |
            increase(kube_pod_container_status_restarts_total{
              namespace=~"squib|squib-test|lunimrank|lunimrank-test|nova|logging"
            }[1h]) > 3
          for: 5m
          labels:
            severity: critical

        - alert: RunningPodsBelowThreshold
          expr: sum(kube_pod_status_phase{phase="Running"}) < 40
          for: 5m
          labels:
            severity: critical

    - name: blockb-node-resources
      rules:
        - alert: NodeDiskWillFillIn24h
          expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0
          for: 30m
          labels:
            severity: warning

The predict_linear rule is the standout. It looks at disk usage over the last 6 hours, draws a trend line, and fires if it predicts the disk will be full within 24 hours. You get a warning a day before the problem, not after.

5. How Do You Get Alerts Into Microsoft Teams in 2026?

Not with Office 365 Incoming Webhooks — Microsoft is retiring those connectors, with final rollout May 18–22, 2026. The replacement is a Power Automate Workflow webhook instead.

The webhook problem

Microsoft is retiring Office 365 Incoming Webhooks in Teams, with final retirement rolling out May 18–22, 2026. The replacement is Power Automate Workflows — you create a flow in Teams that generates a webhook URL.

The workflow template "Post to a channel when a webhook request is received" expects a simple JSON body: 

{"text": "Your alert message here"}

Not Adaptive Cards. Not MessageCards. Just a text field. We learned this the hard way after getting HTTP 202 responses but no messages appearing. The Adaptive Card format is accepted by the trigger but not rendered by the simple template.

Alertmanager configuration

receivers:
  - name: "null"
  - name: teams-alerts
    webhook_configs:
      - url: "https://...powerautomate.com/.../invoke?..."
        send_resolved: true
        max_alerts: 5

route:
  receiver: teams-alerts
  group_by: [namespace, alertname]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [alertname = Watchdog]
      receiver: "null"
    - matchers: [severity = critical]
      receiver: teams-alerts
      group_wait: 10s
      repeat_interval: 1h

Key design decisions:

  • Group by namespace + alertname. If 10 pods restart in the squib namespace, you get ONE Teams message listing all 10, not 10 separate messages.
  • Critical alerts repeat every hour, warnings every 4 hours. Prevents alert fatigue while ensuring nothing is forgotten.
  • Watchdog routed to null. This is a heartbeat alert that always fires. Routing it to null means it does not spam Teams.
  • Inhibition rules. If a critical alert fires, it suppresses the warning version of the same alert. One message, not two.
  • send_resolved: true — When an alert resolves, Teams gets a resolution message. You know the problem is fixed without checking.

ArgoCD deployment notifications

Separately from Prometheus alerts, ArgoCD has its own notifications controller. We pointed it at the same Teams webhook: 

template.app-deployed: |
  webhook:
    teams:
      method: POST
      body: |
        {"text": "✅ Deployed: {{.app.metadata.name}} to {{.app.spec.destination.namespace}}"}

template.app-sync-failed: |
  webhook:
    teams:
      method: POST
      body: |
        {"text": "❌ DEPLOY FAILED: {{.app.metadata.name}} in {{.app.spec.destination.namespace}}"}

Now when a developer pushes code → pipeline builds → ArgoCD syncs → Teams gets: "✅ Deployed: squib-platform-onprem to squib". If the sync fails, the team knows immediately.

6. Why Loki Instead of Elasticsearch for Logs?

We chose Loki over Elasticsearch for log aggregation. The decision was straightforward:

Factor Loki Elasticsearch
RAM 512 MB 4-8 GB minimum (JVM)
Indexing Labels only (namespace, pod) Full-text on every field
Query LogQL (grep-like) Lucene (SQL-like)
Grafana Native datasource Needs Kibana or adapter
Complexity 1 pod 3+ node cluster recommended

We do not need full-text search across all log fields. We need "show me errors in the squib namespace in the last hour." Loki does this with a fraction of the resources.

Promtail runs as a DaemonSet, tailing log files from /var/log/pods/{squib,nova,lunimrank,logging}_*/*/*.log. It extracts namespace, pod, and container labels from the file path and pushes to Loki.

Example queries in Grafana:

# All errors in squib namespace
{namespace="squib"} |~ "(?i)(error|exception|fatal)"

# Specific pod logs
{namespace="lunimrank", pod=~"lunimrank-api.*"}

# Error count by namespace (last 1h)
sum by (namespace) (count_over_time({namespace=~".+"} |~ "(?i)error" [1h]))

7. What it looks like in practice

Scenario: pod crashes at 2 AM

  1. T+0s: Pod crashes, Kubernetes restarts it.
  2. T+30s: Prometheus scrapes kube-state-metrics, sees restart count increased.
  3. T+5m: PodCrashLooping rule condition met (> 3 restarts in 1h, sustained for 5m).
  4. T+5m10s: Alertmanager receives alert, waits 10s (critical group_wait).
  5. T+5m10s: Teams notification: "🔥 PodCrashLooping: squib/squib-auth-xyz is crash looping".
  6. Next morning: Open Grafana → Loki → {namespace="squib", pod="squib-auth-xyz"} → see the exact error.
  7. Fix deployed: ArgoCD syncs → Teams: "✅ Deployed: squib-platform-onprem to squib".
  8. Alert resolves: Teams: "✅ Resolved: PodCrashLooping in squib".

Total time from crash to notification: ~5 minutes. With the bash script, it would have been up to 10 minutes (if the crash happened right after a check), and you would not have the logs or the resolution notification.

Scenario: disk filling up

  1. T-24h: NodeDiskWillFillIn24h fires before even reaching 80% — predictive alert gives you a full day warning.
  2. T+0: If the predictive alert was missed, node disk crosses 80% and NodeDiskAbove80Pct fires.
  3. Fix: docker system prune, clean old images, tune Loki retention pruning.

8. Does This Stack Scale Past 4 Apps?

We currently run 4 products. Our monitoring stack was designed for 40. Why it scales without per-app changes:

  1. Prometheus auto-discovers targets via ServiceMonitors. Add a ServiceMonitor → it is scraped automatically.
  2. kube-state-metrics watches all namespaces. New namespace = new pods = automatically tracked.
  3. Promtail scrapes all pod logs matching the glob pattern. Add a namespace → logs appear in Loki.
  4. Alert rules use regex selectors: namespace=~"squib|lunimrank|nova". Add to the regex, or change to namespace=~".+" for all.
  5. Notification routing is label-based. Group by namespace means each app's alerts are grouped separately without configuration.

The only per-app work needed is if you want custom alerting thresholds for a specific service. The platform-level alerts (pod health, node resources, disk prediction) work for every app automatically.

9. How Do You Reproduce This on a New Server?

Everything is declarative YAML in our gitops repo:

gitops/monitoring/base-onprem/
├── helm-values.yaml              # Prometheus stack Helm values
├── alertmanager-config.yaml      # Teams webhook routing
├── custom-prometheus-rules.yaml  # Block B-specific alert rules
├── node-exporter-servicemonitor.yaml
├── grafana-datasources.yaml      # Prometheus + Loki + Alertmanager
├── argocd-notifications-cm.yaml  # Deploy notification templates
└── kustomization.yaml

To reproduce on a new server:

# 1. Add K3s metrics config (/etc/rancher/k3s/config.yaml)
# 2. Restart K3s
sudo systemctl restart k3s

# 3. Install Prometheus stack
helm install kube-prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  -f helm-values.yaml --wait

# 4. Apply custom configs
kubectl apply -k monitoring/base-onprem/

# 5. Set Teams webhook URL
kubectl -n argocd patch secret argocd-notifications-secret --type merge \
  -p '{"stringData": {"teams-webhook-url": "<YOUR_URL>"}}'

Five commands. Full monitoring stack. 35 alert rules. Teams integration. Done.

10. What Would This Cost as a Managed SaaS?

Component Cloud equivalent Our cost
Prometheus + Alertmanager Datadog ($23/host/mo), New Relic ($0.35/GB) $0 (self-hosted)
Loki + Promtail Datadog Logs ($0.10/GB/mo), Elastic Cloud ($100+/mo) $0 (self-hosted)
Grafana Grafana Cloud ($29/mo Pro) $0 (self-hosted)
Teams alerts PagerDuty ($21/user/mo), OpsGenie ($9/user/mo) $0 (Power Automate free tier)
Total monitoring cost $150–500/month $0

The only cost is the RAM and disk on our mini PC. 2 GB of RAM and 65 GB of disk (for 30-day Prometheus + Loki retention). On a machine with 40 GB RAM and 468 GB disk, this is noise.

The takeaway

You do not need a SaaS monitoring platform to monitor a SaaS product. The open-source stack — Prometheus, Alertmanager, Loki, Grafana — provides everything a small team needs: metrics, logs, alerting, dashboards, and deployment tracking. It runs on the same hardware as your applications. It scales with label-based routing, not per-app configuration. And it costs nothing beyond the electricity to keep the machine on.

Our $5/month mini PC now runs 62 pods, 35 alert rules, 15 scrape targets, 30 days of metrics, 30 days of logs, and sends alerts to Microsoft Teams. CPU: 4%. Memory: 15%. Disk: 6%.

The machine is still barely working.