K05: Inadequate Logging and Monitoring

📋 What Is It?

Inadequate Logging and Monitoring is the condition of running a cluster that cannot see itself being attacked. The controls that would record who did what to the API server, what processes ran inside containers, and which network flows crossed the cluster are switched off, too coarse, never centralized, or collected but never alerted on. It is a meta-weakness: it rarely causes the breach, but it determines whether you notice it in minutes, in months, or never. Dwell time - the interval between compromise and detection - is the metric this control governs.

K05 OWASP Rank

Detection Focus Area

Medium To Prevent

⚠️ Common Exploits

Silent exec: A shell in a pod leaves no trace with audit off and no runtime sensor
Unlogged escalation: New ClusterRoleBindings and token minting happen with no record
Invisible secret reads: Bulk get/list secrets is indistinguishable from normal traffic when audit is None/Metadata-only
Runtime blindness: Container escape via nsenter produces zero telemetry with no Falco/Tetragon
Cryptominer: A miner from an odd registry runs until it shows up on the cloud bill
Lateral movement: East-west scanning goes unseen with no network flow logging
Evidence deletion: Node-local logs are the attacker's to rm when nothing is shipped off-node

🔴 Attack Flow

1. Foothold: shell unrecorded (no runtime sensor)
↓
2. Recon: enumeration blends into normal traffic
↓
3. Escalate: RBAC change raises no flag
↓
4. Objective: mine, read secrets, exfiltrate quietly
↓
5. IMPACT: Cleanup deletes node logs - trail gone!

❌ Vulnerable Code

# Bad: kube-apiserver started with NO audit configuration
kube-apiserver \
  --etcd-servers=https://127.0.0.1:2379 \
  --authorization-mode=Node,RBAC
  # (no --audit-policy-file, no --audit-log-path at all)
# Result: no record of exec, secret reads, RBAC changes, or deletes

# Bad: "has audit logging" but the policy captures nothing useful
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: None          # everything is dropped; the log is empty of signal

# Bad: no runtime sensor anywhere in the cluster
$ kubectl get daemonset -A | grep -iE 'falco|tetragon'
# (no output) -> reverse shells, token reads, host mounts all invisible

# Bad: logs live only on the node and are attacker-deletable
$ rm -f /var/log/pods/*/*/*.log /var/log/containers/*.log
# ...and the only copy of the evidence is gone. No alerting exists.
                

✅ Secure Code

# Good: tiered API audit policy - cheap metadata + full detail where it matters
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
  # Full request+response for the crown-jewel resources
  - level: RequestResponse
    resources:
      - { group: "", resources: ["secrets", "configmaps", "serviceaccounts"] }
      - { group: "rbac.authorization.k8s.io", resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"] }
  # Record exec / attach / port-forward into pods in full
  - level: RequestResponse
    resources:
      - { group: "", resources: ["pods/exec", "pods/attach", "pods/portforward"] }
  # Drop noisy, low-value reads to control cost
  - { level: None, users: ["system:kube-scheduler", "system:kube-proxy"], verbs: ["watch", "get"] }
  # Everything else: metadata is cheap and still valuable
  - level: Metadata

# Good: deploy Falco as a DaemonSet on every node for runtime detection
helm install falco falcosecurity/falco --namespace falco --create-namespace \
  --set falcosidekick.enabled=true

# Good: ship every source off-cluster to tamper-resistant (append-only) storage,
# and alert on high-risk events (exec, privileged pod, cluster-admin bind) via
# Prometheus/Alertmanager -> on-call paging with a runbook attached.
                

✓ Prevention Checklist

Enable API-server audit logging with a tiered policy (not off, not None-everywhere)
Capture RequestResponse for exec, secret access, and RBAC changes
Deploy a runtime sensor (Falco / Tetragon) as a DaemonSet on every node
Ship pod/container logs off-node before the pod can be deleted
Centralize all sources in tamper-resistant, off-cluster (append-only/WORM) storage
Alert on high-risk events - privileged pods, cluster-admin binds, odd registries, exec
Collect and correlate the cloud control-plane and IAM audit trail
Retain logs for weeks-to-months, matching realistic time-to-discovery
Synchronize node clocks (NTP/chrony) so events order into one timeline
Wire alerts into an incident-response process with paging and runbooks
Tune noisy rules - the answer to noise is a sharper policy, never disabling the signal

🔍 Detection & Tools

Runtime & Audit:

Falco Tetragon API audit policy kube-bench kubescape

Aggregation & Alerting:

Fluent Bit Elasticsearch / Loki Prometheus / Alertmanager Falcosidekick SIEM

How to Test:

Check the apiserver for --audit-policy-file / --audit-log-path
Confirm a runtime DaemonSet exists: kubectl get ds -A | grep -iE 'falco|tetragon'
exec into a test pod and verify an alert fires
Create a privileged pod and confirm it is logged and paged
Verify logs are shipped off-cluster and survive pod deletion
Check node clock sync: chronyc tracking

🌍 Real-World

Exposed dashboard to cryptomining (Tesla, 2018): An unauthenticated Kubernetes dashboard led to mining workloads running inside the environment; with no runtime detection or alerting on unusual pulls/CPU, mining ran until cost or performance degradation was noticed.
Long-dwell exfiltration via SA tokens: A compromised workload uses its auto-mounted token to query the API and read secrets; with audit off or Metadata-only, the anomalous token use blends into normal control-plane traffic and continues undetected.
Worming malware across misconfigured clusters: Campaigns such as Hildegard, Kinsing, and TeamTNT tooling target exposed kubelets and API servers, spread laterally, and install miners - exactly the behaviours a Falco/Tetragon rule catches and that go unseen with no sensor.
Common thread: In none of these did inadequate logging cause the initial compromise - in all of them it converted a contained incident into a sustained, expensive, hard-to-scope breach.

📌 Quick Tips

DO NOT assume the cloud provider logs everything
DO NOT treat kubectl logs as durable logging
DO NOT disable audit logging to reduce noise
DO enable tiered API audit logging
DO run a runtime sensor on every node
DO ship logs off-cluster to tamper-resistant storage
DO alert and page, don't just collect

📜 Compliance

Related Standards:

PCI-DSS Requirement 10 - Logging & Monitoring
NIST 800-53 AU family - Audit & Accountability
CIS Kubernetes Benchmark - Logging
SOC 2 CC7.2 - Monitoring
ISO 27001 A.12.4 - Logging & Monitoring

📊 Inadequate Logging and Monitoring