Back

K08: Secrets Management Failures - Overview

Table of Contents

What is a Secrets Management Failure?

Secrets Management Failures occur when sensitive material—database passwords, API keys, TLS private keys, cloud credentials, service-account tokens, signing keys—is created, stored, distributed, or consumed inside a Kubernetes cluster in a way that lets an unintended party read it. It is rarely a single dramatic bug. It is the sum of small, comfortable habits: base64 mistaken for encryption, a password pasted into a manifest, a token committed to Git, a secret exported as an environment variable "because it was easy."

Kubernetes gives you a first-class Secret object, and that object is genuinely useful—but it is frequently misunderstood. A Secret is base64-encoded, not encrypted. By default it is stored in etcd, the cluster's key-value backing store, and unless you have explicitly turned on encryption-at-rest, it sits there as recoverable plaintext. Everything that can read etcd, read the Secret object through the API, read the container's environment, or read the manifest that created it, can read the secret.

Core Concept

Secure secret handling:
  Encoding      -> understood as encoding; real encryption is separate
  etcd          -> encryption-at-rest enabled via a KMS provider
  Source        -> secrets pulled at runtime from a manager (Vault/CSI/ESO)
  Manifests     -> no plaintext secrets; sealed/encrypted for GitOps
  Delivery      -> mounted as files with tight file modes, not env vars
  RBAC          -> get/list on secrets granted to almost no one
  ServiceAccount-> token automount disabled unless the pod needs the API
  Lifecycle     -> rotated on a schedule and after any suspected exposure

Secrets management failure:
  Encoding      -> base64 treated as if it were encryption
  etcd          -> unencrypted; a snapshot or node disk leaks everything
  Source        -> long-lived credentials hardcoded in images and code
  Manifests     -> passwords in YAML and ConfigMaps, committed to Git
  Delivery      -> secrets in env vars, visible in /proc, logs, crash dumps
  RBAC          -> broad get/list/watch on secrets across namespaces
  ServiceAccount-> default token automounted into every pod
  Lifecycle     -> never rotated; the same key has shipped for years

Why It's Critical for Kubernetes

Kubernetes concentrates several conditions that make secrets failures especially damaging:

Why Does This Matter?

Business Impact

Technical Impact

Technical Context

Common Secrets Failure Scenarios in Kubernetes

1. "base64 is encryption" (it is not)

# A Secret manifest - data values are base64, fully reversible
apiVersion: v1
kind: Secret
metadata:
  name: db-creds
type: Opaque
data:
  password: U3VwZXJTZWNyZXQxMjM=   # echo -n 'SuperSecret123' | base64

# Anyone who can read the object simply decodes it:
$ kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d
SuperSecret123

Risk: base64 provides zero confidentiality. It exists so binary values survive YAML, not to hide anything.

2. Unencrypted etcd

# Without encryption-at-rest, the raw value is in etcd as plaintext:
$ ETCDCTL_API=3 etcdctl get /registry/secrets/default/db-creds
...
SuperSecret123     # readable straight out of the datastore / a snapshot

Risk: etcd node disks, backups, and snapshots become credential dumps. Encryption-at-rest with a KMS provider is what makes the stored value opaque.

3. Secrets as environment variables

env:
  - name: DB_PASSWORD
    value: "SuperSecret123"        # plaintext in the manifest AND the pod spec

# Once running, the value is broadly visible inside the pod:
$ cat /proc/1/environ | tr '\0' '\n' | grep DB_PASSWORD
$ kubectl exec pod -- env | grep DB_PASSWORD

Risk: env vars leak into child processes, crash dumps, APM/error trackers, and logs. They are also visible in the pod spec to anyone who can read pods.

4. Secrets in images, manifests, and Git

# Dockerfile - baked into an image layer forever
ENV AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# ConfigMap - a "config" object holding a credential in plaintext
kind: ConfigMap
data:
  DATABASE_URL: "postgres://app:SuperSecret123@db:5432/prod"

# ...then git commit && git push  - now in history on every clone

Risk: image layers and Git history are permanent and widely distributed. A later "delete" does not remove the value from prior layers or commits.

5. Over-permissioned RBAC and automounted tokens

# A Role that lets a workload read every Secret in its namespace
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]

# And the default: every pod gets the ServiceAccount token mounted
#   /var/run/secrets/kubernetes.io/serviceaccount/token

Risk: a single compromised workload can enumerate secrets, and its automounted token is a credential for the API server it may never have needed.

Layers Where Secrets Leak

LayerTypical FailureConsequence
etcd datastoreNo encryption-at-rest; unprotected snapshotsWhole-cluster secret disclosure
Secret object / APIBroad get/list RBAC; base64 assumed safeCluster-wide credential reads
Pod runtimeSecrets as env vars; verbose loggingLeakage via /proc, logs, crash dumps
Container imageKeys baked into layers / DockerfilePermanent, distributable exposure
Source control (GitOps)Plaintext secrets in manifests/ConfigMapsExposure to everyone with repo/history access
CI/CDSecrets echoed into build logsDisclosure to anyone who can read pipelines
ServiceAccountDefault token automounted everywhereReady-made identity for a compromised pod

Real-World Impact

The incidents below are described as classes of failure that have recurred across many organisations. They are illustrative patterns, not specific attributed breaches, and deliberately avoid invented CVE numbers or precise statistics.

Case Class 1: Credentials Leaked Through Source Control

Failure:

Impact:

Root Cause: treating declarative config repositories as a safe home for secrets, with no pre-commit scanning and no separation between configuration and credentials.

Case Class 2: Secrets Recovered From Container Images

Failure:

Impact:

Root Cause: build pipelines that bake secrets into images instead of injecting them at runtime, with no image scanning to catch embedded keys.

Case Class 3: Whole-Cluster Exposure via etcd

Failure:

Impact:

Root Cause: relying on the default that Secrets are "in the cluster" as if that meant "protected," without enabling encryption-at-rest or securing snapshots.

Prevalence and Signals

Secrets Management Failures are consistently among the most common findings in Kubernetes security assessments, precisely because the insecure path is also the easiest path: base64, an env var, a value in YAML, a commit to Git.

Rather than cite precise figures (which vary by source and year), the defensible picture is:

Note: exact percentages and incident counts differ between reports. Treat any single figure as illustrative; the durable takeaway is that secrets leak by default, the leaks are permanent, and rotation—not deletion—is the remedy.

Common Misunderstandings

Myth 1: "Kubernetes Secrets are encrypted"

Reality: A Secret's data is base64-encoded, which is trivially reversible. Confidentiality at rest comes only from enabling etcd encryption-at-rest (ideally with a KMS provider); confidentiality in transit comes from TLS to the API server.

Myth 2: "It's fine because the repo is private"

Reality: Private repos are read by many people and many automated systems, are cloned to laptops and CI runners, and can be made public or leaked. Git history is permanent, so a private-repo secret must still be rotated once committed.

Myth 3: "Environment variables are a safe way to pass secrets"

Reality: Env vars are inherited by child processes, captured by crash reporters and APM agents, printed by debug logging, and readable via /proc and the pod spec. Mounted files with tight permissions are the safer default.

Myth 4: "A ConfigMap is just configuration, so a URL with a password is fine"

Reality: ConfigMaps have no confidentiality semantics at all—no separate RBAC treatment, no encryption story. A connection string with an embedded password in a ConfigMap is a plaintext secret in the clear.

Myth 5: "We deleted the secret, so we're safe"

Reality: Deleting a Secret object, a commit, or an image tag does not erase the value from etcd snapshots, prior Git commits, or earlier image layers. The only reliable response to exposure is to rotate the credential.

Myth 6: "Every pod needs its ServiceAccount token"

Reality: Most workloads never call the Kubernetes API. Automounting the token everywhere hands a free API credential to any compromised container. Disable automount by default and opt in only where it is genuinely required.

How Secrets Management Failures Differ from Related Issues

AspectSecrets Management (K08)Overly Permissive RBAC (K03)Supply-Chain (K02)
Root causeSensitive data mishandled/exposedIdentities granted too much accessUntrusted/compromised artifacts
Where it livesetcd, manifests, images, Git, podsRoles/Bindings, aggregation rulesImages, dependencies, build pipeline
Typical fixExternal manager, encryption, rotationLeast-privilege roles, scopingSigning, provenance, scanning
DetectionSecret scanning, etcd/RBAC auditRBAC review, can-i checksSBOM, image/attestation verification

Note the overlap: broad get/list on secrets is both an RBAC problem (K03) and a secrets-management problem (K08). The categories reinforce each other—least-privilege RBAC is part of protecting secrets.

Key Takeaways

  1. Encoding is not encryption—base64 hides nothing; encryption-at-rest and an external manager provide the real confidentiality.
  2. Delivery matters—prefer mounted files over environment variables to limit where a secret leaks.
  3. Keep secrets out of permanent stores—never in images, manifests, ConfigMaps, or Git; use sealed/encrypted forms for GitOps.
  4. Least privilege on secrets—get/list on secrets is powerful; grant it narrowly and disable default token automount.
  5. Assume exposure is permanent—rotate on a schedule and immediately after any suspected leak; deletion alone never suffices.

How to Identify if You're Vulnerable

Ask these questions about your cluster:

If you answered "no" or "not sure" to several of these, you likely have exploitable secrets exposure today.

Next Steps