Secret Management · Security Champions

Module 6a · Secret Management

Security Champions · Module 6a

Secret Management

Understanding, storing, and managing secrets across your infrastructure. Part 1 of 2 — Fundamentals & Approaches.

Cover

Your journey · Program map

Your journey so far

9 modules. One toolkit. You are at Module 6.

Context

Your toolkit · So far

What you already have

Modules 1–5

M1–2

Champion role, sprint workflow, business language

The operating system for everything that follows

M3

OWASP Top 10 risks through five lenses

You know what to look for and where

M4

Threat modeling: STRIDE → DREAD → tickets

You can find and prioritize threats systematically

M5

Supply chain: three attack surfaces, SBOM, SLSA

You know how to verify what your pipeline ships

Context

Module 6 · The quiet risk

The quiet risk

Module 5 covered your supply chain. But running through all of it is a quieter risk: secrets.

API keys, database credentials, tokens, certificates — they end up in places they shouldn't: .env files in git, Slack messages, CI variables that survive migrations.

By the end, you'll own three things:

1

Map where your secrets are

Know what exists before you can protect it

2

Rotation when people leave

The credential that outlives its owner is the one that gets exploited

3

Quarterly health check

A standing habit, not a one-time audit

Context

Act 1 · Why it matters

This keeps happening

In 2017, Gizmodo published a headline: users were storing passwords in Trello boards — publicly visible ones. The boards were indexed by search engines. Credentials for production systems, findable through Google.

This isn't ancient history. People continue to take insecure shortcuts: hardcoding secrets in source code, storing them in container images and config files, sharing them via email, Slack, and shared folders.

As your architecture grows — more services, more integrations, more APIs — the number of secrets multiplies. Managing them becomes a dedicated discipline.

05 / 27 · Why it matters

Act 1 · Why it matters

The definition

A secret is any piece of information that, if exposed, would grant unauthorized access to a system or data. API keys, passwords, certificates, encryption keys, tokens, connection strings — all secrets.

The important distinction: an identifier tells a system who you are. A secret proves that you are who you claim to be. Your login is an identifier. Your password is a secret.

The test is simple: if leaking this value grants someone access to something, it's a secret.

🔑API Keys

🔐Passwords

📜Certificates

🎫Tokens

Remember

A secret is anything that grants access or proves identity. If leaking it would let someone do something they shouldn't — it's a secret.

06 / 27 · Definitions

Act 1 · Why it matters

Who (or what) has secrets

Secrets exist for three types of identity:

People — users, merchants, developers. Identifier: login or email. Secret: password, MFA token. The most familiar type.

Software — services, APIs, AI agents. Identifier: client ID or service name. Secret: API key, certificate, OAuth client secret. Sometimes the API key serves as both identifier and secret — if leaking it grants access, treat it as a secret.

Hardware — HSM, TPM, physical tokens. Keys stored in hardware that cannot be extracted even with physical access. Cloud providers offer this: AWS KMS is backed by HSM. Compliance frameworks (HIPAA, PCI DSS) sometimes require hardware-backed key storage.

07 / 27 · Identity types

Time to think · Identity types

THREE IDENTITY TYPES

👤

People

login + password
email + MFA
physical badge

⚙️

Software

API key
certificate
OAuth secret
connection string

🔐

Hardware

HSM serial
TPM key
physical token
smart card

Most teams manage people secrets well — password policies, MFA rollouts, SSO integration. The risk isn't that these secrets are unmanaged. It's that they create a false sense of maturity. "We have SSO" doesn't mean your API keys are safe.

Software secrets are the biggest blind spot. API keys shared over Slack, connection strings hardcoded in config, OAuth secrets committed to Git. These often outlive the developer who created them, and nobody tracks who has a copy.

Hardware secrets are rare but high-value. If your compliance framework requires HSM-backed keys (PCI DSS, HIPAA), managing these becomes a first-class concern. Cloud KMS services abstract the hardware away — but the key lifecycle management remains your responsibility.

Every identity type has secrets. Most teams manage people's secrets well. Software and hardware secrets are where the gaps appear.

Click a card for deeper context

Time to think and reflect

Act 1 · Why it matters

Time is a security factor

Long-lived secrets persist for months or years: API keys, certificates, service account passwords, database connection strings. They're convenient because they don't expire — and dangerous for the same reason.

Ephemeral secrets last minutes or hours: JWT tokens, session IDs, IAM role temporary credentials. Their short lifespan limits the exposure window if they're stolen.

Ephemeral is inherently safer, but not all ephemeral tokens are well-protected. IAM role tokens, for example, remain valid until their expiration time regardless of whether the session that created them is still active.

09 / 27 · Lifespan

Time to think · Auth flow

HOW SECRETS BECOME TOKENS

👤

Identity

person or service

→

ID+🔑

Identifier + Secret

→

🛡

Access Broker

Azure AD, Okta, etc.

→

🎟

Token

signed JWT

→

Service A

Service B

Service C

After authentication, the secret transforms into a token. The token becomes the new asset to protect.

Below the JWT icon: Signed, contains claims, verified at each service

Time to think and reflect

Act 1 · Why it matters

Can you trace who used a secret?

You find suspicious activity linked to a token ID in your logs. The critical question: can you trace it back to a specific person or service?

A refresh token — usually yes, through a database lookup that ties it to the issuing session.

An access token — maybe. If claims are logged and the token hasn't expired, you can read its contents. But many systems don't log token payloads.

The broader question: does your secret management solution support audit trails? Many solutions handle storage and distribution well. Far fewer answer "who accessed this secret at 3pm last Tuesday."

11 / 27 · Visibility

Knowledge check

A microservice communicates with a payment processor using a single API key that serves as both identification and authentication. Is the API key a secret?

B — Yes. When a single value serves as both "who you are" and "prove it," leaking that value grants full access. It is a secret and should be stored, rotated, and audited accordingly.

12 / 27 · Quiz

Act 2 · Lifecycle

More than storage

Secret management is the discipline of handling secrets across their full lifecycle. It covers three areas:

Storage — where secrets live, how they're encrypted, who can access them.

Lifecycle — creation, modification, distribution, rotation, and destruction of secrets. Most solutions handle the first three well. Rotation and destruction are where gaps appear.

Recovery and remediation — what happens when a secret is compromised? How quickly can you rotate it? What happens if your secret storage itself fails?

The ideal secret management solution doesn't exist. It should be tailored to your specific needs — and the more comprehensive the solution, the more complex and expensive it becomes.

13 / 27 · Lifecycle

Time to think · Lifecycle

THE SECRET LIFECYCLE

01

Create

generate securely

02

Store

encrypt at rest

03

Distribute

secure channel

04

Use

audit access

05

Rotate

scheduled + triggered

06

Destruct

confirmed deletion

Most solutions handle storage and distribution. The gaps are in rotation, auditing, and destruction.

Orange stages = where most teams skip

Time to think and reflect

Act 2 · Lifecycle

What to look for

Security requirements: encryption at rest and in transit, fine-grained access control, comprehensive access logs, separation of duties.

Usability requirements: central location for all secrets, easy management interface, straightforward integration with existing tools, scalability as the number of secrets grows.

The more ideal the solution, the more expensive it is — both in licensing and in operational overhead. The practical approach: define what you need today and where you'll be in 6 months, then choose the solution that covers that range without over-engineering.

15 / 27 · Requirements

Act 2 · Real case

When the secret store disappears

A team ran self-hosted GitLab CE with all secrets stored in GitLab CI/CD variables — API keys, Azure service principal credentials, database connection strings. Encrypted at rest and in transit. Seemed adequate.

Then they migrated to the commercial edition. The migration crashed. They rebuilt from scratch, uploaded code from local repositories. But the GitLab variables — all the secrets — were gone.

Days of work to recreate and rotate every credential. Azure SPNs, API keys, service connections — all had to be regenerated, redistributed, and re-tested across every environment.

Your secret management solution is also your disaster recovery plan. If it disappears, can you rebuild? How long would it take?

16 / 27 · Case study

Time to think · Secret sprawl

WHERE SECRETS END UP

GitLab CI variables

.env files

Helm values

Docker env vars

Hardcoded in config

Slack messages

~/.aws/credentials

CI/CD pipeline logs

Trello boards

Shared drives

Code comments

Clipboard history

Encrypted at rest, but accessible to anyone with project maintainer access. If you migrate GitLab instances, these variables don't transfer — as the case study showed.

Often .gitignored but not always. Developers copy them between machines, share them over chat. No audit trail of who has which version.

Helm values.yaml files can contain secrets in plaintext. Use helm-secrets plugin or External Secrets Operator to pull from a vault instead.

Visible to any process inside the container and often logged by orchestrators. Prefer mounted volumes or runtime retrieval.

The most dangerous pattern. Persists in git history forever, even after deletion. Requires full credential rotation and history rewrite to remediate.

Searchable by anyone in the workspace. Slack retains message history. One shared API key in a channel is a credential visible to every member, past and future.

Long-lived access keys on developer laptops. If the laptop is compromised, so are all the AWS resources those keys can access. Use SSO and short-lived credentials instead.

Secrets leaked via echo, debug output, or error messages. Pipeline logs are often accessible to the entire team and may be retained for months.

Credentials pasted into cards "temporarily." Board visibility changes and nobody remembers which cards contain secrets.

Shared docs with access credentials, often with broad permissions. No version control, no access logging, no rotation triggers.

TODO comments with temporary passwords, debug keys left behind. Code review should flag these, but often doesn't until a scanner finds them.

Clipboard managers retain copied secrets. Cloud-synced clipboards can spread credentials across devices without any awareness.

Secrets migrate. They start in one place and end up in twelve. Your management strategy must account for all of them.

Red cards = highest-risk locations

Click any card for why it's risky

Time to think and reflect

Knowledge check

Your team stores API keys in GitLab CI/CD variables. A developer leaves the company. Their laptop is returned and wiped. What's the immediate risk?

B — Cached copies. Developers who used the CI/CD variables likely had the secret values on their machines at some point (environment variables during builds, local configs for testing). Without rotating the secrets after their departure, their knowledge of those values continues to grant access. Personnel changes must trigger secret rotation.

18 / 27 · Quiz

Act 3 · Approaches

VCS vs orchestrator

Before comparing specific tools, there are two fundamental approaches to managing secrets:

Version control approach — secrets are encrypted and stored alongside code in Git. Changes go through merge requests. Tools like git-crypt or SOPS handle the encryption.

Orchestrator approach — secrets are stored in a dedicated service (cloud parameter stores, HashiCorp Vault, Kubernetes secrets). Applications retrieve secrets at runtime through API calls.

Each has clear trade-offs. The VCS approach is simpler to start with. The orchestrator approach scales better and provides the audit trails that VCS-based approaches lack.

19 / 27 · Approaches

Act 3 · Approaches

Secrets in Git

With git-crypt, you specify which files should be encrypted in .gitattributes. During git push, those files are encrypted automatically. When someone clones the repository, encrypted files remain encrypted until git-crypt unlock is run with the correct key.

Easy to start, encryption at rest and in transit, version history for secret changes, code review on changes via merge requests, some compartmentalization through access to decryption keys.

No access history (you can't tell who read a secret), difficult rotation (changing a secret requires a new commit), merge conflicts when parallel MRs change different secrets, repository tampering risk. The practical assessment: only good for small projects.

20 / 27 · git-crypt

Time to think · git-crypt flow

GIT-CRYPT FLOW

1

Initialize git + git-crypt in local repository

SETUP

2

Specify encrypted files in .gitattributes

SETUP

3

git push — files encrypted automatically

ENCRYPT

4

Remote stores encrypted blobs only

ENCRYPTED

5

Clone returns encrypted files

ENCRYPTED

6

git-crypt unlock with correct key

DECRYPT

7

Modify secrets locally → git push re-encrypts

ENCRYPT

Transparent encryption on push, decryption on unlock. Simple, but with no access audit trail.

Time to think and reflect

Act 3 · Approaches

Dedicated secret stores

The orchestrator approach stores secrets in a dedicated service. Applications request secrets at runtime through authenticated API calls. Kubernetes secrets, Docker Swarm secrets, and cloud parameter stores all follow this model.

No code changes needed to manage secrets, no decryption keys distributed per developer node, can offload encryption to KMS, auditable access, single source of truth for the current secret value.

Trust bootstrapping required — the application needs initial credentials to authenticate to the secret store (the "secret zero" problem). Orchestrator lock-in if you rely on platform-specific APIs.

The orchestrator approach is where the industry is heading. The remaining question is which orchestrator — cloud-native or platform-agnostic.

22 / 27 · Orchestrator

Time to think · Maturity

THE SECRET MANAGEMENT SPECTRUM

.env

Git vars

git-crypt

Param Store

Secrets Mgr

KMS

Vault

HSM

← Easy to start. Hard to scale. Complex to set up. Scales to enterprise. →

▲ Where is your team right now?

Plaintext files on disk. No encryption, no access control, no audit trail. Fine for local development with fake data. Never acceptable for real credentials.

CI/CD variables (GitLab, GitHub Actions). Encrypted at rest, scoped to projects. Better than .env files, but no access audit and limited rotation support.

Encrypted files in Git. Secrets travel with code and go through merge requests. Good for small teams. Breaks down with parallel changes and provides no access log.

AWS key-value store. Path-based access control via IAM. Scales well, but no built-in rotation and 4KB/10K limits.

The sweet spot for most AWS teams. Runtime retrieval, built-in RDS rotation, KMS encryption under the hood. Per-request pricing can add up at scale.

HSM-backed key management. Envelope encryption, full key lifecycle, certificate management. Required for compliance frameworks mandating hardware key storage.

Platform-agnostic secret management. Works across AWS, Azure, GCP, and on-prem. Complete audit trails, dynamic secrets, policy-based access. Operational overhead to run.

Dedicated hardware for key storage. Keys never leave the device. Maximum security, maximum cost and complexity. Required by some financial and government regulations.

Every team is somewhere on this spectrum. The question isn't where you are — it's where you need to be in six months.

Click any segment to learn more

Time to think and reflect

Knowledge check

Your team manages secrets encrypted in Git via merge request workflow. Two developers submit parallel MRs that each modify different secrets in the same encrypted file. What happens?

B — Merge conflict on encrypted file. Encrypted files are opaque to Git's merge logic. When two MRs modify the same encrypted file, Git sees a binary conflict that can't be auto-resolved. The developer resolving the conflict must decrypt, merge manually, and re-encrypt — a fragile process at scale. This is a fundamental limitation of the VCS approach.

24 / 27 · Quiz

Summary · Part 1

What you covered

01: A secret is anything that grants access if leaked. The test: if it's both identifier and proof of ownership, it's a secret.

02: Three identity types have secrets: people, software, and hardware. Software secrets are the most commonly mismanaged.

03: The secret lifecycle has six stages — create, store, distribute, use, rotate, destruct. Most teams handle the first three but skip rotation and destruction.

04: The VCS approach (git-crypt) works for small projects. The orchestrator approach (dedicated stores) scales better and provides audit trails.

05: Your secret management solution is also your disaster recovery plan. Plan for the scenario where it fails.

Your role as a Champion: you own three things regarding secrets — knowing where they live in your project right now (the sprawl map), ensuring rotation happens when someone leaves the team, and running the health checklist quarterly. You don't manage the vault infrastructure — you ensure your team's secrets are in the vault, not in Slack.

Run secret scanning in your pipeline as a pre-commit hook or CI step. gitleaks — fast, regex-based, catches most patterns. truffleHog — scans full git history including deleted commits. detect-secrets (Yelp) — lower false-positive rate, good for large repos. GitHub Advanced Security — native integration if you're on GitHub Enterprise. Adding any one of these to your pipeline is a single-sprint task.

25 / 27 · Summary

Module 6a · Your results

YOUR RESULTS

26 / 27 · Results

Next · Part 2

Part 2: AWS Parameter Store, Secrets Manager, and KMS. Azure Key Vault. Kubernetes secrets. HashiCorp Vault. And how to choose between them.

27 / 27 · Bridge