Secret Management · Cloud Solutions

Module 6b · Secret Management

Security Champions · Module 6b

Secret Management — Cloud Solutions

AWS, Azure, Kubernetes, and HashiCorp Vault. Part 2 of 2.

Developers — focus on the decision tree and anti-patterns. The platform-specific details are reference material for when you need them.

Cover

Time to think · Part 1 recap

PART 1 RECAP

Secret lifecycle

01

02

03

04

05

06

Maturity spectrum

.env

Git

crypt

Store

Mgr

Vault

VCS approach (git-crypt, SOPS) vs Orchestrator approach (stores, vaults)

Part 1 covered what secrets are, their lifecycle, and two approaches. Part 2: specific cloud solutions and how to choose.

01 / 28 · Recap

Act 4 · AWS

Three tools, one purpose

AWS provides three services for managing secrets, each at a different level of complexity:

Parameter Store — the simplest option. Key-value storage with built-in encryption.

Secrets Manager — purpose-built for secrets. Adds rotation, programmatic access, and tighter integration.

KMS (Key Management Service) — enterprise-grade. HSM-backed, full key lifecycle management.

Three solutions for the same fundamental purpose is a design choice by AWS — but it creates a trap for teams that don't think ahead. Choose based on where you'll be in six months, not where you are today.

03 / 28 · AWS

Act 4 · AWS

The simple option

Parameter Store provides encrypted key-value storage integrated with IAM for access control and CloudTrail for audit logging.

Secrets are organized by namespace paths (e.g., /production/payment-service/db-password), and IAM policies can grant access at the path level — one team gets access to /production/payment-service/* while another gets /production/user-service/*.

Scalable, serverless, path-based permissions, integrates with CloudWatch and Lambda for event-driven workflows. No "secret zero" problem — services authenticate via IAM roles.

10,000 parameter limit per account, 4KB maximum per secret value, no built-in rotation mechanism.

04 / 28 · Parameter Store

Time to think · Architecture

AWS PARAMETER STORE

AWS boundary

🔐Parameter Store

↕

📦ECSread via IAM role

🖥EC2read via aws-sdk

⚡Lambdaread via IAM role

⬇ access keys required

💻Local machineoutside AWS boundary

ECS task roles are the cleanest pattern. The container inherits IAM permissions from its task definition — no credentials in environment variables or config files. Parameter Store access is just an IAM policy attachment.

EC2 instance profiles provide IAM role credentials automatically, but many teams bypass them with hardcoded access keys in code. If your EC2 code uses aws configure with long-lived keys, that's the gap to close first.

Lambda execution roles are automatically scoped. The risk isn't authentication — it's that Lambda functions often cache secret values. If you don't set a TTL on cached secrets, a rotated credential can stay in use beyond its intended lifetime.

The weakest link. Developer laptops with ~/.aws/credentials holding long-lived access keys. Use AWS SSO with temporary credentials instead. If a laptop is compromised, short-lived tokens limit the blast radius.

Inside AWS, services authenticate via IAM roles — no additional credentials needed. Outside AWS, access keys are required.

Click a service for how it authenticates

Time to think and reflect

Act 4 · AWS

The recommended option

Secrets Manager was created because KMS was too complex for most developer needs. It provides encryption by default, programmatic secret retrieval, and built-in rotation for RDS database credentials via Lambda functions.

It uses KMS under the hood for encryption — you can choose the default service key or specify your own customer-managed key.

Runtime secret retrieval (no secrets in config files), automated RDS rotation, powerful IAM integration, simpler than running a high-availability Vault cluster.

AWS lock-in, per-request pricing can add up at scale, CloudTrail logs management events but data access audit is limited.

06 / 28 · Secrets Manager

Act 4 · AWS

The enterprise option

KMS provides HSM-backed key management with full control over key types, rotation policies, and certificate management. It uses envelope encryption: you generate a data encryption key (DEK), encrypt your data with the DEK, then encrypt the DEK with a KMS master key. The plaintext DEK is immediately discarded.

This means your encrypted data and the encrypted DEK can be stored together. To decrypt, you first ask KMS to decrypt the DEK (which requires IAM permissions and STS token validation), then use the decrypted DEK locally.

KMS is powerful but complex. Use it when you need HSM-backed keys, certificate management, or compliance requirements that mandate hardware key storage. For most teams, Secrets Manager is sufficient.

07 / 28 · KMS

Time to think · Envelope encryption

ENVELOPE ENCRYPTION WITH KMS

Operator

AWS KMS

Server

Client

1

Operator generates DEK, encrypts secrets

ENCRYPT

2

Operator creates KMS Master Key

SETUP

3

Operator encrypts DEK with Master Key

WRAP

4

KMS returns encrypted DEK

WRAPPED

5

Operator discards plaintext DEK

DESTROY

6

Encrypted secrets + encrypted DEK → Server

STORE

7

Client requests encrypted bundle from Server

FETCH

8

Client asks KMS to decrypt DEK (STS + IAM)

AUTH

9

KMS returns decrypted DEK

UNWRAP

10

Client decrypts secrets locally with DEK

DECRYPT

The master key never leaves KMS. The data key exists in plaintext only during active use — then it's discarded.

Time to think and reflect

Act 4 · AWS

Which one to choose

Use Parameter Store when you need simple key-value storage, your secrets fit within the 4KB/10K limits, and you don't need automatic rotation.

Use Secrets Manager when you work with databases (especially RDS, where rotation is built in), need programmatic retrieval at runtime, or want a managed solution without operating your own infrastructure.

Use KMS when compliance requires HSM-backed keys, you need certificate management, or you require envelope encryption for large datasets.

09 / 28 · AWS recommendation

Time to think · Decision tree

AWS SECRET MANAGEMENT DECISION TREE

Simple key-value, under 10K params?

→ YES → Parameter Store

Need DB rotation (RDS)?

→ YES → Secrets Manager

Need HSM, certs, compliance?

→ YES → KMS

Multi-cloud environment?

→ YES → HashiCorp Vault

Think six months ahead. Migrating between AWS secret services costs time.

Time to think and reflect

Act 5 · Azure & Kubernetes

Azure's combined solution

Azure Key Vault is a single service that combines functionality equivalent to Parameter Store, Secrets Manager, and KMS. It stores keys, secrets, and certificates in one component, scoped to a specific resource group and region.

It offers both software-backed and hardware-backed (HSM) key options, with integration across Azure Functions, App Services, service principals, and managed identities.

Key Vault is component-based — it lives in a specific region. If your Key Vault is in US East but your resources are in UK South, you may have a compliance violation. Data residency requirements apply to the key storage location, not just the resources using the keys.

11 / 28 · Azure

Time to think · Managed identity

AZURE MANAGED SERVICE IDENTITY

①

Your Code (VM) → MSI Extension

calls localhost/oauth2/token

②

MSI Extension → Azure AD

authenticates with AAD

③

Your Code → Key Vault / ARM

uses token to access

Managed Service Identity eliminates the "secret zero" problem. Azure injects and rotates credentials without your code managing any secrets directly.

Time to think and reflect

Act 5 · Kubernetes

The container world

Kubernetes stores secrets in etcd and makes them available to pods as mounted volumes or environment variables. The flow: admin creates a secret via kubectl → API Server stores it in etcd → when a pod is scheduled, the secret is provisioned to the node → mounted into the pod.

Secrets are namespace-scoped (only accessible within the same namespace). They use tmpfs (deleted when the pod terminates). Maximum size is 1MB. Mounting a secret as a volume masks the directory's existing contents.

The most critical fact about Kubernetes secrets: they are base64-encoded by default, not encrypted. Base64 is an encoding format, not a security mechanism. Anyone with etcd access can read every secret in plaintext.

13 / 28 · K8s secrets

Time to think · K8s architecture

KUBERNETES SECRET FLOW

Master

⌨️ kubectl → API Server

🔄 Scheduler

📋 Replication Controllers

💾 etcd — all secrets stored here

⇄

Worker Nodes

🔧 kubelet

🐳 Container Engine

📦 Pod 1

📦 Pod 2

📦 Pod N

Without RBAC, any authenticated user can read all secrets. Default K8s installs allow broad access. Implement namespace-scoped Roles and ClusterRoles that restrict get/list on secrets to specific service accounts.

The single biggest risk in a K8s cluster. etcd stores all secrets as base64 — readable by anyone with direct access. A compromised etcd backup exposes every secret. Enable EncryptionConfiguration for encryption at rest and restrict network access to the API Server only.

Root access on a worker node = secret access. An attacker with root can impersonate the kubelet and request secrets for any pod scheduled on that node. Disable anonymous authentication and require TLS client certificates for kubelet communication.

All secret operations go through the API Server and are stored in etcd. Securing both is essential.

Red item = single point of secret exposure

Click highlighted items for attack vectors

Time to think and reflect

Act 5 · Kubernetes

What to lock down

API Server — by default, any user who can access it can read all secrets across all namespaces. Implement RBAC to restrict secret access per namespace and per role.

etcd — write access to etcd is equivalent to root access on the entire cluster. Enable encryption at rest. Restrict network access to etcd to the API Server only.

Worker nodes — root access on a node allows reading secrets by impersonating the kubelet. Disable anonymous authentication on kubelet. Use TLS for all communication.

For teams outgrowing K8s native secrets: use the External Secrets Operator to sync secrets from an external store (Vault, AWS Secrets Manager) into Kubernetes secrets. Your pods consume them the same way, but the source of truth moves to a more capable system.

15 / 28 · K8s security

Knowledge check

Your Kubernetes secrets are stored in etcd without encryption at rest. A developer with kubectl access can read secrets across all namespaces. What should you fix first?

C — Both controls address different attack vectors. RBAC without encryption means a compromised etcd backup exposes everything. Encryption without RBAC means every developer can still read all secrets through kubectl. Vault migration (D) is a longer-term solution, but the immediate fix is applying both RBAC and etcd encryption.

16 / 28 · Quiz

Act 5 · Kubernetes

When K8s secrets are enough

Use Kubernetes native secrets when your secrets don't change often, are used exclusively within the cluster, and your team has implemented RBAC and etcd encryption.

Move to an external store (Vault with K8s auth method, or External Secrets Operator with cloud secret managers) when secrets need to be shared across clusters, across cloud providers, or when you need rotation, fine-grained audit trails, or compliance-grade access controls.

Practical rule

K8s native secrets are base64-encoded, not encrypted. They're fine for non-sensitive config. For real secrets, use an external vault with CSI driver injection.

17 / 28 · K8s recommendation

Act 6 · Vault

The platform-agnostic option

HashiCorp Vault is a dedicated secret management platform that works across AWS, Azure, GCP, and on-premises infrastructure. It can use cloud Key Vaults as storage backends while providing a unified API and policy layer on top.

At scale, it's often cheaper than cloud-native Secrets Manager because you pay for infrastructure rather than per-request. It supports backup and replication across regions and clouds using Consul as the coordination layer.

Multi-cloud environments, cloud migration plans, need for cross-platform secret management, requirement for detailed audit trails, or organizations that want to avoid vendor lock-in for their secret infrastructure.

18 / 28 · Vault

Time to think · Vault flow

HASHICORP VAULT — ACCESS FLOW

Operator

HashiCorp Vault

Client Node

1

Operator starts with plaintext secrets

SEED

2

Operator seeds policies (read/write per path)

POLICY

3

Operator associates auth roles with policies

ROLE

4

Operator seeds secrets by path into Vault

STORE

5

Client requests authentication from Vault

AUTH

6

Vault logs request → returns access token

📝 LOGGED

7

Client requests secret by path with token

FETCH

8

Vault logs request → returns secret if ACL permits

📝 LOGGED

Every access is logged. Every secret is policy-controlled. This is what a complete audit trail looks like.

Time to think and reflect

Time to think · K8s + Vault

KUBERNETES-VAULT INTEGRATION

Controller

Init Container

App Container

1

Controller watches API server for pods with vault-approle annotation

WATCH

2

Controller creates a secret_id using the role from pod annotation

CREATE

3

Controller pushes wrapped secret_id to Init Container over HTTPS

PUSH

4

Init Container unwraps the secret_id

UNWRAP

5

Init Container retrieves auth token using role_id + secret_id from Vault

AUTH

6

Init Container writes auth token to shared volume

WRITE

7

App Container reads auth token from the volume

USE

The init container pattern ensures secrets never appear as environment variables. They're injected via volume at pod startup and are ephemeral.

Time to think and reflect

Knowledge check

Your team runs services on both AWS and Azure. You need to store API keys, rotate database credentials monthly, and provide audit trails for compliance. Which solution fits best?

D — HashiCorp Vault. The multi-cloud requirement eliminates single-cloud solutions (A, B). Git-based approaches (C) don't support rotation or audit trails. Vault operates across both clouds, supports rotation, provides comprehensive audit logging, and avoids vendor lock-in.

21 / 28 · Quiz

Act 7 · Practice

What not to do

In source code, config files, container images. The path of least resistance that becomes a long-term liability. Use parameter stores or vaults from day one.

Visible to any process, logged by crash reporters, exposed in Kubernetes pod specs. Use mounted volumes or runtime retrieval instead.

API keys created years ago and never rotated. When compromised, the exposure window is unlimited. Schedule rotation for all long-lived credentials.

Multiple people or services using the same credentials. When something goes wrong, you can't determine who did it. One credential per identity, always.

The most common and most dangerous anti-pattern. You won't. Schedule it now or accept that it won't happen.

22 / 28 · Anti-patterns

★ Best practice

Secret health check

Best practice

Running a periodic secret health check using a standardized checklist. Use this quarterly to assess your secret management posture:

This checklist works as a standalone tool — bookmark or screenshot it. Run it once a quarter and track how many items pass over time.

All secrets encrypted at rest and in transit. No plaintext storage anywhere.

Role-based access. No shared master keys. Principle of least privilege applied.

Scheduled rotation for all long-lived credentials. Personnel changes trigger immediate rotation.

The system can answer "who accessed which secret, when" for any date in the past 90 days.

Source code, environment variables, CI/CD logs, and container images are free of credentials.

Secret storage backup exists and has been tested. You know how long full restoration takes.

23 / 28 · Best practice

Practice

Your assignment

List every location where secrets are stored in your project today. All of them — git variables, parameter stores, env files, hardcoded values.

Pick one secret. Can you determine who accessed it last week? If not, that's your first gap.

When was the last time any credential in your project was rotated? If you can't answer, that's your second gap.

Use the health check from the previous slide. How many items pass?

Identify the single highest-risk anti-pattern in your project and address it this sprint.

24 / 28 · Assignment

Time to think · Decision map

CHOOSING YOUR SOLUTION

Single cloud — AWS?

→ simple → Parameter Store

→ rotation → Secrets Manager

→ HSM/certs → KMS

Single cloud — Azure?

→ Azure Key Vault

Kubernetes only?

→ simple → K8s Secrets + RBAC + etcd enc

→ need more → External Secrets Operator

Multi-cloud or migrating?

→ HashiCorp Vault

Best for: simple configs, feature flags, non-rotating secrets under 4KB. Trade-off: no rotation, limited size. Migrate to Secrets Manager when you outgrow it.

Best for: database credentials, API keys needing rotation, most AWS-native teams. Trade-off: per-request pricing, AWS lock-in. The recommended starting point for most.

Best for: compliance (PCI DSS, HIPAA), certificate management, encrypting large datasets. Trade-off: operational complexity. Don't use unless compliance requires it.

Best for: Azure-native teams needing keys + secrets + certs in one place. Trade-off: region-scoped (compliance risk), Azure lock-in. MSI integration is its strongest feature.

Best for: internal-only secrets, small clusters, teams with RBAC + etcd encryption in place. Trade-off: base64 not encryption, no audit trail, no rotation. The floor, not the ceiling.

Best for: teams already on K8s but needing external secret sources. Trade-off: adds complexity but keeps pods consuming secrets the same way. Bridges K8s native to cloud-managed.

Best for: multi-cloud, cross-platform, or teams planning cloud migration. Trade-off: operational overhead to run Vault itself. Cheaper at scale than per-request cloud pricing.

Start where you are. Know where you're going. The best solution is the one your team will actually use.

Click any solution for trade-offs

Time to think and reflect

Summary

What you covered

1. AWS provides three options at increasing complexity — Parameter Store for simple needs, Secrets Manager for most teams, KMS for enterprise/compliance.

2. Azure Key Vault combines all three functions. Managed Service Identity eliminates the "secret zero" problem.

3. Kubernetes secrets are base64-encoded, not encrypted. RBAC and etcd encryption are both required for adequate security.

4. HashiCorp Vault is the platform-agnostic answer for multi-cloud environments, providing unified policy management and audit trails.

5. Choose based on where you'll be in six months. Start with the simplest adequate solution and plan the migration path.

26 / 28 · Summary

Module 6b · Your results

YOUR RESULTS

27 / 28 · Results

Security Configuration & Access Control

Why configurations go wrong, how to establish baselines, and how to enforce access control policies.

28 / 28 · Bridge