Secrets Management
Centralized secrets management with HashiCorp Vault and Kubernetes Secrets.
Overview
All secrets are managed securely: - Secrets Engine: HashiCorp Vault (primary) - Kubernetes: Encrypted at rest with KMS - Access Control: RBAC with audit logging - Rotation: Automatic rotation every 90 days - Compliance: NIST 800-53, FedRAMP, SOC 2
Architecture
graph TB
A[Application] --> B[Vault Agent]
B --> C[HashiCorp Vault]
C --> D[AWS KMS/HSM]
A --> E[Kubernetes Secrets]
E --> F[etcd Encryption]
HashiCorp Vault
Vault Setup
Installation:
# Install Vault
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install vault
# Start Vault server
vault server -config=/etc/vault/config.hcl
config.hcl:
storage "raft" {
path = "/opt/vault/data"
node_id = "vault-1"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_cert_file = "/etc/certs/vault.crt"
tls_key_file = "/etc/certs/vault.key"
}
seal "awskms" {
region = "us-west-2"
kms_key_id = "arn:aws:kms:us-west-2:123456789012:key/abc123"
}
api_addr = "https://vault.local.bluefly.io:8200"
cluster_addr = "https://vault.local.bluefly.io:8201"
ui = true
Secret Engines
KV Secrets Engine (v2)
# Enable KV secrets engine
vault secrets enable -path=secret kv-v2
# Write secret
vault kv put secret/database/postgres \
username=postgres_user \
password=secure-password-123
# Read secret
vault kv get secret/database/postgres
# List secrets
vault kv list secret/database
Database Dynamic Secrets
# Enable database secrets engine
vault secrets enable database
# Configure PostgreSQL connection
vault write database/config/postgresql \
plugin_name=postgresql-database-plugin \
allowed_roles="readonly,readwrite" \
connection_url="postgresql://{{username}}:{{password}}@postgres.local:5432/llm_platform" \
username="vault_admin" \
password="vault-admin-password"
# Create role for dynamic credentials
vault write database/roles/readwrite \
db_name=postgresql \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
default_ttl="1h" \
max_ttl="24h"
# Generate dynamic credentials
vault read database/creds/readwrite
Output:
Key Value
--- -----
lease_id database/creds/readwrite/abc123
lease_duration 1h
lease_renewable true
password A1a-random-password-xyz
username v-token-readwrite-abc123
Transit Secrets Engine
# Enable transit engine
vault secrets enable transit
# Create encryption key
vault write -f transit/keys/bluefly-platform
# Encrypt data
vault write transit/encrypt/bluefly-platform \
plaintext=$(echo "sensitive-data" | base64)
# Decrypt data
vault write transit/decrypt/bluefly-platform \
ciphertext="vault:v1:encrypted-data-here"
# Rotate key
vault write -f transit/keys/bluefly-platform/rotate
Access Policies
policies/developer.hcl:
# Read application secrets
path "secret/data/app/*" {
capabilities = ["read", "list"]
}
# Read database credentials
path "database/creds/readwrite" {
capabilities = ["read"]
}
# Encrypt/decrypt with transit
path "transit/encrypt/bluefly-platform" {
capabilities = ["update"]
}
path "transit/decrypt/bluefly-platform" {
capabilities = ["update"]
}
Apply Policy:
# Create policy
vault policy write developer policies/developer.hcl
# Assign policy to user
vault write auth/userpass/users/developer \
password=dev-password \
policies=developer
Authentication Methods
AppRole (for services)
# Enable AppRole
vault auth enable approle
# Create role
vault write auth/approle/role/agent-brain \
secret_id_ttl=24h \
token_ttl=1h \
token_max_ttl=4h \
policies=agent-brain
# Get role ID
vault read auth/approle/role/agent-brain/role-id
# Generate secret ID
vault write -f auth/approle/role/agent-brain/secret-id
Application Login:
import Vault from 'node-vault';
const vault = Vault({
apiVersion: 'v1',
endpoint: 'https://vault.local.bluefly.io:8200'
});
// Login with AppRole
const result = await vault.approleLogin({
role_id: process.env.VAULT_ROLE_ID,
secret_id: process.env.VAULT_SECRET_ID
});
// Set token
vault.token = result.auth.client_token;
// Read secret
const secret = await vault.read('secret/data/app/config');
console.log(secret.data.data);
Kubernetes Auth
# Enable Kubernetes auth
vault auth enable kubernetes
# Configure
vault write auth/kubernetes/config \
kubernetes_host=https://kubernetes.default.svc:443 \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token
# Create role
vault write auth/kubernetes/role/agent-brain \
bound_service_account_names=agent-brain \
bound_service_account_namespaces=llm-platform \
policies=agent-brain \
ttl=1h
Pod Annotation:
apiVersion: v1
kind: Pod
metadata:
name: agent-brain
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "agent-brain"
vault.hashicorp.com/agent-inject-secret-config: "secret/data/app/agent-brain"
spec:
serviceAccountName: agent-brain
containers:
- name: agent-brain
image: agent-brain:latest
env:
- name: DATABASE_URL
value: "file:///vault/secrets/config"
Kubernetes Secrets
Encrypted at Rest
kube-apiserver:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded-32-byte-key>
- identity: {}
Apply Configuration:
# Create encryption config
kubectl create secret generic -n kube-system \
encryption-config --from-file=encryption-config.yaml
# Update kube-apiserver
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml
Sealed Secrets
Install Sealed Secrets Controller:
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/controller.yaml
Create Sealed Secret:
# Create regular secret
kubectl create secret generic my-secret \
--from-literal=password=super-secret \
--dry-run=client -o yaml > secret.yaml
# Seal it
kubeseal < secret.yaml > sealed-secret.yaml
# Apply sealed secret (safe to commit)
kubectl apply -f sealed-secret.yaml
External Secrets Operator
Install ESO:
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
external-secrets/external-secrets \
-n external-secrets-system \
--create-namespace
SecretStore:
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
namespace: llm-platform
spec:
provider:
vault:
server: "https://vault.local.bluefly.io:8200"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "external-secrets"
serviceAccountRef:
name: "external-secrets"
ExternalSecret:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: database-credentials
namespace: llm-platform
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: postgres-credentials
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: secret/database/postgres
property: username
- secretKey: password
remoteRef:
key: secret/database/postgres
property: password
Secret Types
API Keys
Storage in Vault:
vault kv put secret/api-keys/openai \
api_key=sk-proj-abc123... \
organization=org-xyz789
Usage:
const secret = await vault.read('secret/data/api-keys/openai');
const apiKey = secret.data.data.api_key;
const openai = new OpenAI({ apiKey });
Database Credentials
Dynamic Credentials (recommended):
// Get dynamic credentials
const creds = await vault.read('database/creds/readwrite');
const pool = new Pool({
host: 'postgres.local.bluefly.io',
database: 'llm_platform',
user: creds.data.username,
password: creds.data.password
});
// Credentials auto-expire after 1 hour
// Vault automatically revokes them
OAuth Client Secrets
vault kv put secret/oauth/gitlab \
client_id=abc123 \
client_secret=secret-xyz789 \
redirect_uri=https://llm.bluefly.io/auth/callback
Encryption Keys
# Store in transit engine (never leaves Vault)
vault write -f transit/keys/data-encryption
# Encrypt
vault write transit/encrypt/data-encryption \
plaintext=$(echo "data" | base64)
# Decrypt (key material never exposed)
vault write transit/decrypt/data-encryption \
ciphertext="vault:v1:..."
Secret Rotation
Automatic Rotation
Rotation Schedule: - API keys: 90 days - Database passwords: 30 days - OAuth secrets: 90 days - Encryption keys: 90 days
Rotation Script:
class SecretRotationService {
async rotateSecret(path: string): Promise<void> {
// 1. Generate new secret
const newSecret = crypto.randomBytes(32).toString('hex');
// 2. Write to Vault with new version
await this.vault.write(`secret/data/${path}`, {
data: { value: newSecret }
});
// 3. Update applications (gradual rollout)
await this.updateApplications(path, newSecret);
// 4. Audit log
await this.auditLog.log({
event: 'secret_rotated',
path,
timestamp: new Date()
});
// 5. Delete old version after grace period
setTimeout(async () => {
await this.vault.delete(`secret/metadata/${path}`);
}, 24 * 60 * 60 * 1000); // 24 hours
}
}
Database Password Rotation
// Vault automatically handles rotation
vault write database/rotate-root/postgresql
// Application automatically gets new credentials
const creds = await vault.read('database/creds/readwrite');
Secret Injection
Environment Variables
Vault Agent:
vault {
address = "https://vault.local.bluefly.io:8200"
}
auto_auth {
method {
type = "approle"
config = {
role_id_file_path = "/vault/role-id"
secret_id_file_path = "/vault/secret-id"
}
}
}
template {
source = "/vault/templates/config.env.tmpl"
destination = "/app/.env"
}
config.env.tmpl:
{{ with secret "secret/data/app/config" }}
DATABASE_URL=postgresql://{{ .Data.data.db_user }}:{{ .Data.data.db_password }}@postgres.local:5432/llm_platform
OPENAI_API_KEY={{ .Data.data.openai_key }}
{{ end }}
File Injection
Sidecar Container:
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
initContainers:
- name: vault-agent
image: vault:1.15
command: ["vault", "agent", "-config=/vault/config.hcl"]
volumeMounts:
- name: vault-config
mountPath: /vault
- name: secrets
mountPath: /secrets
containers:
- name: app
image: myapp:latest
volumeMounts:
- name: secrets
mountPath: /etc/secrets
readOnly: true
Audit & Compliance
Audit Logging
Enable Audit:
vault audit enable file file_path=/var/log/vault/audit.log
Audit Events:
{
"time": "2025-01-15T10:00:00Z",
"type": "response",
"auth": {
"client_token": "hmac-sha256:abc123",
"accessor": "hmac-sha256:xyz789",
"display_name": "approle",
"policies": ["default", "agent-brain"]
},
"request": {
"operation": "read",
"path": "secret/data/app/config"
},
"response": {
"secret": true,
"data": {
"metadata": {
"created_time": "2025-01-15T09:00:00Z",
"version": 1
}
}
}
}
Compliance Reports
# Generate secrets audit report
vault audit list -detailed
# List all secrets
vault kv list -format=json secret/ > secrets-inventory.json
# Check secret age
vault kv metadata get secret/app/config
Best Practices
1. Never Commit Secrets
✅ Good:
const apiKey = process.env.OPENAI_API_KEY;
❌ Bad:
const apiKey = "sk-proj-abc123..."; // NEVER!
2. Use Short-Lived Credentials
// Dynamic credentials with 1-hour TTL
const creds = await vault.read('database/creds/readwrite');
// Auto-revoked after 1 hour
3. Rotate Regularly
# Scheduled rotation
0 0 * * 0 /usr/bin/rotate-secrets.sh # Weekly
4. Least Privilege
# Only grant required permissions
path "secret/data/app/myapp/*" {
capabilities = ["read"]
}
5. Monitor Access
// Alert on unusual access patterns
if (accessCount > 1000 && timeWindow < 60) {
alert('Unusual secret access pattern detected');
}