vault-sync-operator

Vault Sync Operator Development Summary

Project Overview

Successfully created a complete Kubernetes operator called vault-sync-operator that automatically syncs Kubernetes secrets to HashiCorp Vault using annotations on Deployments and Secrets.

Key Features Implemented

1. Dual Sync Modes: Watches both Kubernetes Deployments and Secrets for specific annotations and syncs secrets to Vault
2. Automatic Secret Synchronization: Supports both deployment-based and direct secret synchronization
3. Vault Kubernetes Authentication: Uses Vault’s Kubernetes auth backend for secure authentication
4. Selective Key Synchronization: Allows choosing specific keys from secrets to sync
5. Key Prefixing: Supports adding prefixes to secret keys when storing in Vault
6. Cleanup on Deletion: Automatically removes secrets from Vault when deployments or secrets are deleted
7. Finalizer Management: Uses Kubernetes finalizers to ensure proper cleanup

Project Structure

vault-sync-operator/
├── cmd/
│   └── main.go                 # Main application entry point
├── internal/
│   ├── controller/
│   │   ├── deployment_controller.go  # Deployment reconciler logic
│   │   ├── secret_controller.go      # Secret reconciler logic
│   │   └── sync_common.go           # Shared sync functionality
│   ├── vault/
│   │   ├── client.go           # Vault client with K8s auth
│   │   └── health.go           # Vault health checks
│   ├── goruntime/
│   │   ├── config.go           # Go runtime optimization
│   │   └── config_test.go      # Runtime configuration tests
│   └── metrics/
│       └── metrics.go          # Prometheus metrics
├── config/
│   ├── default/               # Kustomize default configuration
│   ├── manager/               # Manager deployment configuration
│   ├── rbac/                  # RBAC permissions
├── docs/                      # Documentation
│   ├── README.md              # Documentation index
│   ├── PROJECT_SUMMARY.md     # Complete project summary
│   ├── DEPLOYMENT.md          # Deployment guide
│   ├── VAULT-SETUP-GUIDE.md   # Vault configuration guide
│   ├── VAULT-AUTH-TROUBLESHOOTING.md # Auth troubleshooting
│   ├── VAULT-ADDRESS-CONFIGURATION.md # Address configuration
│   ├── VM-DEPLOYMENT-README.md # VM deployment guide
│   ├── multi-cluster-deployment.md # Multi-cluster guide
│   ├── performance-optimizations.md # Performance guide
│   ├── secret-rotation-detection.md # Secret rotation
│   └── ci-cd-pipeline.md      # CI/CD documentation
├── examples/                  # Example deployment files
├── scripts/
│   ├── setup-vault.sh         # Vault configuration script
│   ├── dev.sh                 # Development environment setup
│   ├── deploy-on-vm.sh        # VM deployment script
│   ├── build-vm-manifests.sh  # VM manifest builder
│   ├── validate-and-package.sh # Manifest validation
│   └── validate-manifests.sh  # Manifest syntax validation
├── test/                      # Test files
│   └── suite_test.go          # Test suite setup
├── hack/
│   └── boilerplate.go.txt     # License header template
├── charts/                    # Helm chart
│   └── vault-sync-operator/   # Operator Helm chart
├── deploy/
│   └── manual/                # Manual deployment manifests
├── Dockerfile                 # Container image build
├── Makefile                   # Build and deployment targets
├── go.mod                     # Go module dependencies
└── README.md                  # Main documentation

Core Components

1. Vault Client (internal/vault/client.go)

• Implements Kubernetes authentication with Vault
• Handles token management and renewal
• Provides methods for writing and deleting secrets

2. Deployment Controller (internal/controller/deployment_controller.go)

• Watches Kubernetes Deployments for vault-sync annotations
• Manages finalizers for proper cleanup
• Orchestrates secret synchronization to Vault

3. Main Application (cmd/main.go)

• Sets up the controller manager
• Configures command-line flags and logging
• Initializes Vault client and registers both Deployment and Secret controllers

Annotations Used

Annotation	Required	Description	Example
vault-sync.io/path	Yes	Vault storage path (enables sync)	"secret/data/my-app"
vault-sync.io/secrets	No	Custom secret configuration JSON	See examples

Note: The presence of vault-sync.io/path automatically enables vault sync on both Deployments and Secrets. The vault-sync.io/secrets annotation is optional and only needed for selective key syncing or prefixing.

Example Usage

Deployment-Based Sync

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  annotations:
    vault-sync.io/path: "secret/data/my-app"
    # Optional: vault-sync.io/secrets for custom configuration
spec:
  # ... deployment spec

Direct Secret Sync

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
  annotations:
    vault-sync.io/path: "secret/data/my-secret"
    # Optional: vault-sync.io/secrets for custom configuration
type: Opaque
data:
  key1: dmFsdWUx  # base64 encoded value

Security Features

1. RBAC Permissions: Required permissions for deployments and secrets (read/watch/update for annotations and finalizers)
2. Service Account Authentication: Uses Kubernetes service account tokens
3. Vault Policies: Configurable Vault policies for least privilege access
4. Secure Communication: TLS support for Vault communication

Deployment

The operator supports multiple deployment methods:

1. Helm Chart (Recommended) - Most flexible and production-ready
2. Kustomize - Good for GitOps workflows
3. Manual kubectl - Simple direct deployment

See the Deployment Guide for detailed installation instructions.

Configuration Options

The operator supports various configuration flags:

• --vault-addr: Vault server address
• --vault-role: Kubernetes auth role name
• --vault-auth-path: Vault auth path
• --metrics-bind-address: Metrics endpoint address
• --health-probe-bind-address: Health probe address
• --leader-elect: Enable leader election

Testing and Examples

• Basic Example: Simple secret sync with prefixing
• Multiple Secrets Example: Complex scenario with multiple secrets and different prefixes
• Test Suite: Ginkgo/Gomega based tests for controller logic

Production Features

1. Comprehensive Monitoring: Prometheus metrics for sync operations, errors, and performance tracking
2. Robust Error Handling: Enhanced error handling with retry logic and detailed logging for troubleshooting
3. Multi-tenancy Support: Namespace-based isolation and configurable Vault paths for secure multi-tenant usage
4. Automated CI/CD: Continuous integration and automated releases via GitHub Actions
5. Container Optimization: Runtime optimization specifically tuned for Kubernetes environments
6. Health Monitoring: Built-in Vault connectivity and authentication health probes
7. Secret Rotation Detection: Automatic detection and synchronization of secret changes
8. Multi-cluster Support: Complete documentation and deployment patterns for multi-cluster environments

Prerequisites

• Kubernetes cluster (v1.20+): Target deployment environment
• HashiCorp Vault: Configured with Kubernetes authentication
• kubectl: Configured to access your cluster

The operator is production-ready for deployment in Kubernetes environments with HashiCorp Vault integration.

Monitoring and Observability

Health and Readiness Checks

• Health Check (/healthz): Validates Vault server connectivity
• Readiness Check (/readyz): Ensures Vault authentication is working
• Integrated with Kubernetes probes for automatic health monitoring

Prometheus Metrics

The operator exposes comprehensive metrics on port :8080:

Sync Operations

• vault_sync_operator_sync_attempts_total: Sync attempt counters with success/failure labels
• vault_sync_operator_sync_duration_seconds: Operation duration histograms
• vault_sync_operator_secrets_discovered: Number of auto-discovered secrets

Error Tracking

• vault_sync_operator_secret_not_found_errors_total: Missing Kubernetes secrets
• vault_sync_operator_secret_key_missing_errors_total: Missing keys within secrets
• vault_sync_operator_config_parse_errors_total: Configuration parsing failures
• vault_sync_operator_vault_write_errors_total: Vault write errors by type

Authentication

• vault_sync_operator_auth_attempts_total: Vault authentication success/failure rates

Error Handling

Comprehensive error detection and reporting for:

• Missing Kubernetes secrets or keys (with available keys listed in logs)
• Vault authentication failures
• Vault permission denials
• Invalid configuration parsing
• Network connectivity issues
• Path validation errors

All errors are logged with structured context and tracked via Prometheus metrics for monitoring and alerting

Container Runtime Optimization

The operator is optimized for Kubernetes environments with automatic Go runtime configuration:

• Automatic GOMAXPROCS: Uses go.uber.org/automaxprocs to respect container CPU limits
• Memory Limit Awareness: GOMEMLIMIT configured from container memory limits
• Container-aware GC: Optimized garbage collection for container environments
• Runtime Validation: Startup logging and metrics for runtime configuration verification
• Production Build: Optimized Docker builds with static binaries and reduced image size

Runtime Metrics

• vault_sync_operator_runtime_info: Tracks GOMAXPROCS, GOMEMLIMIT, and GC configuration
• Startup validation logs for troubleshooting container resource detection

Implementation Details

1. Safety Features

• Preservation Annotation: vault-sync.io/preserve-on-delete: "true" prevents Vault secret deletion when deployments are deleted
• Finalizer Management: Ensures proper cleanup order and prevents orphaned resources
• Error Recovery: Comprehensive error handling with retry logic and detailed logging

2. Secret Generator Compatibility

• Runtime Secret Reading: Reads secrets when syncing, not when they’re created
• Auto-Discovery: Finds secrets referenced in pod templates regardless of creation method
• Generator-Friendly: Works with Kustomize, Helm, and other secret generation tools
• Helpful Error Messages: Suggests checking if generators have run when secrets are missing

3. Multi-Cluster Architecture

• Per-Cluster Deployment: Standard Kubernetes operator pattern (recommended approach)
• Cluster-Aware Paths: Optional cluster prefixing for Vault path organization
• Independent Operation: Each cluster operates independently for security and simplicity
• Shared Vault: Multiple clusters can use the same Vault server with different auth backends

Multi-Cluster Benefits:

• Security Isolation: Each cluster’s secrets remain within cluster boundaries
• Failure Isolation: One cluster’s issues don’t affect others
• Simple Architecture: No cross-cluster networking complexity
• Standard Practice: Follows established Kubernetes operator conventions

This site is open source. Improve this page.