Certificate Management

vCluster automatically generates and manages TLS certificates for secure communication between components. This page explains certificate architecture, rotation procedures, and best practices.

Certificate Architecture

vCluster uses multiple certificate authorities (CAs) and certificates for different components:

Certificate Authorities

vCluster creates the following CAs:

Kubernetes CA (ca.crt, ca.key)
- Signs all Kubernetes component certificates
- Used for API server, controller manager, scheduler
Server CA (server-ca.crt, server-ca.key)
- Used for server certificate validation
- Copy of the main Kubernetes CA
Client CA (client-ca.crt, client-ca.key)
- Validates client certificates
- Copy of the main Kubernetes CA
etcd CA (etcd/ca.crt, etcd/ca.key)
- Dedicated CA for etcd cluster
- Ensures etcd communication security
Front Proxy CA (front-proxy-ca.crt, front-proxy-ca.key)
- Used for extension API servers
- Enables aggregated API servers

Component Certificates

From pkg/certs/constants.go, vCluster manages these certificates:

const (
    // API Server certificates
    APIServerCertName = "apiserver.crt"
    APIServerKeyName  = "apiserver.key"
    
    // Kubelet client certificates
    APIServerKubeletClientCertName = "apiserver-kubelet-client.crt"
    APIServerKubeletClientKeyName  = "apiserver-kubelet-client.key"
    
    // etcd certificates
    EtcdServerCertName = "etcd/server.crt"
    EtcdServerKeyName  = "etcd/server.key"
    EtcdPeerCertName   = "etcd/peer.crt"
    EtcdPeerKeyName    = "etcd/peer.key"
    
    // Service account keys
    ServiceAccountPublicKeyName  = "sa.pub"
    ServiceAccountPrivateKeyName = "sa.key"
)

Certificate Validity

By default, certificates are valid for 10 years:

// From pkg/certs/constants.go
const CertificateValidity = time.Hour * 24 * 365 * 10

Certificate Storage

Certificates are stored in a Kubernetes Secret in the host cluster:

apiVersion: v1
kind: Secret
metadata:
  name: my-vcluster-certs
  namespace: vcluster-namespace
  labels:
    app: vcluster
    vcluster-name: my-vcluster
type: Opaque
data:
  ca.crt: <base64-encoded>
  ca.key: <base64-encoded>
  apiserver.crt: <base64-encoded>
  apiserver.key: <base64-encoded>
  # ... other certificates

The secret naming follows the pattern:

// From pkg/certs/ensure.go
func CertSecretName(vClusterName string) string {
    return vClusterName + "-certs"
}

Automatic Certificate Generation

vCluster automatically generates certificates during initialization:

// From pkg/certs/ensure.go
func Generate(ctx context.Context, serviceCIDR, certificatesDir string, options *config.VirtualClusterConfig) error {
    // Create kubeadm config
    kubeadmConfig, err := GenerateInitKubeadmConfig(serviceCIDR, certificatesDir, options)
    if err != nil {
        return fmt.Errorf("create kubeadm config: %w", err)
    }
    
    // Generate certificates
    err = EnsureCerts(ctx, currentNamespace, currentNamespaceClient, certificatesDir, options, kubeadmConfig)
    if err != nil {
        return fmt.Errorf("ensure certs: %w", err)
    }
    
    return nil
}

Extra SANs (Subject Alternative Names)

Configure additional SANs for the API server certificate:

controlPlane:
  proxy:
    extraSANs:
      - "vcluster.example.com"
      - "192.168.1.100"
      - "*.vcluster.svc.cluster.local"

The extra SANs are automatically added to certificates:

// From pkg/certs/ensure.go
func GetEtcdExtraSANs(options *config.VirtualClusterConfig) []string {
    extraSans := []string{"localhost"}
    
    if options.ControlPlane.Standalone.Enabled {
        extraSans = append(extraSans, "127.0.0.1", "0.0.0.0")
    } else {
        // Add service names
        etcdService := options.Name + "-etcd"
        extraSans = append(extraSans,
            etcdService,
            etcdService+"-headless",
            etcdService+"."+currentNamespace,
        )
    }
    
    // Add custom SANs
    extraSans = append(extraSans, options.ControlPlane.Proxy.ExtraSANs...)
    return extraSans
}

Certificate Rotation

vCluster supports automatic certificate rotation for leaf certificates and manual rotation for CA certificates.

Automatic Leaf Certificate Renewal

vCluster automatically renews leaf certificates that are expiring:

// From pkg/certs/ensure.go
func certsExpiringSoon(secretData map[string][]byte) bool {
    for _, secretKey := range certMap {
        if !strings.HasSuffix(secretKey, ".crt") {
            continue
        }
        // Skip CA certs
        if isCAFile(secretKey) {
            continue
        }
        
        certs, err := certhelper.ParseCertsPEM(pemBytes)
        if err != nil {
            return true
        }
        
        for _, cert := range certs {
            if certhelper.IsCertExpired(cert) {
                return true
            }
        }
    }
    return false
}

When certificates are expiring:

CA certificates and service account keys are preserved
Expiring leaf certificates are removed
New leaf certificates are generated using existing CAs
The certificate secret is updated

Manual Certificate Rotation

Rotate leaf certificates manually:

vcluster certs rotate --vcluster my-vcluster --namespace vcluster-namespace

The rotation process:

// From pkg/certs/rotate.go
func Rotate(ctx context.Context, vConfig *config.VirtualClusterConfig, pkiPath string, withCA bool, log log.Logger) error {
    // Backup existing certificates
    log.Info("Backing up previous PKI directory")
    backupDir := filepath.Join(pkiPath, "../pki.bak/" + timestamp)
    if err := backupDirectory(pkiPath, backupDir); err != nil {
        return fmt.Errorf("backing up PKI directory: %w", err)
    }
    
    // Remove certificates (preserve SA keys and optionally CA)
    excludeFuncs := []excludeFunc{excludeSAFiles}
    if !withCA {
        excludeFuncs = append(excludeFuncs, excludeCAFiles)
    }
    
    if err := removeFiles(pkiPath, excludeFuncs...); err != nil {
        return fmt.Errorf("removing files from PKI directory: %w", err)
    }
    
    // Generate new certificates
    if err := generateCertificates(pkiPath, kubeadmConfig); err != nil {
        return fmt.Errorf("creating pki assets: %w", err)
    }
    
    // Update secret
    return SyncSecret(ctx, vConfig.HostNamespace, CertSecretName(vConfig.Name), pkiPath, vConfig.HostClient)
}

Rotate CA Certificates

Rotate the entire PKI including CA certificates:

vcluster certs rotate-ca --vcluster my-vcluster --namespace vcluster-namespace

Rotating CA certificates is a disruptive operation that requires:

Restarting all vCluster components
Regenerating all kubeconfig files
Updating all client certificates

Check Certificate Expiry

Check certificate expiration dates:

vcluster certs check --vcluster my-vcluster --namespace vcluster-namespace

This displays:

{
  "filename": "apiserver.crt",
  "subject": "CN=kube-apiserver",
  "issuer": "CN=kubernetes",
  "expiryTime": "2034-01-01T00:00:00Z",
  "status": "OK"
}

Custom Certificate Configuration

Use External Certificates

Provide your own certificates by creating the secret before vCluster starts:

# Create certificate secret
kubectl create secret generic my-vcluster-certs \
  --from-file=ca.crt=./ca.crt \
  --from-file=ca.key=./ca.key \
  --from-file=apiserver.crt=./apiserver.crt \
  --from-file=apiserver.key=./apiserver.key \
  --namespace vcluster-namespace

vCluster will use existing certificates if the secret exists.

Generate Kubeconfig with Custom Certificates

Create a kubeconfig file with custom certificates:

// From pkg/certs/ensure.go
func CreateKubeConfig(spec *KubeConfigOptions, path string) error {
    config, err := BuildKubeConfig(spec)
    if err != nil {
        return fmt.Errorf("failed to build kubeconfig: %w", err)
    }
    
    return kubeconfigutil.WriteToDisk(path, config)
}

type KubeConfigOptions struct {
    CACert        string
    CAKey         string
    Organizations []string
    APIServer     string
    ClientName    string
}

Use this to create admin kubeconfigs:

# Extract certificates from secret
kubectl get secret my-vcluster-certs -n vcluster-namespace -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
kubectl get secret my-vcluster-certs -n vcluster-namespace -o jsonpath='{.data.ca\.key}' | base64 -d > ca.key

# Create custom kubeconfig
vcluster connect my-vcluster --namespace vcluster-namespace --update-current=false

Certificate Synchronization

In standalone mode, certificates are stored on disk:

controlPlane:
  standalone:
    enabled: true

Certificates location: /data/pki/ For high availability, certificates are synced from the secret:

// From pkg/certs/rotate.go
func SyncSecret(ctx context.Context, secretNamespace, secretName, pkiPath string, client kubernetes.Interface) error {
    secret, err := client.CoreV1().Secrets(secretNamespace).Get(ctx, secretName, metav1.GetOptions{})
    if err != nil {
        return fmt.Errorf("getting cert secret %s: %w", secretName, err)
    }
    
    data := map[string][]byte{}
    for k, v := range certMap {
        d, err := os.ReadFile(filepath.Join(pkiPath, k))
        if err != nil {
            continue
        }
        data[v] = d
    }
    
    // Patch the secret
    patch := crclient.MergeFrom(oldSecret)
    patchBytes, err := patch.Data(secret)
    
    _, err = client.CoreV1().Secrets(secretNamespace).Patch(ctx, secretName, patch.Type(), patchBytes, metav1.PatchOptions{})
    return err
}

Best Practices

Monitor Certificate Expiry

Set up monitoring for certificate expiration:

# Check certificates regularly
*/30 * * * * vcluster certs check --vcluster my-vcluster

Backup Certificates

Backup certificate secrets regularly:

# Export certificate secret
kubectl get secret my-vcluster-certs -n vcluster-namespace -o yaml > vcluster-certs-backup.yaml

# Backup to secure storage
kubectl get secret my-vcluster-certs -n vcluster-namespace -o json | \
  gpg --encrypt --recipient admin@example.com > certs-backup.gpg

Rotate Before Expiry

Rotate certificates well before expiration:

Leaf certificates: Rotate 30 days before expiry
CA certificates: Plan rotation 90 days in advance

Use Short-Lived Certificates

For enhanced security, use shorter certificate lifetimes:

# Set custom validity period (development only)
export DEVELOPMENT=true
export VCLUSTER_CERTS_VALIDITYPERIOD=2160h  # 90 days

vcluster certs rotate --vcluster my-vcluster

Secure Private Keys

Protect certificate private keys:

apiVersion: v1
kind: Secret
metadata:
  name: my-vcluster-certs
  annotations:
    # Prevent accidental deletion
    meta.helm.sh/release-namespace: vcluster-namespace
type: Opaque

Restrict access to certificate secrets:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cert-secret-access
  namespace: vcluster-namespace
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["my-vcluster-certs"]
    verbs: ["get"]  # Read-only

Test Certificate Rotation

Test rotation in a non-production environment first:

# Create test vCluster
vcluster create test-rotation --namespace test

# Perform rotation
vcluster certs rotate --vcluster test-rotation --namespace test

# Verify connectivity
vcluster connect test-rotation --namespace test
kubectl get nodes

Troubleshooting

Certificate Expired Errors

If you encounter expired certificate errors:

# Check certificate status
vcluster certs check --vcluster my-vcluster --namespace vcluster-namespace

# Rotate certificates
vcluster certs rotate --vcluster my-vcluster --namespace vcluster-namespace

# Restart vCluster pods
kubectl rollout restart statefulset my-vcluster -n vcluster-namespace

Invalid Certificate Errors

Verify certificate contents:

# Extract and inspect certificate
kubectl get secret my-vcluster-certs -n vcluster-namespace \
  -o jsonpath='{.data.apiserver\.crt}' | base64 -d | \
  openssl x509 -text -noout

# Check SANs
openssl x509 -in apiserver.crt -text | grep -A1 "Subject Alternative Name"

Connection Refused After Rotation

Restart vCluster components:

# Delete pods to force recreation with new certificates
kubectl delete pods -l release=my-vcluster -n vcluster-namespace

# Wait for pods to be ready
kubectl wait --for=condition=ready pod -l release=my-vcluster -n vcluster-namespace

Certificate Secret Not Found

Regenerate certificates:

# Delete existing secret if corrupted
kubectl delete secret my-vcluster-certs -n vcluster-namespace

# Restart vCluster to regenerate
kubectl rollout restart statefulset my-vcluster -n vcluster-namespace

Get Started

Architecture

Deployment

Operations

Resource Syncing

Use Cases

Security

Integrations

Certificate Management

Certificate Architecture

Certificate Authorities

Component Certificates

Certificate Validity

Certificate Storage

Automatic Certificate Generation

Extra SANs (Subject Alternative Names)

Certificate Rotation

Automatic Leaf Certificate Renewal

Manual Certificate Rotation

Rotate CA Certificates

Check Certificate Expiry

Custom Certificate Configuration

Use External Certificates

Generate Kubeconfig with Custom Certificates

Certificate Synchronization

Best Practices

Monitor Certificate Expiry

Backup Certificates

Rotate Before Expiry

Use Short-Lived Certificates

Secure Private Keys

Test Certificate Rotation

Troubleshooting

Certificate Expired Errors

Invalid Certificate Errors

Connection Refused After Rotation

Certificate Secret Not Found

Further Reading

Get Started

Architecture

Deployment

Operations

Resource Syncing

Use Cases

Security

Integrations

​Certificate Architecture

​Certificate Authorities

​Component Certificates

​Certificate Validity

​Certificate Storage

​Automatic Certificate Generation

​Extra SANs (Subject Alternative Names)

​Certificate Rotation

​Automatic Leaf Certificate Renewal

​Manual Certificate Rotation

​Rotate CA Certificates

​Check Certificate Expiry

​Custom Certificate Configuration

​Use External Certificates

​Generate Kubeconfig with Custom Certificates

​Certificate Synchronization

​Best Practices

​Monitor Certificate Expiry

​Backup Certificates

​Rotate Before Expiry

​Use Short-Lived Certificates

​Secure Private Keys

​Test Certificate Rotation

​Troubleshooting

​Certificate Expired Errors

​Invalid Certificate Errors

​Connection Refused After Rotation

​Certificate Secret Not Found

​Further Reading

Certificate Architecture

Certificate Authorities

Component Certificates

Certificate Validity

Certificate Storage

Automatic Certificate Generation

Extra SANs (Subject Alternative Names)

Certificate Rotation

Automatic Leaf Certificate Renewal

Manual Certificate Rotation

Rotate CA Certificates

Check Certificate Expiry

Custom Certificate Configuration

Use External Certificates

Generate Kubeconfig with Custom Certificates

Certificate Synchronization

Best Practices

Monitor Certificate Expiry

Backup Certificates

Rotate Before Expiry

Use Short-Lived Certificates

Secure Private Keys

Test Certificate Rotation

Troubleshooting

Certificate Expired Errors

Invalid Certificate Errors

Connection Refused After Rotation

Certificate Secret Not Found

Further Reading