Upgrading Virtual Clusters

Overview

Upgrading vCluster involves updating the control plane components, syncing mechanisms, and potentially the virtual Kubernetes version. This guide covers safe upgrade procedures and best practices.

Upgrade Types

vCluster Version

Update vCluster control plane and syncing components

Kubernetes Version

Upgrade the virtual Kubernetes API server version

Configuration Changes

Update vCluster configuration and features

Upgrading vCluster CLI

Before upgrading virtual clusters, update your CLI to the latest version:

vcluster upgrade

Upgrade to Specific Version

vcluster upgrade --version v0.20.0

Verify CLI Version

vcluster --version

Upgrading a Virtual Cluster

Pre-Upgrade Checklist

Review Release Notes

Check the vCluster release notes for:

Breaking changes
New features
Known issues
Migration guides

Create Backup

Always create a snapshot before upgrading:

vcluster snapshot create my-vcluster \
  oci://ghcr.io/my-org/backups:my-vcluster-pre-upgrade-$(date +%Y%m%d) \
  --namespace production

Check Current Version

Document the current version:

helm list -n production
vcluster describe my-vcluster --namespace production

Plan Maintenance Window

Schedule during low-traffic periods
Notify stakeholders
Prepare rollback plan
Have monitoring ready

Standard Upgrade Procedure

Helm Driver
Platform Driver
Docker Driver

# Update Helm repository
helm repo update

# Upgrade vCluster
helm upgrade my-vcluster vcluster \
  --repo https://charts.loft.sh \
  --namespace production \
  --reuse-values

Or upgrade to a specific version:

helm upgrade my-vcluster vcluster \
  --repo https://charts.loft.sh \
  --version 0.20.0 \
  --namespace production \
  --reuse-values

Upgrades are managed through the vCluster Platform UI or API:

# List available versions
vcluster platform list versions

# Upgrade via CLI
vcluster platform upgrade my-vcluster \
  --version 0.20.0 \
  --project my-project

# Stop current cluster
vcluster delete my-vcluster --driver docker

# Create with new version
vcluster create my-vcluster \
  --driver docker \
  --chart-version 0.20.0

# Restore from backup if needed
vcluster restore my-vcluster <backup-location>

Upgrade with Configuration Changes

If you need to modify configuration during upgrade:

# Export current configuration
vcluster describe my-vcluster --config-only > vcluster.yaml

# Edit vcluster.yaml with desired changes
vim vcluster.yaml

# Upgrade with new configuration
helm upgrade my-vcluster vcluster \
  --repo https://charts.loft.sh \
  --namespace production \
  --values vcluster.yaml

Post-Upgrade Verification

Check Pod Status

Verify all pods are running:

kubectl get pods -n production -l release=my-vcluster

Wait for all pods to reach Running state:

kubectl wait --for=condition=ready pod \
  -l app=vcluster,release=my-vcluster \
  -n production \
  --timeout=300s

Verify API Server

Test connectivity to the virtual cluster:

vcluster connect my-vcluster --namespace production
kubectl get nodes
kubectl get pods --all-namespaces

Check Syncing

Verify resources are syncing correctly:

# From virtual cluster context
kubectl create deployment test-nginx --image=nginx

# Check if it appears in host namespace
vcluster disconnect
kubectl get pods -n production | grep nginx

# Clean up
vcluster connect my-vcluster --namespace production
kubectl delete deployment test-nginx

Review Logs

Check for any errors or warnings:

kubectl logs -n production -l app=vcluster,release=my-vcluster --tail=100

Run Smoke Tests

Execute your application-specific health checks and tests.

Upgrading Kubernetes Version

To upgrade the virtual Kubernetes version:

# vcluster.yaml
controlPlane:
  distro:
    k8s:
      enabled: true
      version: "1.29.0"  # Specify desired version

Apply the upgrade:

helm upgrade my-vcluster vcluster \
  --repo https://charts.loft.sh \
  --namespace production \
  --values vcluster.yaml

Kubernetes version upgrades should be done incrementally (e.g., 1.27 → 1.28 → 1.29). Skipping versions may cause compatibility issues.

Rollback Procedures

If the upgrade fails or causes issues:

Helm Rollback

# View upgrade history
helm history my-vcluster -n production

# Rollback to previous version
helm rollback my-vcluster -n production

# Or rollback to specific revision
helm rollback my-vcluster 2 -n production

Snapshot Restore Rollback

For major issues, restore from pre-upgrade snapshot:

# Restore from backup
vcluster restore my-vcluster \
  oci://ghcr.io/my-org/backups:my-vcluster-pre-upgrade-20240115 \
  --namespace production

Upgrade Strategies

Blue-Green Upgrade

Minimize downtime with blue-green deployment:

Create Green Cluster

vcluster create my-vcluster-green --namespace production-green

Restore Data

vcluster restore my-vcluster-green \
  oci://ghcr.io/my-org/backups:my-vcluster-latest \
  --namespace production-green

Test Green Cluster

vcluster connect my-vcluster-green --namespace production-green
# Run tests

Switch Traffic

Update ingress or service endpoints to point to green cluster.

Monitor

Watch for issues. Keep blue cluster running for quick rollback.

Decommission Blue

After verification period:

vcluster delete my-vcluster --namespace production

Canary Upgrade

Gradually migrate workloads:

Create Canary Cluster

Deploy upgraded version alongside production:

vcluster create my-vcluster-canary \
  --namespace production-canary \
  --values vcluster-v0.20.yaml

Route Small Percentage

Configure load balancer to send 10% of traffic to canary.

Monitor Metrics

Compare performance and error rates between versions.

Gradually Increase

Incrementally increase traffic to canary: 10% → 25% → 50% → 100%

Complete Migration

Once validated, make canary the primary cluster.

Automated Upgrades

Renovate Bot Configuration

Automate Helm chart updates with Renovate:

{
  "extends": ["config:base"],
  "helm-values": {
    "fileMatch": [".*\\.ya?ml$"]
  },
  "packageRules": [
    {
      "matchDatasources": ["helm"],
      "matchPackageNames": ["vcluster"],
      "groupName": "vCluster",
      "automerge": false,
      "schedule": ["before 6am on monday"]
    }
  ]
}

GitOps-Based Upgrades

Using ArgoCD or Flux:

# argocd-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-vcluster
  namespace: argocd
spec:
  source:
    repoURL: https://charts.loft.sh
    targetRevision: 0.20.0  # Update this to trigger upgrade
    chart: vcluster
    helm:
      values: |
        # Your vcluster values
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: false
      selfHeal: true

Troubleshooting Upgrades

Pods Stuck in Pending State

Symptoms: Virtual cluster pods won’t start after upgrade.Diagnosis:

kubectl describe pod -n production -l release=my-vcluster
kubectl get events -n production --sort-by='.lastTimestamp'

Common Causes:

Insufficient resources
Image pull errors
PVC mount issues

Solutions:

Check resource availability:

kubectl top nodes
kubectl describe node <node-name>

Verify image access:

kubectl get pods -n production -l release=my-vcluster -o jsonpath='{.items[*].spec.containers[*].image}'

Check PVC status:
```
kubectl get pvc -n production
```

API Server Not Responding

Symptoms: Cannot connect to virtual cluster after upgrade.Diagnosis:

kubectl logs -n production -l app=vcluster,release=my-vcluster
kubectl get svc -n production

Solutions:

Restart the virtual cluster:

kubectl rollout restart statefulset -n production my-vcluster

Check service endpoints:
```
kubectl get endpoints -n production
```

Verify network policies:

kubectl get networkpolicy -n production

Syncing Stopped Working

Symptoms: Resources not syncing between virtual and host cluster.Diagnosis:

# Check syncer logs
kubectl logs -n production -l app=vcluster,release=my-vcluster -c syncer

# Check RBAC
kubectl auth can-i --list --as=system:serviceaccount:production:vc-my-vcluster

Solutions:

Verify sync configuration in vcluster.yaml
Check service account permissions
Review network connectivity between pods

Restart syncer:

kubectl delete pod -n production -l app=vcluster,release=my-vcluster

CRD Version Conflicts

Error: CRD versions incompatible after upgrade.Solution:

Identify conflicting CRDs:
```
kubectl get crd
```

Update CRDs manually if needed:

kubectl apply -f https://github.com/loft-sh/vcluster/releases/download/v0.20.0/crds.yaml

Or use Helm’s CRD upgrade:

helm upgrade my-vcluster vcluster \
  --repo https://charts.loft.sh \
  --namespace production \
  --reuse-values \
  --force

Best Practices

Test in Staging First

Always test upgrades in non-production environments before applying to production.

Incremental Updates

Upgrade one minor version at a time to minimize compatibility issues.

Backup Before Upgrade

Create snapshots before every upgrade. Automate this step.

Monitor During Upgrade

Watch metrics and logs during and after the upgrade process.

Document Everything

Keep records of upgrade procedures, issues encountered, and resolutions.

Plan Rollback

Always have a tested rollback plan before starting an upgrade.

Next Steps

Monitoring

Set up monitoring to track upgrade success and cluster health

Troubleshooting

Learn how to diagnose and fix common issues

Get Started

Architecture

Deployment

Operations

Resource Syncing

Use Cases

Security

Integrations

​Overview

​Upgrade Types

vCluster Version

Kubernetes Version

Configuration Changes

​Upgrading vCluster CLI

​Upgrade to Specific Version

​Verify CLI Version

​Upgrading a Virtual Cluster

​Pre-Upgrade Checklist

​Standard Upgrade Procedure

​Upgrade with Configuration Changes

​Post-Upgrade Verification

​Upgrading Kubernetes Version

​Rollback Procedures

​Helm Rollback

​Snapshot Restore Rollback

​Upgrade Strategies

​Blue-Green Upgrade

​Canary Upgrade

​Automated Upgrades

​Renovate Bot Configuration

​GitOps-Based Upgrades

​Troubleshooting Upgrades

​Best Practices

Test in Staging First

Incremental Updates

Backup Before Upgrade

Monitor During Upgrade

Document Everything

Plan Rollback

​Next Steps

Monitoring

Troubleshooting

Overview

Upgrade Types

Upgrading vCluster CLI

Upgrade to Specific Version

Verify CLI Version

Upgrading a Virtual Cluster

Pre-Upgrade Checklist

Standard Upgrade Procedure

Upgrade with Configuration Changes

Post-Upgrade Verification

Upgrading Kubernetes Version

Rollback Procedures

Helm Rollback

Snapshot Restore Rollback

Upgrade Strategies

Blue-Green Upgrade

Canary Upgrade

Automated Upgrades

Renovate Bot Configuration

GitOps-Based Upgrades

Troubleshooting Upgrades

Best Practices

Next Steps