Skip to content

Storage Tasks

Common storage and data how-tos.

Restoring PVC that was deleted

If a persistent volume claim (PVC) is deleted (or the entire namespace was deleted, including PVCs), you may still be able to recover the data relatively easily. By default, our persistent volumes have a Retain Reclaim Policy, meaning that they will be kept even if their bound PersistentVolumeClaim (PVC) is deleted. When this occurs, the associated PersistentVolume (PV) will end up with a status of Released. You will need to manually create a replacement PVC and modify the PV to in order to recreate what was there previously.

This example came from a deletion of the jupyterhub namespace. We wanted to recover some of the user associated PVCs (the hub-db-dir ones were not important for us to restore). A subset of that process is used here.

  1. Check PV still exists and is in the Released phase, e.g. 
$ kubectl get pv | grep jupyter
pvc-1378705c-2b84-487a-bac3-dce4860f6624                     1Gi        RWO            Retain           Released   jupyterhub/hub-db-dir                                                                       aisb-hdd                646d
pvc-1fdf7b5a-056f-4faa-8307-9fb8741b5771                     10Gi       RWO            Retain           Released   jupyterhub/claim-kcuster                                                                    aisb-hdd                448d
pvc-27cc26fd-60ed-49ca-ab00-e8fbfb3d8ccc                     1Gi        RWO            Retain           Released   jupyterhub/hub-db-dir                                                                       aisb-hdd                646d
pvc-40ba91d0-70d5-45fe-8a5e-2af06d9fc65d                     10Gi       RWO            Retain           Released   jupyterhub/claim-dmiller                                                                    aisb-hdd                232d
pvc-4fa521be-d59c-4b44-9705-dc4383c32fbe                     10Gi       RWO            Retain           Released   jupyterhub/claim-brian                                                                      aisb-hdd                541d
pvc-76e75b27-802e-45c4-9bca-010f33744b28                     1Gi        RWO            Retain           Released   jupyterhub/hub-db-dir                                                                       aisb-hdd                646d
pvc-7d06fb43-d562-4b1c-9deb-f8c662e38257                     1Gi        RWO            Retain           Released   jupyterhub/hub-db-dir                                                                       aisb-hdd                646d
...
  1. We need to label each PV we want to restore uniquely so that our new PVCs can use a label selector to bind to the correct one. In this case I used a label called restore but this can be anything.
$ kubectl label pv pvc-1fdf7b5a-056f-4faa-8307-9fb8741b5771 restore=kcuster \
    kubectl label pv pvc-40ba91d0-70d5-45fe-8a5e-2af06d9fc65d restore=dmiller \
    kubectl label pv pvc-4fa521be-d59c-4b44-9705-dc4383c32fbe restore=brian
    ...
  1. Create the PVCs to replace the ones that were missing. The spec should match what the PV offers, so in this case the namespace/name should be the same, storage request matches the original storage amount, and the accessModes is the same as the PV ("ReadWriteOnce" = "RWO"). Finally, the matchLabels selector should match what you labeled the PV with in the previous step so that Kubernetes knows which PV this PVC should use. This is what one of the ones I created looked like:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: claim-dmiller
  namespace: jupyterhub
spec:
  resources:
    requests:
      storage: 10Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  selector:
    matchLabels:
      restore: dmiller
  1. If you look at your PVCs after creating them, you'll see that they are still in a Pending state. This is because the PV is still in the Released status and associated with the previous (now deleted) claim. To change that, I edit the PV, double checking that it is the one I want and is labeled correctly, and delete the entire spec.claimRef field.
$ kubectl edit pv pvc-40ba91d0-70d5-45fe-8a5e-2af06d9fc65d

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: aisb-ceph-cluster.rbd.csi.ceph.com
  creationTimestamp: "2022-04-28T13:40:21Z"
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    restore: dmiller
  name: pvc-40ba91d0-70d5-45fe-8a5e-2af06d9fc65d
  resourceVersion: "1081386939"
  uid: 3b3a2517-d21d-490d-881c-706125188b83
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  claimRef:                                     # Delete this field and all its subfields through uid after double checking
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: claim-dmiller                         # Double check this is indeed the claim you want and it matches the label you intended above
    namespace: jupyterhub                       # Double check this is the correct namespace
    resourceVersion: "1081386937"
    uid: 66e859ac-c062-4a40-a135-947f396c7d3b
  csi:
    controllerExpandSecretRef:
...

Once you complete this step, the PV will change to the Available status briefly before it becomes Bound again with the appropriate PVC you created in step 3.

Once your PVCs are restored, check your application to make sure they are using the new ones correctly. It's possible they will need to have some labels or annotations added before the application uses them correctly. This will be on a case-by-case basis, hopefully logs are helpful in determining why it isn't being picked up.