Testing steps for https://github.com/kubernetes-csi/csi-driver-host-path/pull/569

test-plan.md

This docs contain e2e testing steps for kubernetes-csi/csi-driver-host-path#569

Prerequisites

Build and deployment

1. Build the project and push CSI hostpath plugin image to private repo

$ make push-multiarch PULL_BASE_REF=master REGISTRY_NAME=prasadg193 BUILD_PLATFORMS="linux amd64 amd64"
# This builds and pushes image prasadg193/hostpathplugin:canary

2. Deploy CSI hostpath plugin with external-snapshot-metadata sidecar

   a. Manual steps:

   Steps to generate certs and required manifest are given here: https://github.com/PrasadG193/csi-driver-host-path/tree/snapshot-metadata-service-deploy/examples/external-snapshot-metadata

   b. Using deploy.sh script

   Alternatively, one can use updated deploy.sh script from repo:

   i. clone the branch update-deploy-script from forked repo PrasadG193/csi-driver-host-path

   ii. Execute deploy script to setup hostpath plugin driver with external-snapshot-metadata change

      $ SNAPSHOT_METADATA_TESTS=true ./deploy/kubernetes-1.27/deploy.sh
   

3. Create CSI Hostpath storageclass

$ kubectl create -f examples/csi-storageclass.yaml

Sample SnapshotMetadata client

We will be using sample client to interact with external-snapshot-metadata service to query changed block metadata information between two CSI snapshots on Kubernetes.: https://github.com/PrasadG193/external-snapshot-metadata-client

The client with all necessary permissions can be deployed using manifest from: https://github.com/PrasadG193/external-snapshot-metadata-client/tree/main/deploy

This client performs following actions as recommended in KEP:

1. Find Driver name for the snapshot
2. Discover SnapshotMetadataService resource for the driver which contains endpoint, audience and CA cert
3. Create SA Token with expected audience and permissions
4. Make gRPC call GetMetadataAllocated and GetMetadataDelta with appropriate params from SnapshotMetadataService resource , generated SA token
5. Stream response and print on console

Test GetMetadataAllocated

1. Create a volume with Block mode access

kubectl apply -f - <<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-raw
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: csi-hostpath-sc
  volumeMode: Block
  resources:
    requests:
      storage: 10Mi
EOF

2. Mount the PVC to a pod

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: pod-raw
  labels:
    name: busybox-test
spec:
  restartPolicy: Always
  containers:
    - image: gcr.io/google_containers/busybox
      command: ["/bin/sh", "-c"]
      args: [ "tail -f /dev/null" ]
      name: busybox
      volumeDevices:
        - name: vol
          devicePath: /dev/loop3 # This device path needs to be replaced with the site specific
  volumes:
    - name: vol
      persistentVolumeClaim:
        claimName: pvc-raw
EOF

3. Snapshot PVC pvc-raw

kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: raw-pvc-snap-1
spec:
  volumeSnapshotClassName: csi-hostpath-snapclass
  source:
    persistentVolumeClaimName: pvc-raw
EOF

Wait for snapshot to be ready

$ kg vs snapshot-1
NAME         READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS            SNAPSHOTCONTENT                                    CREATIONTIME   AGE
snapshot-1   true         pvc-raw                             10Mi          csi-hostpath-snapclass   snapcontent-70b40b27-80d4-448b-bd9a-a87079c1a248   28s            29s

4. Using external-snapshot-metadata-client which uses GetMetadataAllocated gRPC to allocated blocks metadata

$ kubectl exec -ti cbt-client-fdc8f549c-stql2 -n cbt-client -- /external-snapshot-metadata-client -client-namespace cbt-client -service-account cbt-client -snapshot-1 raw-pvc-snap-1
.
.
.
.
## Response received from external-snapshot-metadata service:
{"blockMetadataType":"FIXED_LENGTH", "volumeCapacityBytes":"10485760", "blockMetadata":[{"sizeBytes":"4096"}, {"byteOffset":"4096", "sizeBytes":"4096"}, {"byteOffset"
:"8192", "sizeBytes":"4096"}, {"byteOffset":"12288", "sizeBytes":"4096"}, {"byteOffset":"16384", "sizeBytes":"4096"}, {"byteOffset":"20480", "sizeBytes":"4096"}, {"by
teOffset":"24576", "sizeBytes":"4096"}, {"byteOffset":"28672", "sizeBytes":"4096"}, {"byteOffset":"32768", "sizeBytes":"4096"}, {"byteOffset":"36864", "sizeBytes":"40
96"}, {"byteOffset":"40960", "sizeBytes":"4096"}, {"byteOffset":"45056", "sizeBytes":"4096"}, {"byteOffset":"49152", "sizeBytes":"4096"}, {"byteOffset":"53248", "size
Bytes":"4096"}, {"byteOffset":"57344", "sizeBytes":"4096"}, {"byteOffset":"6144
.
.
.
.
":"4096"}, {"byteOffset":"10440704", "sizeBytes":"4096"}, {"byteOffset":"10444800", "sizeBytes":"4096"}, {"byteOffset":"10448896", "sizeBytes":"4096"}, {"byteOffset":"10452992", "sizeBytes":"4096"}, {"byteOffset":"10457088", "sizeBytes":"4096"}, {"byteOffset":"10461184", "sizeBytes":"4096"}, {"byteOffset":"10465280", "sizeBytes":"4096"}, {"byteOffset":"10469376", "sizeBytes":"4096"}, {"byteOffset":"10473472", "sizeBytes":"4096"}, {"byteOffset":"10477568", "sizeBytes":"4096"}, {"byteOffset":"10481664", "sizeBytes":"4096"}]}
2024/12/09 12:42:34 finished

Test GetMetadataDelta

1. Change couple of blocks in the mounted device file in pod-raw Pod

$ kubectl exec -ti pod-raw -- sh

### change blocks 12, 13, 15 and 20
/ # dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=12 conv=notrunc
1+0 records in
1+0 records out
/ # dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=13 conv=notrunc
1+0 records in
1+0 records out
/ # dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=15 conv=notrunc
1+0 records in
1+0 records out
/ # dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=20 conv=notrunc
1+0 records in
1+0 records out

2. Snapshot pvc-raw again

kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: raw-pvc-snap-2
spec:
  volumeSnapshotClassName: csi-hostpath-snapclass
  source:
    persistentVolumeClaimName: pvc-raw
EOF

Wait for snapshot to be ready

$ kubectl get vs
NAME             READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS            SNAPSHOTCONTENT                                    CREATIONTIME   AGE
raw-pvc-snap-1   true         pvc-raw                             10Mi          csi-hostpath-snapclass   snapcontent-ef10f725-4261-4e80-af37-906708796700   7m40s          7m40s
raw-pvc-snap-2   true         pvc-raw                             10Mi          csi-hostpath-snapclass   snapcontent-188562cb-03b3-4b70-b12d-28900527bca8   23s            23s

3. Using external-snapshot-metadata-client which uses GetMetadataDelta gRPC to allocated blocks metadata

$ kubectl exec -ti cbt-client-fdc8f549c-stql2 -n cbt-client -- /external-snapshot-metadata-client -client-namespace cbt-client -service-account cbt-client -snapshot-1 raw-pvc-snap-1 -snapshot-2 raw-pvc-snap-2
.
.
.
.
## Response received from external-snapshot-metadata service:
{"blockMetadataType":"FIXED_LENGTH", "volumeCapacityBytes":"10485760", "blockMetadata":[{"byteOffset":"49152", "sizeBytes":"4096"}, {"byteOffset":"53248", "sizeBytes":"4096"}, {"byteOffset":"61440", "sizeBytes":"4096"}, {"byteOffset":"81920", "sizeBytes":"4096"}]}
2024/12/09 12:51:36 finished