Trustee in enclave

README.MD

Trustee in enclave

Setup

• kubernetes running in a trusted environment (in my case minikube on laptop) (1)
  • trustee-operator deployed
• Openshift in public cloud or bare metal (in my case snp baremetal on virtlab801) (2)
  • OSC deployed

Created sealed secret in (2)

SECRET_NAME=bootstrap
KEY_NAME=trustee-config
POINTER=$(podman run -it quay.io/confidential-devhub/coco-tools:0.3.0 /tools/secret seal vault --resource-uri kbs:///default/${SECRET_NAME}/${KEY_NAME} --provider kbs | grep -v "Warning")
echo $POINTER
oc create secret generic sealed-secret --from-literal=bootstrap.tar.gz=$POINTER -n default

Configure trustee (1)

Prepare bootstrap directory

For example, kbs-config.toml and https/token certificates:

bootstrap/
├── https.crt
├── https.key
├── kbs-config.toml
├── token.crt
└── token.key

Create archive and k8s secret for trustee:

tar cvfz bootstrap.tar.gz bootstrap
kubectl create secret generic -n trustee-operator-system bootstrap --from-file=trustee-config=bootstrap.tar.gz
export CR_NAME=$(kubectl get kbsconfig -n trustee-operator-system -o=jsonpath='{.items[0].metadata.name}') && kubectl patch KbsConfig -n trustee-operator-system $CR_NAME --type=json -p='[{"op":"add", "path":"/spec/kbsSecretResources/-", "value":"bootstrap"}]'

Enable port forwarding (if needed)

Instructions for exposing trustee service running in (1) to the vpn network. 10.45.225.51 is my laptop vpn address

nohup kubectl port-forward -n trustee-operator-system svc/kbs-service 8080:8080 --address 10.45.225.51 > /tmp/kbs-port-forward.log 2>&1 &
sudo firewall-cmd --zone=redhat --add-port=8080/tcp --permanent
sudo firewall-cmd --reload

Copy VCEK (if needed)

This is for caching locally the VCEK certificate in trustee. This is the upstream guide: https://github.com/confidential-containers/trustee/blob/main/attestation-service/docs/amd-offline-certificate-cache.md

Note: kbs-config configmap must have snp caching enabled:

[attestation_service.verifier_config.snp_verifier]
vcek_sources = [ { type = "OfflineStore" } ]

26e367d83df19b32d587dec648509560235572a3af18f1c72ce78a8c741ee97ad7e0f4ca3e7be32d20dfe5d7f1539f3191077659cb119624e8bb74afe9bec9ee is the hardware-id (retrieved with the snphost tool)

vcek.der is the vcek certificate for host virtlab801

POD_NAME=$(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system)
oc exec -it -n trustee-operator-system $POD_NAME -- mkdir -p /opt/confidential-containers/attestation-service/kds-store/vcek/26e367d83df19b32d587dec648509560235572a3af18f1c72ce78a8c741ee97ad7e0f4ca3e7be32d20dfe5d7f1539f3191077659cb119624e8bb74afe9bec9ee
kubectl exec -i -n trustee-operator-system $POD_NAME -- sh -c "cat > /opt/confidential-containers/attestation-service/kds-store/vcek/26e367d83df19b32d587dec648509560235572a3af18f1c72ce78a8c741ee97ad7e0f4ca3e7be32d20dfe5d7f1539f3191077659cb119624e8bb74afe9bec9ee/vcek.der" < ./vcek/26e367d83df19b32d587dec648509560235572a3af18f1c72ce78a8c741ee97ad7e0f4ca3e7be32d20dfe5d7f1539f3191077659cb119624e8bb74afe9bec9ee/vcek.der

Create trustee coco pod (2)

init data

export TRUSTEE_URL=http://10.45.225.51:8080
envsubst < initdata.toml.in > initdata.toml
export INITDATA=$(cat initdata.toml | gzip | base64 -w0)
echo $INITDATA

Generate deployment with initdata

Given the trustee-deployment.yaml.in template:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: trustee-deployment
  namespace: trustee-operator-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kbs
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        io.katacontainers.config.hypervisor.default_memory: "4096"
        io.katacontainers.config.hypervisor.kernel_params: "agent.guest_components_rest_api=all"
        io.katacontainers.config.hypervisor.cc_init_data: "${INITDATA}"
      labels:
        app: kbs
    spec:
      runtimeClassName: kata-cc
      nodeName: virtlab801.virt.eng.rdu2.dc.redhat.com

      initContainers:
        - name: secret-unpacker
          image: busybox:latest
          command: ["sh", "-c"]
          args:
            - |
              # 1. Wait for the Attestation Agent to deposit the bundle
              # 2. Unpack it into the shared volume
              # 3. Clean up the archive to save space
              until [ -f /sealed/secret-value/bootstrap.tar.gz ]; do sleep 1; done;
              tar -xzvf /sealed/secret-value/bootstrap.tar.gz -C /trustee/config/
              rm /sealed/secret-value/bootstrap.tar.gz
          volumeMounts:
            - name: secret-volume
              mountPath: /sealed/secret-value
            - name: trustee-config-volume
              mountPath: /trustee/config
            - name: coco-workdir-volume
              mountPath: /opt/confidential-containers

      containers:
      - command:
        - /usr/local/bin/kbs
        - --config-file
        - /trustee/config/bootstrap/kbs-config.toml
        env:
        - name: RUST_LOG
          value: debug
        image: ghcr.io/confidential-containers/key-broker-service:built-in-as-v0.17.0
        imagePullPolicy: IfNotPresent
        name: kbs
        ports:
        - containerPort: 8080
          name: kbs
          protocol: TCP
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - name: trustee-config-volume
          mountPath: /trustee/config
        - name: coco-workdir-volume
          mountPath: /opt/confidential-containers
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
        - name: secret-volume
          secret:
            secretName: sealed-secret
        - name: trustee-config-volume
          emptyDir:
            medium: Memory
        - name: coco-workdir-volume
          emptyDir:
            medium: Memory

Generate and apply the deployment manifest:

envsubst < trustee-deployment.yaml.in > trustee-deployment.yaml
oc create -f trustee-deployment.yaml

Trustee pod is now running in enclave with kata-cc runtime class