Skip to content

Instantly share code, notes, and snippets.

@Himan10

Himan10/gist:ca1d6de28054bf7fcc15f928931e58b4

Created March 5, 2026 04:28
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save Himan10/ca1d6de28054bf7fcc15f928931e58b4 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save Himan10/ca1d6de28054bf7fcc15f928931e58b4 to your computer and use it in GitHub Desktop.
Download ZIP
docker-image-signing
Raw
gistfile1.txt
# Container Image Signing, Verification & Policy Enforcement
> **Tools Covered:** Notation · AWS Signer · Kyverno · Amazon ECR
> **Goal:** Sign container images cryptographically, store signatures in ECR using the OCI referrers API, and enforce "signed-only" policies at the Kubernetes admission layer — all within an AWS-native stack.
---
## Table of Contents
1. [Overview & Architecture](#1-overview--architecture)
2. [Notation — Container Signing](#2-notation--container-signing)
- [Installation](#21-installation)
- [Key Management (local & remote)](#22-key-management)
- [Signing an Image](#23-signing-an-image)
- [Verifying a Signature](#24-verifying-a-signature)
3. [AWS Signer — Managed Signing with Notation](#3-aws-signer--managed-signing-with-notation)
- [Prerequisites & IAM](#31-prerequisites--iam)
- [Install the AWS Signer Notation Plugin](#32-install-the-aws-signer-notation-plugin)
- [Create a Signing Profile](#33-create-a-signing-profile)
- [Sign an Image in ECR](#34-sign-an-image-in-ecr)
- [Verify via AWS Signer](#35-verify-via-aws-signer)
4. [ECR — Registry-Specific Configuration](#4-ecr--registry-specific-configuration)
- [How ECR Stores Notation Signatures](#41-how-ecr-stores-notation-signatures)
- [ECR Authentication for Notation](#42-ecr-authentication-for-notation)
- [Fine-Grained IAM Policies for ECR](#43-fine-grained-iam-policies-for-ecr)
- [ECR Repository Policies](#44-ecr-repository-policies)
- [ECR Immutable Tags](#45-ecr-immutable-tags)
- [ECR Lifecycle Policies for Signature Artifacts](#46-ecr-lifecycle-policies-for-signature-artifacts)
- [IRSA for Kyverno on EKS](#47-irsa-for-kyverno-on-eks)
- [ECR Token Refresh for Non-IRSA Environments](#48-ecr-token-refresh-for-non-irsa-environments)
- [Cross-Account ECR: Sign in CI, Deploy in Prod](#49-cross-account-ecr-sign-in-ci-deploy-in-prod)
- [VPC Endpoints for Private Cluster Access](#410-vpc-endpoints-for-private-cluster-access)
- [Verifying AWS Signer Revocation Status](#411-verifying-aws-signer-revocation-status)
5. [Kyverno — Policy Enforcement for Signed Images](#5-kyverno--policy-enforcement-for-signed-images)
- [Installation](#51-installation)
- [How Kyverno Intercepts Workloads](#52-how-kyverno-intercepts-workloads)
- [Verify Notation Signatures](#53-verify-notation-signatures)
- [Verify AWS Signer Signatures](#54-verify-aws-signer-signatures)
- [Block Unsigned Images — Full Policy Examples](#55-block-unsigned-images--full-policy-examples)
- [Audit Mode vs. Enforce Mode](#56-audit-mode-vs-enforce-mode)
- [Testing Policies](#57-testing-policies)
6. [End-to-End Workflow](#6-end-to-end-workflow)
7. [Best Practices](#7-best-practices)
- [Notation Best Practices](#71-notation-best-practices)
- [AWS Signer Best Practices](#72-aws-signer-best-practices)
- [Kyverno Best Practices](#73-kyverno-best-practices)
- [ECR Best Practices](#74-ecr-best-practices)
- [General Supply-Chain Security Best Practices](#75-general-supply-chain-security-best-practices)
8. [Troubleshooting Reference](#8-troubleshooting-reference)
---
## 1. Overview & Architecture
```
Developer ──► Build Image ──► Push to Registry (ECR / GHCR / etc.)
│
notation sign ◄── AWS Signer Plugin
│ (or local key)
Signature stored as
OCI artifact in registry
│
Kubernetes ──► kubectl apply ──► Kyverno Admission Webhook
│
┌────────────┴──────────────┐
│ verifyImages policy │
│ fetches signature from │
│ registry & verifies cert │
└────────────┬──────────────┘
✅ Signed │ ❌ Unsigned
ADMIT │ DENY (403)
```
**Key concepts:**
| Concept | Description |
|---|---|
| **Notation** | CLI + SDK from the Notary Project (CNCF). Signs and verifies OCI artifacts using X.509 certificates. |
| **AWS Signer** | Fully managed signing service. Acts as a Notation *plugin* — private keys never leave AWS KMS. |
| **OCI referrers** | Signatures are stored as OCI artifacts linked to the original image digest via the OCI referrers API. |
| **Kyverno** | Kubernetes-native policy engine. Its `verifyImages` rule calls out to the registry to confirm a valid signature exists before admitting a Pod. |
| **Trust store / trust policy** | Notation's model for deciding *which* certificates are trusted for *which* images. |
---
## 2. Notation — Container Signing
### 2.1 Installation
**macOS (Homebrew)**
```bash
brew install notation
```
**Linux (binary)**
```bash
NOTATION_VERSION=1.2.0
curl -Lo notation.tar.gz \
https://github.com/notaryproject/notation/releases/download/v${NOTATION_VERSION}/notation_${NOTATION_VERSION}_linux_amd64.tar.gz
tar -xzf notation.tar.gz -C /usr/local/bin notation
notation version
```
**Verify installation**
```bash
notation version
# notation v1.2.0
```
---
### 2.2 Key Management
Notation supports two key storage backends:
#### Option A — Local key pair (development / testing only)
```bash
# Generate a self-signed certificate + private key
notation cert generate-test \
--default \
"acme-dev"
# List keys and certificates
notation key ls
notation cert ls
```
> **Warning:** Self-signed keys are suitable only for local testing. Use a proper CA or AWS Signer for production.
#### Option B — External key (AWS KMS, HashiCorp Vault, etc.)
Keys managed by a plugin (see Section 3 for AWS Signer). The private key never leaves the KMS.
```bash
# After plugin installation, add a KMS-backed key
notation key add \
--plugin com.amazonaws.signer.notation.plugin \
--id "arn:aws:signer:us-east-1:123456789012:/signing-profiles/my-profile" \
--plugin-config "aws-region=us-east-1" \
"aws-signer-key"
```
---
### 2.3 Signing an Image
Always sign by **digest** (not by tag) to prevent tag mutation attacks.
```bash
# 1. Get the digest of the image you just pushed
IMAGE="123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp"
DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' ${IMAGE}:latest)
# e.g. 123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp@sha256:abc123...
# 2. Sign the image (uses the default key)
notation sign "${DIGEST}"
# 3. Sign with a specific key
notation sign --key "aws-signer-key" "${DIGEST}"
# 4. Add custom metadata (annotations) to the signature
notation sign \
--key "aws-signer-key" \
--annotation "io.acme/signed-by=ci-pipeline" \
--annotation "io.acme/build-id=${CI_BUILD_ID}" \
"${DIGEST}"
```
After signing, the signature is stored as an OCI artifact in the same repository, linked to the image digest via the OCI referrers API.
```bash
# Inspect what's stored alongside the image
notation inspect "${DIGEST}"
```
---
### 2.4 Verifying a Signature
Notation uses a **trust policy** (`trustpolicy.json`) to specify which certificates are trusted and for which registries/repositories.
#### Create a trust store
```bash
# Add a CA certificate to the named trust store "acme-ca"
notation cert add \
--type ca \
--store "acme-ca" \
./acme-root-ca.pem
```
#### Create a trust policy (`~/.config/notation/trustpolicy.json`)
```json
{
"version": "1.0",
"trustPolicies": [
{
"name": "acme-policy",
"registryScopes": [
"123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp"
],
"signatureVerification": {
"level": "strict"
},
"trustStores": ["ca:acme-ca"],
"trustedIdentities": [
"x509.subject: C=US, O=Acme Corp, CN=Acme Signing CA"
]
},
{
"name": "aws-signer-policy",
"registryScopes": [
"123456789012.dkr.ecr.us-east-1.amazonaws.com/*"
],
"signatureVerification": {
"level": "strict"
},
"trustStores": ["signingAuthority:aws-signer-ts"],
"trustedIdentities": [
"arn:aws:signer:us-east-1:123456789012:/signing-profiles/my-profile"
]
}
]
}
```
#### Verify
```bash
notation verify "${DIGEST}"
# Successfully verified signature for 123456789012.dkr.ecr.../myapp@sha256:abc123
```
---
## 3. AWS Signer — Managed Signing with Notation
AWS Signer provides a managed PKI where **private keys are generated and stored in AWS KMS** — they never exist on disk. It integrates with Notation via an official plugin.
### 3.1 Prerequisites & IAM
**Required IAM permissions for the signing principal (CI role):**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"signer:StartSigningJob",
"signer:DescribeSigningJob",
"signer:GetSigningProfile",
"signer:ListSigningJobs"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchGetImage",
"ecr:GetDownloadUrlForLayer",
"ecr:PutImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload"
],
"Resource": "*"
}
]
}
```
**Required IAM permissions for verification (Kyverno / runtime role):**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"signer:GetRevocationStatus"
],
"Resource": "*"
}
]
}
```
---
### 3.2 Install the AWS Signer Notation Plugin
```bash
# Download the plugin
PLUGIN_VERSION=1.0.298
curl -Lo notation-aws-signer-plugin.zip \
https://d2hvyiie56hcat.cloudfront.net/linux/amd64/plugin/${PLUGIN_VERSION}/notation-com.amazonaws.signer.notation.plugin_linux_amd64.zip
# Verify the checksum (always verify)
curl -Lo SHA256SUMS \
https://d2hvyiie56hcat.cloudfront.net/linux/amd64/plugin/${PLUGIN_VERSION}/checksums.txt
sha256sum --check SHA256SUMS
# Install the plugin
unzip notation-aws-signer-plugin.zip
notation plugin install --file ./notation-com.amazonaws.signer.notation.plugin
# Confirm
notation plugin ls
# NAME DESCRIPTION VERSION
# com.amazonaws.signer.notation.plugin AWS Signer plugin... 1.0.298
```
---
### 3.3 Create a Signing Profile
```bash
# Create a signing profile for container images
aws signer put-signing-profile \
--profile-name "my-container-signing-profile" \
--platform-id "Notation-OCI-SHA384-ECDSA" \
--signature-validity-period "value=12,type=MONTHS" \
--signing-material "certificateArn=arn:aws:acm:us-east-1:123456789012:certificate/abc-123" \
--region us-east-1
# Or let AWS Signer manage the certificate entirely (recommended)
aws signer put-signing-profile \
--profile-name "my-container-signing-profile" \
--platform-id "Notation-OCI-SHA384-ECDSA" \
--signature-validity-period "value=12,type=MONTHS" \
--region us-east-1
# Retrieve the profile ARN
aws signer get-signing-profile \
--profile-name "my-container-signing-profile" \
--query 'arn' --output text
```
---
### 3.4 Sign an Image in ECR
```bash
# Set variables
REGION="us-east-1"
ACCOUNT_ID="123456789012"
REPO="myapp"
TAG="v1.0.0"
PROFILE_ARN="arn:aws:signer:${REGION}:${ACCOUNT_ID}:/signing-profiles/my-container-signing-profile"
IMAGE="${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${REPO}:${TAG}"
# Authenticate to ECR
aws ecr get-login-password --region ${REGION} | \
docker login --username AWS --password-stdin \
"${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com"
# Push image and capture digest
docker push ${IMAGE}
DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' ${IMAGE})
# Add the AWS Signer key to Notation
notation key add \
--plugin "com.amazonaws.signer.notation.plugin" \
--id "${PROFILE_ARN}" \
--plugin-config "aws-region=${REGION}" \
"aws-signer-key"
# Sign
notation sign \
--key "aws-signer-key" \
"${DIGEST}"
# Confirm the signature was stored
notation ls "${DIGEST}"
```
---
### 3.5 Verify via AWS Signer
```bash
# Download the AWS Signer root CA bundle
curl -Lo aws-signer-notation-root.pem \
https://d2hvyiie56hcat.cloudfront.net/aws-signer-notation-root.cert.pem
# Add to a trust store
notation cert add \
--type signingAuthority \
--store "aws-signer-ts" \
./aws-signer-notation-root.pem
# Verify
notation verify "${DIGEST}"
```
---
## 4. ECR — Registry-Specific Configuration
Everything in sections 2 and 3 applies generically to any OCI registry. This section covers what is specific to **Amazon ECR** — authentication mechanics, IAM design, how signatures are stored and cleaned up, and how to wire Kyverno into ECR without long-lived credentials.
---
### 4.1 How ECR Stores Notation Signatures
ECR supports the **OCI Distribution Spec v1.1 referrers API** — the same mechanism Notation uses to attach signatures to images. When `notation sign` runs:
1. It creates a **Notary v2 signature envelope** (JWS format) containing:
- The signed image digest
- The signing certificate chain
- Any custom annotations you passed with `--annotation`
2. It pushes this envelope to ECR as an OCI manifest with `artifactType: application/vnd.cncf.notary.signature`
3. ECR links the signature to the image via the referrers API — the signature manifest's `subject` field points to the signed image's digest
```
ECR Repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp
│
├── sha256:abc123 (image manifest — your application image)
│ └── [referrer] sha256:def456 (Notary signature artifact)
│ artifactType: application/vnd.cncf.notary.signature
│
└── sha256:aaa111 (older image version)
└── [referrer] sha256:bbb222 (its signature)
```
**Inspect what ECR has stored:**
```bash
# List all OCI referrers (signatures) attached to an image digest
aws ecr describe-images \
--repository-name myapp \
--image-ids imageDigest=sha256:abc123 \
--region us-east-1
# Using the OCI referrers API directly via notation
notation ls 123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp@sha256:abc123
# Using crane (lightweight OCI client) to browse referrers
crane manifest 123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp@sha256:abc123 \
--platform=all 2>/dev/null | jq .
```
> **Note:** ECR does not require any special feature flag for OCI referrers support — it is enabled by default for all private ECR repositories. Public ECR (`public.ecr.aws`) also supports referrers.
---
### 4.2 ECR Authentication for Notation
ECR uses **short-lived IAM-derived tokens** (valid 12 hours) instead of static credentials. Notation reads credentials through the same Docker credential chain that `docker push/pull` uses.
#### Option A — Docker ECR Credential Helper (recommended)
The credential helper automatically refreshes the ECR token on each request, so you never have to manually re-authenticate.
```bash
# Install on Linux (Go binary)
go install github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login@latest
# Install on Amazon Linux / AL2023
sudo yum install -y amazon-ecr-credential-helper
# Install on macOS
brew install docker-credential-helper-ecr
# Configure Docker (and Notation) to use it for all ECR registries
# Edit ~/.docker/config.json
{
"credHelpers": {
"123456789012.dkr.ecr.us-east-1.amazonaws.com": "ecr-login",
"234567890123.dkr.ecr.eu-west-1.amazonaws.com": "ecr-login",
"public.ecr.aws": "ecr-login"
}
}
```
Once configured, `notation sign`, `notation verify`, and `notation ls` all authenticate transparently using whatever AWS credentials are active in the environment (IAM role, instance profile, `~/.aws/credentials`, environment variables).
#### Option B — Manual login (one-off or scripting)
```bash
REGION="us-east-1"
ACCOUNT_ID="123456789012"
aws ecr get-login-password --region ${REGION} | \
docker login \
--username AWS \
--password-stdin \
"${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com"
# Token is valid for 12 hours — plan refresh in long-running pipelines
```
#### Option C — Environment variable credentials (CI without instance profile)
```bash
# In GitHub Actions or any CI, set via OIDC or static credentials
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_SESSION_TOKEN="..." # if using assumed role / OIDC
export AWS_REGION="us-east-1"
# notation will pick these up automatically via the AWS SDK credential chain
notation sign "${DIGEST}"
```
---
### 4.3 Fine-Grained IAM Policies for ECR
Separate the IAM permissions for the three different principals that interact with ECR in a signed-image workflow.
#### Signing role (CI pipeline)
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ECRAuthentication",
"Effect": "Allow",
"Action": "ecr:GetAuthorizationToken",
"Resource": "*"
},
{
"Sid": "ECRPushImage",
"Effect": "Allow",
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage"
],
"Resource": "arn:aws:ecr:us-east-1:123456789012:repository/myapp"
},
{
"Sid": "ECRPushSignature",
"Effect": "Allow",
"Action": [
"ecr:PutImage",
"ecr:BatchCheckLayerAvailability",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:DescribeImages"
],
"Resource": "arn:aws:ecr:us-east-1:123456789012:repository/myapp"
},
{
"Sid": "AWSSigner",
"Effect": "Allow",
"Action": [
"signer:StartSigningJob",
"signer:DescribeSigningJob",
"signer:GetSigningProfile"
],
"Resource": "arn:aws:signer:us-east-1:123456789012:/signing-profiles/production-profile"
}
]
}
```
#### Node / kubelet pull role (EKS node group IAM role)
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ECRAuthentication",
"Effect": "Allow",
"Action": "ecr:GetAuthorizationToken",
"Resource": "*"
},
{
"Sid": "ECRPullImages",
"Effect": "Allow",
"Action": [
"ecr:BatchGetImage",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchCheckLayerAvailability"
],
"Resource": "arn:aws:ecr:us-east-1:123456789012:repository/*"
}
]
}
```
#### Kyverno IRSA role (admission webhook — pulls image manifests + signatures for verification)
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ECRAuthentication",
"Effect": "Allow",
"Action": "ecr:GetAuthorizationToken",
"Resource": "*"
},
{
"Sid": "ECRReadForVerification",
"Effect": "Allow",
"Action": [
"ecr:BatchGetImage",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchCheckLayerAvailability",
"ecr:DescribeImages",
"ecr:ListImages"
],
"Resource": "arn:aws:ecr:us-east-1:123456789012:repository/*"
},
{
"Sid": "SignerRevocationCheck",
"Effect": "Allow",
"Action": "signer:GetRevocationStatus",
"Resource": "*"
}
]
}
```
---
### 4.4 ECR Repository Policies
ECR repository policies are **resource-based IAM policies** attached directly to a repository. They control access at the registry level, independent of identity-based IAM policies. Use them to enforce a hard boundary: only the CI signing role can push; everything else is read-only.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowCIPushAndSign",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:role/GitHubActionsSigningRole"
},
"Action": [
"ecr:PutImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:BatchCheckLayerAvailability",
"ecr:DescribeImages"
]
},
{
"Sid": "AllowEKSNodePull",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:role/EKSNodeGroupRole"
},
"Action": [
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability"
]
},
{
"Sid": "AllowKyvernoPull",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:role/KyvernoIRSARole"
},
"Action": [
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability",
"ecr:DescribeImages",
"ecr:ListImages"
]
},
{
"Sid": "DenyDeleteExceptAdmin",
"Effect": "Deny",
"Principal": "*",
"Action": [
"ecr:DeleteRepository",
"ecr:BatchDeleteImage"
],
"Condition": {
"ArnNotLike": {
"aws:PrincipalArn": "arn:aws:iam::123456789012:role/RegistryAdminRole"
}
}
}
]
}
```
**Apply the policy:**
```bash
aws ecr set-repository-policy \
--repository-name myapp \
--policy-text file://ecr-repo-policy.json \
--region us-east-1
# Verify it was applied
aws ecr get-repository-policy \
--repository-name myapp \
--region us-east-1 | jq -r '.policyText | fromjson'
```
---
### 4.5 ECR Immutable Tags
Enable immutable tags so that a version tag (e.g., `v1.2.3`) cannot be overwritten after it has been pushed and signed. This is a complementary defence to signing — it prevents tag confusion attacks even if an attacker gains push access.
```bash
aws ecr put-image-tag-mutability \
--repository-name myapp \
--image-tag-mutability IMMUTABLE \
--region us-east-1
```
**Via Terraform:**
```hcl
resource "aws_ecr_repository" "myapp" {
name = "myapp"
image_tag_mutability = "IMMUTABLE"
image_scanning_configuration {
scan_on_push = true
}
encryption_configuration {
encryption_type = "KMS"
kms_key = aws_kms_key.ecr.arn
}
}
```
With immutable tags enabled, attempting to push a different image under an existing tag returns:
```
Error response from daemon: Tag myapp:v1.2.3 is immutable in the repository.
```
---
### 4.6 ECR Lifecycle Policies for Signature Artifacts
Notation signatures are stored as **untagged OCI manifests** in ECR. When you delete an image, ECR does **not** automatically delete its attached signatures — they become orphaned. A lifecycle policy handles cleanup.
> **Warning:** ECR lifecycle policies match on tag status and age, not on `artifactType`. An aggressive "delete all untagged after N days" policy will also delete Notation signatures. Test changes in a non-production repository first and run `notation ls` after applying to confirm signatures are intact.
```json
{
"rules": [
{
"rulePriority": 5,
"description": "Expire untagged images (includes orphaned signatures) after 14 days",
"selection": {
"tagStatus": "untagged",
"countType": "sinceImagePushed",
"countUnit": "days",
"countNumber": 14
},
"action": { "type": "expire" }
},
{
"rulePriority": 10,
"description": "Keep only the last 20 tagged releases",
"selection": {
"tagStatus": "tagged",
"tagPrefixList": ["v"],
"countType": "imageCountMoreThan",
"countNumber": 20
},
"action": { "type": "expire" }
}
]
}
```
```bash
aws ecr put-lifecycle-policy \
--repository-name myapp \
--lifecycle-policy-text file://ecr-lifecycle.json \
--region us-east-1
# Preview what will be expired before applying
aws ecr get-lifecycle-policy-preview \
--repository-name myapp \
--region us-east-1
```
**Safer approach — expire only orphaned signatures:**
Because signatures and images share the same repository, the safest strategy is to keep untagged manifests for longer than your longest-running workload (so signatures are still present for Kyverno to verify), and rely on a separate cleanup script that cross-references `notation ls` output against live ECR images.
```bash
#!/bin/bash
# orphan-signature-cleanup.sh
# Deletes signatures in ECR whose subject image no longer exists
REPO="123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp"
REGION="us-east-1"
# Get all image digests that are present
LIVE_DIGESTS=$(aws ecr describe-images \
--repository-name myapp \
--region ${REGION} \
--query 'imageDetails[].imageDigest' \
--output text)
# For each signature artifact, check if its subject image still exists
aws ecr describe-images \
--repository-name myapp \
--region ${REGION} \
--filter tagStatus=UNTAGGED \
--query 'imageDetails[?artifactMediaType==`application/vnd.cncf.notary.signature`].imageDigest' \
--output text | tr '\t' '\n' | while read SIG_DIGEST; do
SUBJECT=$(notation inspect "${REPO}@${SIG_DIGEST}" 2>/dev/null | grep -oP 'sha256:[a-f0-9]+' | head -1)
if ! echo "${LIVE_DIGESTS}" | grep -q "${SUBJECT}"; then
echo "Deleting orphaned signature ${SIG_DIGEST} (subject ${SUBJECT} not found)"
aws ecr batch-delete-image \
--repository-name myapp \
--region ${REGION} \
--image-ids imageDigest="${SIG_DIGEST}"
fi
done
```
---
### 4.7 IRSA for Kyverno on EKS
This is the **most important ECR-specific configuration** for Kyverno. The admission webhook needs to authenticate to ECR to fetch image manifests and OCI referrers (signatures) at admission time. On EKS, use **IRSA (IAM Roles for Service Accounts)** — no long-lived credentials, automatic token rotation.
#### Step 1: Ensure the EKS cluster has an OIDC provider
```bash
# Check if OIDC provider exists
aws eks describe-cluster \
--name my-cluster \
--query "cluster.identity.oidc.issuer" \
--output text
# If not set up, associate one
eksctl utils associate-iam-oidc-provider \
--cluster my-cluster \
--region us-east-1 \
--approve
```
#### Step 2: Create the IAM role with OIDC trust
```bash
CLUSTER="my-cluster"
REGION="us-east-1"
ACCOUNT_ID="123456789012"
OIDC_PROVIDER=$(aws eks describe-cluster \
--name ${CLUSTER} \
--query "cluster.identity.oidc.issuer" \
--output text | sed 's|https://||')
# Trust policy allowing the three Kyverno service accounts to assume this role
cat > kyverno-trust-policy.json << EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"${OIDC_PROVIDER}:sub": [
"system:serviceaccount:kyverno:kyverno-admission-controller",
"system:serviceaccount:kyverno:kyverno-background-controller",
"system:serviceaccount:kyverno:kyverno-reports-controller"
],
"${OIDC_PROVIDER}:aud": "sts.amazonaws.com"
}
}
}
]
}
EOF
# Create role
aws iam create-role \
--role-name KyvernoECRRole \
--assume-role-policy-document file://kyverno-trust-policy.json \
--description "Allows Kyverno to pull images and verify signatures from ECR"
# Attach ECR + Signer read policy (the JSON from section 4.3)
aws iam put-role-policy \
--role-name KyvernoECRRole \
--policy-name KyvernoECRPolicy \
--policy-document file://kyverno-ecr-policy.json
```
#### Step 3: Install Kyverno with IRSA annotations
```bash
ROLE_ARN="arn:aws:iam::${ACCOUNT_ID}:role/KyvernoECRRole"
helm install kyverno kyverno/kyverno \
--namespace kyverno \
--create-namespace \
--set admissionController.replicas=3 \
--set backgroundController.replicas=2 \
--set cleanupController.replicas=2 \
--set reportsController.replicas=2 \
--set admissionController.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="${ROLE_ARN}" \
--set backgroundController.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="${ROLE_ARN}" \
--set reportsController.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="${ROLE_ARN}"
```
Or via `values.yaml` for GitOps:
```yaml
# kyverno-values.yaml
admissionController:
replicas: 3
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/KyvernoECRRole"
backgroundController:
replicas: 2
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/KyvernoECRRole"
reportsController:
replicas: 2
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/KyvernoECRRole"
```
#### Step 4: Verify IRSA is working
```bash
# Confirm the AWS token is being projected into Kyverno pods
kubectl describe pod -n kyverno \
-l app.kubernetes.io/component=admission-controller \
| grep -A8 "aws-iam-token"
# Should show a projected volume:
# aws-iam-token:
# Type: Projected (a volume that contains injected data from multiple sources)
# TokenExpirationSeconds: 86400
# Watch Kyverno logs for ECR auth issues when a Pod is admitted
kubectl logs -n kyverno \
-l app.kubernetes.io/component=admission-controller \
--follow | grep -i "ecr\|unauthorized\|403\|registry"
```
---
### 4.8 ECR Token Refresh for Non-IRSA Environments
If your Kubernetes cluster cannot use IRSA (e.g., EKS Anywhere, on-premises Kubernetes pointing at ECR, or EKS with IRSA disabled), you must provide ECR credentials via a Kubernetes `Secret` and refresh it before the 12-hour token expiry.
#### Create the initial secret
```bash
kubectl create secret docker-registry ecr-pull-secret \
--namespace kyverno \
--docker-server=123456789012.dkr.ecr.us-east-1.amazonaws.com \
--docker-username=AWS \
--docker-password="$(aws ecr get-login-password --region us-east-1)"
```
#### Reference it in the Kyverno Helm values
```yaml
# kyverno-values.yaml
existingImagePullSecrets:
- ecr-pull-secret
```
#### Automate token rotation with a CronJob
```yaml
# ecr-token-refresher.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: ecr-token-refresher
namespace: kyverno
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: ecr-token-refresher
namespace: kyverno
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "create", "patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ecr-token-refresher
namespace: kyverno
subjects:
- kind: ServiceAccount
name: ecr-token-refresher
namespace: kyverno
roleRef:
kind: Role
name: ecr-token-refresher
apiGroup: rbac.authorization.k8s.io
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: ecr-token-refresher
namespace: kyverno
spec:
schedule: "0 */6 * * *" # every 6 hours — well within the 12h ECR token lifetime
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
spec:
serviceAccountName: ecr-token-refresher
restartPolicy: OnFailure
containers:
- name: refresher
image: amazon/aws-cli:latest
env:
- name: AWS_REGION
value: "us-east-1"
- name: ECR_REGISTRY
value: "123456789012.dkr.ecr.us-east-1.amazonaws.com"
command:
- /bin/sh
- -c
- |
set -e
TOKEN=$(aws ecr get-login-password --region ${AWS_REGION})
kubectl create secret docker-registry ecr-pull-secret \
--namespace kyverno \
--docker-server="${ECR_REGISTRY}" \
--docker-username=AWS \
--docker-password="${TOKEN}" \
--dry-run=client -o yaml | kubectl apply -f -
echo "ECR token refreshed at $(date)"
```
```bash
kubectl apply -f ecr-token-refresher.yaml
# Verify next scheduled run
kubectl get cronjob -n kyverno ecr-token-refresher
```
---
### 4.9 Cross-Account ECR: Sign in CI, Deploy in Prod
A common pattern is to build and sign images in a dedicated CI/tools AWS account, then run workloads from a separate production account. Kyverno in the production cluster needs to authenticate to the CI account's ECR and verify the signatures.
```
Account A (CI/Tools): 111111111111
ECR: 111111111111.dkr.ecr.us-east-1.amazonaws.com/myapp
AWS Signer profile in Account A
Account B (Production): 222222222222
EKS cluster
Kyverno: reads image + signature from Account A ECR
verifies against Account A's AWS Signer root CA
```
#### ECR repository policy on Account A — allow Account B to pull
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowCrossAccountNodePull",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::222222222222:role/EKSNodeGroupRole"
},
"Action": [
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability"
]
},
{
"Sid": "AllowCrossAccountKyvernoPull",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::222222222222:role/KyvernoIRSARole"
},
"Action": [
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability",
"ecr:DescribeImages",
"ecr:ListImages"
]
}
]
}
```
Apply in Account A:
```bash
aws ecr set-repository-policy \
--repository-name myapp \
--policy-text file://cross-account-policy.json \
--region us-east-1
```
#### IAM policy on Account B — allow Kyverno to authenticate to Account A's ECR
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ECRAuthSourceAccount",
"Effect": "Allow",
"Action": "ecr:GetAuthorizationToken",
"Resource": "*"
},
{
"Sid": "ECRPullFromSourceAccount",
"Effect": "Allow",
"Action": [
"ecr:BatchGetImage",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchCheckLayerAvailability",
"ecr:DescribeImages"
],
"Resource": "arn:aws:ecr:us-east-1:111111111111:repository/*"
},
{
"Sid": "SignerRevocationSourceAccount",
"Effect": "Allow",
"Action": "signer:GetRevocationStatus",
"Resource": "arn:aws:signer:us-east-1:111111111111:/signing-profiles/*"
}
]
}
```
> **Important:** `ecr:GetAuthorizationToken` must be called against **the source account's ECR endpoint** — the auth call itself is not cross-account. The cross-account access is granted by the resource policy on the repository. Always authenticate to `111111111111.dkr.ecr.us-east-1.amazonaws.com` (Account A's endpoint) when pulling images from Account A.
#### Kyverno ClusterPolicy for cross-account images
```yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-cross-account-ecr-images
spec:
validationFailureAction: Enforce
background: true
rules:
- name: verify-ci-account-images
match:
any:
- resources:
kinds: [Pod]
namespaces: ["production"]
verifyImages:
- type: Notary
# Images live in Account A (111111111111)
imageReferences:
- "111111111111.dkr.ecr.us-east-1.amazonaws.com/*"
attestors:
- entries:
- certificates:
# AWS Signer root CA — same CA regardless of account
cert: |-
-----BEGIN CERTIFICATE-----
<AWS Signer Notation Root CA>
-----END CERTIFICATE-----
mutateDigest: true
required: true
```
---
### 4.10 VPC Endpoints for Private Cluster Access
If your EKS cluster is fully private (no internet egress), you need AWS PrivateLink VPC Interface Endpoints so that nodes and Kyverno can reach ECR and AWS Signer without traversing the internet.
**Required endpoints:**
| Service | Endpoint type | Service name |
|---|---|---|
| ECR API | Interface | `com.amazonaws.<region>.ecr.api` |
| ECR Docker registry | Interface | `com.amazonaws.<region>.ecr.dkr` |
| S3 (ECR layer storage) | Gateway | `com.amazonaws.<region>.s3` |
| AWS Signer | Interface | `com.amazonaws.<region>.signer` |
| STS (for IRSA token exchange) | Interface | `com.amazonaws.<region>.sts` |
```bash
REGION="us-east-1"
VPC_ID="vpc-12345678"
SUBNET_IDS="subnet-aaa111 subnet-bbb222"
SG_ID="sg-abc123def456" # allow inbound 443 from EKS node SG
# 1. ECR API endpoint
aws ec2 create-vpc-endpoint \
--vpc-id ${VPC_ID} \
--service-name com.amazonaws.${REGION}.ecr.api \
--vpc-endpoint-type Interface \
--subnet-ids ${SUBNET_IDS} \
--security-group-ids ${SG_ID} \
--private-dns-enabled \
--region ${REGION}
# 2. ECR Docker registry endpoint
aws ec2 create-vpc-endpoint \
--vpc-id ${VPC_ID} \
--service-name com.amazonaws.${REGION}.ecr.dkr \
--vpc-endpoint-type Interface \
--subnet-ids ${SUBNET_IDS} \
--security-group-ids ${SG_ID} \
--private-dns-enabled \
--region ${REGION}
# 3. S3 Gateway endpoint (ECR image layers are stored in S3)
aws ec2 create-vpc-endpoint \
--vpc-id ${VPC_ID} \
--service-name com.amazonaws.${REGION}.s3 \
--vpc-endpoint-type Gateway \
--route-table-ids rtb-123456 \
--region ${REGION}
# 4. AWS Signer endpoint (revocation checks by Notation/Kyverno)
aws ec2 create-vpc-endpoint \
--vpc-id ${VPC_ID} \
--service-name com.amazonaws.${REGION}.signer \
--vpc-endpoint-type Interface \
--subnet-ids ${SUBNET_IDS} \
--security-group-ids ${SG_ID} \
--private-dns-enabled \
--region ${REGION}
# 5. STS endpoint (required for IRSA token exchange)
aws ec2 create-vpc-endpoint \
--vpc-id ${VPC_ID} \
--service-name com.amazonaws.${REGION}.sts \
--vpc-endpoint-type Interface \
--subnet-ids ${SUBNET_IDS} \
--security-group-ids ${SG_ID} \
--private-dns-enabled \
--region ${REGION}
```
**Security group for endpoints** — inbound rule:
```bash
# Allow inbound HTTPS from EKS node security group
aws ec2 authorize-security-group-ingress \
--group-id ${SG_ID} \
--protocol tcp \
--port 443 \
--source-group <eks-node-security-group-id> \
--region ${REGION}
```
**Validate connectivity from inside the cluster:**
```bash
kubectl run ecr-test --image=amazon/aws-cli:latest --restart=Never \
--namespace=default \
--command -- aws ecr describe-repositories --region us-east-1
kubectl logs ecr-test
kubectl delete pod ecr-test
```
---
### 4.11 Verifying AWS Signer Revocation Status
When Notation (or Kyverno's `verifyImages`) verifies a signature produced by the AWS Signer plugin, it automatically calls `signer:GetRevocationStatus` to confirm the signing job and profile have not been revoked. This call goes to the AWS Signer endpoint — in a private VPC you need the Signer VPC endpoint from 4.10.
**Check revocation status manually:**
```bash
# Get the signing job ID from the signature
notation inspect 123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp@sha256:abc123
# Output includes:
# Signed Attributes:
# signingTime: 2025-11-01 10:30:00 +0000 UTC
# Unsigned Attributes:
# signingAgent: AWS Signer/1.0
# timestamp: ...
# signingJobId: abc-def-123-456 <── use this
# Check job revocation
aws signer describe-signing-job \
--job-id "abc-def-123-456" \
--region us-east-1 \
--query '{status: status, statusReason: statusReason, revocationRecord: revocationRecord}'
```
**Revoke a compromised signature:**
```bash
# Revoke a single signing job (revokes only this image's signature)
aws signer revoke-signature \
--job-id "abc-def-123-456" \
--reason "Image found to contain malware" \
--region us-east-1
# Revoke all signatures from a profile version (emergency — affects all images signed with this version)
PROFILE_VERSION=$(aws signer get-signing-profile \
--profile-name "production-profile" \
--query 'profileVersionArn' --output text \
--region us-east-1)
aws signer revoke-signing-profile \
--profile-name "production-profile" \
--profile-version "${PROFILE_VERSION##*/}" \
--effective-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
--reason "Signing key compromise" \
--region us-east-1
```
After revocation, Kyverno's `verifyImages` will start blocking newly scheduled Pods that reference the revoked images (on their next admission check). Existing running Pods are not evicted — use `kubectl rollout restart` on affected Deployments to force re-admission.
---
## 5. Kyverno — Policy Enforcement for Signed Images
### 5.1 Installation
**Via Helm (recommended for production)**
```bash
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
# Install Kyverno with HA (3 replicas)
helm install kyverno kyverno/kyverno \
--namespace kyverno \
--create-namespace \
--set admissionController.replicas=3 \
--set backgroundController.replicas=2 \
--set cleanupController.replicas=2 \
--set reportsController.replicas=2 \
--version 3.2.6
# Verify pods are running
kubectl get pods -n kyverno
```
**Check the webhooks are registered**
```bash
kubectl get validatingwebhookconfigurations | grep kyverno
kubectl get mutatingwebhookconfigurations | grep kyverno
```
---
### 5.2 How Kyverno Intercepts Workloads
```
kubectl apply (Pod/Deployment/etc.)
│
▼
kube-apiserver
│ AdmissionReview request
▼
Kyverno Admission Controller (webhook)
│
├── verifyImages rule fires
│ │
│ ▼
│ Fetch OCI referrers (signatures) from registry
│ │
│ ▼
│ Verify signature against trust material
│ │
│ ✅ Valid ──► ADMIT (optionally mutate image ref to digest)
│ ❌ Invalid ──► DENY with descriptive error
```
Kyverno evaluates **every** resource that creates a Pod: `Pod`, `Deployment`, `StatefulSet`, `DaemonSet`, `Job`, `CronJob`, `ReplicaSet`.
---
### 5.3 Verify Notation Signatures
Kyverno's `verifyImages` rule supports Notation natively. You provide the certificate(s) and a trust policy inline.
```yaml
# kyverno-notation-verify.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-notation-signatures
annotations:
policies.kyverno.io/title: Verify Notation Signatures
policies.kyverno.io/description: >
Blocks any Pod from running an image that does not carry a valid
Notation signature from the Acme Corp signing CA.
spec:
validationFailureAction: Enforce # or Audit
background: true
rules:
- name: verify-image-signature
match:
any:
- resources:
kinds:
- Pod
namespaces:
- "production"
- "staging"
verifyImages:
- type: Notary
imageReferences:
- "123456789012.dkr.ecr.us-east-1.amazonaws.com/*"
attestors:
- entries:
- certificates:
cert: |-
-----BEGIN CERTIFICATE-----
<BASE64 DER of your signing CA certificate>
-----END CERTIFICATE-----
```
**Key fields explained:**
| Field | Purpose |
|---|---|
| `type: Notary` | Use the Notation/Notary verification path |
| `imageReferences` | Glob patterns — which images this rule covers |
| `attestors[].entries[].certificates.cert` | The CA or leaf cert used to verify the signature chain |
| `validationFailureAction: Enforce` | Deny admission; use `Audit` to log-only during rollout |
---
### 5.4 Verify AWS Signer Signatures
```yaml
# kyverno-aws-signer-verify.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-aws-signer-signatures
annotations:
policies.kyverno.io/title: Verify AWS Signer Signatures
policies.kyverno.io/description: >
All images in production must be signed by the AWS Signer
profile used in the CI pipeline. Unsigned images are denied.
spec:
validationFailureAction: Enforce
background: true
rules:
- name: verify-aws-signed-image
match:
any:
- resources:
kinds:
- Pod
exclude:
any:
- resources:
namespaces:
- kube-system
- kyverno
verifyImages:
- type: Notary
imageReferences:
- "123456789012.dkr.ecr.us-east-1.amazonaws.com/*"
attestors:
- entries:
- certificates:
# AWS Signer Notation root CA
# Download from: https://d2hvyiie56hcat.cloudfront.net/aws-signer-notation-root.cert.pem
cert: |-
-----BEGIN CERTIFICATE-----
MIIBtjCCAVygAwIBAgITBmyfz5m/jAo54vB4ikPmljZbyjAKBggqhkjOPQQDAzA5
<... AWS Signer root CA content ...>
-----END CERTIFICATE-----
rekor:
url: "" # leave empty; AWS Signer uses its own revocation
```
> **Getting the AWS Signer root CA:** Download it from the official URL and base64-encode it:
> ```bash
> curl -s https://d2hvyiie56hcat.cloudfront.net/aws-signer-notation-root.cert.pem
> ```
---
### 5.5 Block Unsigned Images — Full Policy Examples
#### Example 1: Strict — Block all unsigned images cluster-wide
```yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: block-unsigned-images
spec:
validationFailureAction: Enforce
background: false # webhooks only, no background scan for this rule
rules:
- name: check-image-signature
match:
any:
- resources:
kinds: [Pod]
exclude:
any:
- resources:
# Exclude system namespaces — adjust to your environment
namespaces:
- kube-system
- kube-public
- kyverno
- cert-manager
verifyImages:
- type: Notary
imageReferences:
- "*" # All images
attestors:
- entries:
- certificates:
cert: |-
-----BEGIN CERTIFICATE-----
<YOUR SIGNING CA CERTIFICATE>
-----END CERTIFICATE-----
# Mutate image reference to digest to prevent tag mutation
mutateDigest: true
# Require the image to be referenced by digest in the spec
verifyDigest: true
# Require the image to be present in the registry
required: true
```
#### Example 2: Per-namespace enforcement with different trust anchors
```yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-images-by-environment
spec:
validationFailureAction: Enforce
background: true
rules:
# Production: must be signed by production signing profile
- name: verify-production-images
match:
any:
- resources:
kinds: [Pod]
namespaces: ["production"]
verifyImages:
- type: Notary
imageReferences:
- "123456789012.dkr.ecr.us-east-1.amazonaws.com/*"
attestors:
- entries:
- certificates:
cert: |-
-----BEGIN CERTIFICATE-----
<PRODUCTION CA CERT>
-----END CERTIFICATE-----
mutateDigest: true
required: true
# Staging: signed by either production or staging profile
- name: verify-staging-images
match:
any:
- resources:
kinds: [Pod]
namespaces: ["staging"]
verifyImages:
- type: Notary
imageReferences:
- "123456789012.dkr.ecr.us-east-1.amazonaws.com/*"
attestors:
- count: 1 # at least 1 of the following must match
entries:
- certificates:
cert: |-
-----BEGIN CERTIFICATE-----
<PRODUCTION CA CERT>
-----END CERTIFICATE-----
- certificates:
cert: |-
-----BEGIN CERTIFICATE-----
<STAGING CA CERT>
-----END CERTIFICATE-----
mutateDigest: true
required: true
```
#### Example 3: Block images with expired signatures
```yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: block-expired-signatures
spec:
validationFailureAction: Enforce
background: true
rules:
- name: check-signature-expiry
match:
any:
- resources:
kinds: [Pod]
namespaces: ["production"]
verifyImages:
- type: Notary
imageReferences:
- "123456789012.dkr.ecr.us-east-1.amazonaws.com/*"
attestors:
- entries:
- certificates:
cert: |-
-----BEGIN CERTIFICATE-----
<CA CERT>
-----END CERTIFICATE-----
# Notation already checks signature validity window —
# this setting makes Kyverno also enforce it
required: true
mutateDigest: true
```
#### Example 4: Require custom annotations in the signature
```yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-signed-with-build-id
spec:
validationFailureAction: Enforce
background: true
rules:
- name: verify-signature-annotations
match:
any:
- resources:
kinds: [Pod]
namespaces: ["production"]
verifyImages:
- type: Notary
imageReferences:
- "123456789012.dkr.ecr.us-east-1.amazonaws.com/*"
attestors:
- entries:
- certificates:
cert: |-
-----BEGIN CERTIFICATE-----
<CA CERT>
-----END CERTIFICATE-----
# Verify that the signature carries a required annotation
conditions:
- any:
- key: "{{ request.object.metadata.annotations.\"io.acme/build-id\" }}"
operator: NotEquals
value: ""
required: true
mutateDigest: true
```
---
### 5.6 Audit Mode vs. Enforce Mode
| Mode | `validationFailureAction` | Behavior |
|---|---|---|
| **Audit** | `Audit` | Policy violations are **logged** as `PolicyReport` objects but workloads are **admitted**. Use during initial rollout. |
| **Enforce** | `Enforce` | Policy violations **block** admission with a descriptive error message. |
**Recommended rollout strategy:**
```
Week 1-2: Deploy policy in Audit mode
── Monitor PolicyReports for violations
── Fix unsigned images in your pipelines
Week 3: Switch non-production namespaces to Enforce
── Fix remaining violations
Week 4+: Switch production namespaces to Enforce
```
```bash
# View policy reports
kubectl get policyreport -A
kubectl get clusterpolicyreport
# Describe a specific report
kubectl describe policyreport -n production <report-name>
```
---
### 5.7 Testing Policies
```bash
# Test with a known-unsigned image (should be blocked)
kubectl run test-unsigned \
--image=public.ecr.aws/amazonlinux/amazonlinux:latest \
--namespace=production
# Error from server: admission webhook "mutate.kyverno.svc" denied the request:
# image public.ecr.aws/amazonlinux/amazonlinux:latest failed signature verification
# Test with a known-signed image (should succeed)
kubectl run test-signed \
--image=123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp@sha256:abc123 \
--namespace=production
# pod/test-signed created
# Dry-run a manifest
kubectl apply --dry-run=server -f deployment.yaml
# Use kyverno CLI for offline testing
kyverno apply kyverno-aws-signer-verify.yaml \
--resource test-pod.yaml \
--registry-access
```
**Install the Kyverno CLI:**
```bash
brew install kyverno
# or
curl -LO https://github.com/kyverno/kyverno/releases/latest/download/kyverno_linux_x86_64.tar.gz
```
---
## 6. End-to-End Workflow
Below is a complete CI/CD pipeline integrating all three tools.
```
┌─────────────────────────────────────────────────────────────┐
│ CI/CD Pipeline (GitHub Actions / GitLab CI / Jenkins) │
│ │
│ 1. docker build -t ${IMAGE}:${TAG} . │
│ 2. docker push ${IMAGE}:${TAG} │
│ 3. DIGEST=$(crane digest ${IMAGE}:${TAG}) │
│ 4. notation sign --key aws-signer-key ${IMAGE}@${DIGEST} │
│ 5. notation verify ${IMAGE}@${DIGEST} ← self-check │
└─────────────────────────────────────────────────────────────┘
│
▼
Registry: Image + OCI signature artifact
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Kubernetes Cluster │
│ │
│ Developer: kubectl apply -f deployment.yaml │
│ │ │
│ ▼ │
│ Kyverno Admission Webhook │
│ ── fetches signature from registry │
│ ── verifies against AWS Signer root CA │
│ ── checks signature timestamp + validity │
│ ── mutates image ref to digest (optional) │
│ │ │
│ ADMIT or DENY │
└─────────────────────────────────────────────────────────────┘
```
**GitHub Actions example:**
```yaml
# .github/workflows/build-sign-push.yml
name: Build, Sign & Push
on:
push:
branches: [main]
permissions:
id-token: write # for OIDC auth to AWS
contents: read
jobs:
build-sign-push:
runs-on: ubuntu-latest
env:
AWS_REGION: us-east-1
ECR_REGISTRY: 123456789012.dkr.ecr.us-east-1.amazonaws.com
ECR_REPOSITORY: myapp
IMAGE_TAG: ${{ github.sha }}
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials (OIDC)
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsSigningRole
aws-region: ${{ env.AWS_REGION }}
- name: Login to ECR
uses: aws-actions/amazon-ecr-login@v2
- name: Build and push image
id: build
run: |
IMAGE="${ECR_REGISTRY}/${ECR_REPOSITORY}:${IMAGE_TAG}"
docker build -t "${IMAGE}" .
docker push "${IMAGE}"
DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' "${IMAGE}")
echo "digest=${DIGEST}" >> "$GITHUB_OUTPUT"
- name: Install Notation
run: |
curl -Lo notation.tar.gz \
https://github.com/notaryproject/notation/releases/download/v1.2.0/notation_1.2.0_linux_amd64.tar.gz
tar -xzf notation.tar.gz -C /usr/local/bin notation
- name: Install AWS Signer plugin
run: |
curl -Lo plugin.zip \
https://d2hvyiie56hcat.cloudfront.net/linux/amd64/plugin/1.0.298/notation-com.amazonaws.signer.notation.plugin_linux_amd64.zip
unzip plugin.zip
notation plugin install --file ./notation-com.amazonaws.signer.notation.plugin
- name: Add signing key
run: |
notation key add \
--plugin com.amazonaws.signer.notation.plugin \
--id "arn:aws:signer:${AWS_REGION}:123456789012:/signing-profiles/my-profile" \
--plugin-config "aws-region=${AWS_REGION}" \
"ci-signing-key"
- name: Sign image
run: |
notation sign \
--key "ci-signing-key" \
--annotation "github.com/workflow=${{ github.workflow }}" \
--annotation "github.com/sha=${{ github.sha }}" \
"${{ steps.build.outputs.digest }}"
- name: Verify signature (self-check)
run: |
notation verify "${{ steps.build.outputs.digest }}"
```
---
## 7. Best Practices
### 7.1 Notation Best Practices
**Key & Certificate Management**
- **Never sign with self-signed certs in production.** Use an internal CA (AWS Private CA, HashiCorp Vault PKI, or a corporate CA) for trust chain validation.
- **Use short-lived signing certificates** (e.g., 90-day leaves) with an online OCSP responder or CRL endpoint to allow rapid revocation.
- **Rotate signing keys** on a schedule. Notation supports multiple trust policies so old and new keys can coexist during rotation.
- Store CA private keys in an HSM or managed service (AWS KMS). The signing key should never exist on a build agent filesystem.
**Signing Workflow**
- Always sign by **image digest** (`sha256:...`), never by tag. Tags are mutable — a signed tag can later point to an unsigned or malicious image.
- Sign **after the image is pushed** to the registry, not before — the signature references the registry-assigned digest.
- Perform a **verification self-check** in the pipeline immediately after signing (`notation verify`). Fail the pipeline if this step fails.
- Store the signing command output and trust policy as **pipeline artifacts** for audit purposes.
**Trust Policy**
- Scope `registryScopes` as narrowly as possible (specific repositories, not `*`).
- Set `signatureVerification.level` to `strict` in production. Available levels:
- `strict` — all checks enforced
- `permissive` — some checks are logged but not enforced
- `audit` — all checks logged
- `skip` — no verification (never use in production)
- Use `trustedIdentities` to pin the expected certificate subject (CN, O, etc.) rather than trusting the entire CA for any subject.
---
### 7.2 AWS Signer Best Practices
**Profile & Key Management**
- Create **separate signing profiles** per environment (dev/staging/production) so a compromised dev key cannot sign production images.
- Set an appropriate **signature validity period** on the profile. Match this to your image lifecycle (e.g., 12 months for long-lived services, 3 months for frequently updated images).
- Use **IAM conditions** to restrict which CI roles can invoke which signing profiles:
```json
{
"Effect": "Allow",
"Action": "signer:StartSigningJob",
"Resource": "arn:aws:signer:*:*:/signing-profiles/production-profile",
"Condition": {
"StringEquals": {
"aws:PrincipalTag/Environment": "production"
}
}
}
```
- Enable **AWS CloudTrail** for `signer.amazonaws.com` events to maintain an audit log of all signing operations.
- Use **SCPs (Service Control Policies)** in AWS Organizations to prevent signing profiles from being deleted without approval.
**Revocation**
- AWS Signer supports **signature revocation** at the job level and profile level. Integrate revocation checks into your verify step.
- Have a documented **revocation runbook** — know how to revoke a compromised signature before an incident occurs:
```bash
# Revoke a specific signing job (single signature)
aws signer revoke-signature \
--job-id "abc-123-signing-job-id" \
--reason "Key compromise" \
--region us-east-1
# Revoke all signatures from a profile (nuclear option)
aws signer revoke-signing-profile \
--profile-name "compromised-profile" \
--profile-version "abc123" \
--effective-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
--reason "Profile compromise" \
--region us-east-1
```
**ECR Integration**
- Enable **ECR image scanning** alongside signing — signatures prove provenance, scanning proves hygiene.
- Use **ECR lifecycle policies** to clean up untagged signature artifacts that outlive their parent image.
- Consider enabling **ECR immutable tags** to prevent tag mutation independent of signing.
---
### 7.3 Kyverno Best Practices
**Deployment & Reliability**
- Run **at least 3 replicas** of the Kyverno admission controller in production to avoid a single point of failure.
- Configure **PodDisruptionBudgets** for Kyverno to prevent all replicas from being evicted simultaneously.
- Set `failurePolicy: Fail` on the webhook configuration in production (it's the default) — this means if Kyverno is unavailable, admission is blocked. This is safer than `Ignore` (which would admit all traffic including unsigned images).
- Keep Kyverno's **webhook timeout** low (default 10s) and ensure your registry responds quickly. Consider using **Kyverno's image pull secret** feature for private registries.
**Policy Design**
- Use `exclude` blocks to exempt system namespaces (`kube-system`, `kyverno`) to avoid accidental lockout.
- Set `background: true` to also scan **existing** resources for compliance (generates `PolicyReport` objects).
- Use `mutateDigest: true` in `verifyImages` to automatically pin image references to their digest upon admission — this prevents post-admission tag swapping.
- Scope policies to specific `imageReferences` patterns rather than `"*"` — this avoids unintended failures for sidecar containers injected by service meshes.
- Use **`PolicyException`** for break-glass scenarios rather than disabling policies:
```yaml
apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
name: emergency-exception
namespace: production
spec:
exceptions:
- policyName: verify-aws-signer-signatures
ruleNames: ["verify-aws-signed-image"]
match:
any:
- resources:
kinds: [Pod]
names: ["emergency-debug-pod"]
namespaces: ["production"]
```
**Monitoring & Alerting**
- Monitor the `PolicyReport` and `ClusterPolicyReport` CRDs for violations:
```bash
kubectl get policyreport -A -o json | \
jq '.items[] | select(.summary.fail > 0) | {name: .metadata.name, ns: .metadata.namespace, fail: .summary.fail}'
```
- Export Kyverno metrics to Prometheus and alert on:
- `kyverno_policy_results_total{status="fail"}` — policy violations
- `kyverno_admission_requests_total{resource_request_operation="CREATE"}` — admission volume
- Webhook response times > 5 seconds
**RBAC**
- Limit who can create/update/delete Kyverno policies. Only platform/security teams should have write access to `ClusterPolicy` and `Policy` resources.
- Use a separate `ServiceAccount` for Kyverno with only the permissions it needs — don't give it cluster-admin.
---
### 7.4 ECR Best Practices
- **Enable immutable tags on all production repositories.** One misconfigured `docker push` should not overwrite a signed image.
- **Encrypt ECR repositories with a customer-managed KMS key** — this allows you to revoke decryption access to an entire repository by disabling the key.
- **Separate ECR repositories per environment** (e.g., `myapp-dev`, `myapp-staging`, `myapp-prod`) with separate IAM policies and signing profiles. Never share a production repository with development.
- **Use ECR replication for multi-region deployments** rather than pushing and signing the same image to multiple regions. Sign once in the source region; the signature replicates alongside the image. Confirm OCI referrers replicate correctly in your ECR version.
- **Enable ECR scan-on-push** (`image_scanning_configuration.scan_on_push = true`) — combine signature verification (provenance) with vulnerability scanning (hygiene).
- **Do not delete ECR images before removing workloads that reference them.** If Kyverno attempts to verify a signature for an image that no longer exists in ECR, the admission will fail. Delete images from ECR only after confirming no running workloads reference them.
- **Store the AWS Signer root CA certificate in AWS Secrets Manager or Parameter Store** rather than hardcoding it in Kyverno `ClusterPolicy` manifests. Reference it dynamically or automate updates when AWS rotates the root CA.
- **Monitor ECR image pull errors** via CloudWatch Metrics → `AWS/ECR` namespace. Spikes in `RegistryPullThroughCacheRequestErrors` or `RepositoryEventCount` can indicate policy-related auth failures.
- **Use ECR pull-through cache** for public base images (Docker Hub, ECR Public, Quay) so all images — including base images — flow through your controlled registry and can be signed before being used.
---
### 7.5 General Supply-Chain Security Best Practices
**Defense in Depth**
- Signing is one layer — combine it with:
- **Vulnerability scanning** (Trivy, ECR scanning, Snyk)
- **SBOM generation** (Syft) and attestation (also storable as OCI artifacts)
- **Base image pinning** and regular updates
- **Admission control** (Kyverno, OPA/Gatekeeper)
- **Runtime security** (Falco, AWS GuardDuty for EKS)
**Key Compromise Incident Response**
1. Immediately revoke compromised signatures (AWS Signer revocation or Notation CRL).
2. Rotate the signing key/profile.
3. Re-sign all images in the registry that were signed with the compromised key.
4. Audit recent deployments to identify potentially malicious images.
5. Update trust policies to remove the old certificate.
**Audit & Compliance**
- Maintain an immutable audit log: which images were signed, when, by whom, and from which pipeline run. AWS CloudTrail covers the AWS Signer side; supplement with pipeline artifact storage.
- Periodically **re-verify** all running workloads against current policies using Kyverno background scanning (`background: true`).
- Run **periodic drills**: attempt to deploy an unsigned image to confirm enforcement is working.
**Image Naming**
- Adopt a consistent naming convention: `<registry>/<org>/<repo>:<semver>@sha256:<digest>`
- Always reference images by digest in Kubernetes manifests (Kyverno's `mutateDigest` can enforce this automatically).
---
## 8. Troubleshooting Reference
| Symptom | Likely Cause | Resolution |
|---|---|---|
| `notation sign` fails with `401 Unauthorized` | Not authenticated to ECR | Run `aws ecr get-login-password \| docker login ...` or configure the ECR credential helper |
| `notation sign` fails with `no basic auth credentials` | Docker config missing credentials for the registry | Confirm `~/.docker/config.json` has `credHelpers` entry for the ECR registry endpoint |
| `notation verify` fails with `no signature found` | Wrong digest, or signing step was skipped | Run `notation ls <digest>` — if empty, the sign step failed silently; check CI pipeline logs |
| `notation verify` fails with `certificate expired` | Signing certificate has passed its validity period | Re-sign with a valid certificate; rotate the signing profile |
| `notation verify` fails with `trust policy not found` | `trustpolicy.json` missing or scope mismatch | Check `~/.config/notation/trustpolicy.json`; the `registryScopes` must exactly match the ECR URL prefix |
| `notation sign` succeeds but `notation ls` shows empty | OCI referrers not stored — possible ECR endpoint issue | Confirm you authenticated to the correct ECR endpoint; check for `ecr:PutImage` permission on the signing role |
| AWS Signer plugin not found by Notation | Plugin binary not in correct Notation plugin directory | Re-run `notation plugin install --file ...` and verify with `notation plugin ls` |
| AWS Signer sign fails with `AccessDeniedException` | CI role missing `signer:StartSigningJob` permission or wrong profile ARN | Check IAM policy and confirm `--id` flag matches the full signing profile ARN |
| Kyverno blocks all Pods after policy applied | `exclude` block missing for system namespaces | Add `exclude.any.resources.namespaces` for `kube-system`, `kyverno`, `cert-manager` |
| Kyverno `verifyImages` fails with `401` or `403` | Kyverno cannot authenticate to ECR | Confirm IRSA annotations on Kyverno service accounts; or refresh `ecr-pull-secret` |
| Kyverno webhook timeout (`context deadline exceeded`) | ECR or AWS Signer endpoint unreachable from cluster | For private clusters: add VPC endpoints for `ecr.api`, `ecr.dkr`, `s3`, `signer`, `sts` (see §4.10) |
| `PolicyReport` shows violations but Pods still run | Policy is in `Audit` mode | Change `validationFailureAction` to `Enforce` |
| ECR token expired in Kyverno (non-IRSA cluster) | 12-hour ECR token lifetime exceeded | Confirm the `ecr-token-refresher` CronJob is running and succeeding every 6 hours |
| Cross-account pull fails (`Repository does not exist`) | ECR resource policy on source account missing target account principal | Add the production account's roles to the ECR repository policy in the CI account (see §4.9) |
| Revocation check fails for AWS Signer (`connection refused`) | Signer VPC endpoint missing in private cluster | Create the `com.amazonaws.<region>.signer` Interface endpoint (see §4.10) |
| Running Pods not evicted after signature revoked | Kyverno only checks on admission, not at runtime | Run `kubectl rollout restart deployment/<name>` to trigger re-admission; or use Kyverno background scan + external eviction logic |
---
*Document version: 2.0 · Tools: Notation v1.2 · AWS Signer Notation Plugin v1.0.298 · Kyverno v3.2 · Amazon ECR*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment