electricjesus/calico-istio-alp-ai-agent-guide.md

## calico-istio-alp-ai-agent-guide.md

      
    Raw
  

              calico-istio-alp-ai-agent-guide.md
            
          
    Calico + Istio App Layer Policy Setup - AI Agent Execution Guide

Target: AI code agents (Claude Code, GitHub Copilot Workspace, etc.)
Audience: Automated execution systems
Cluster Requirements: Kubernetes cluster with Calico CNI installed

Execution Prerequisites Check

Before starting, verify these conditions. If any check fails, follow the remediation steps.
Check 1: Kubernetes Cluster Access

kubectl cluster-info
Expected: Cluster info displayed successfully
If fails: Set correct KUBECONFIG or ensure cluster is running
Check 2: Calico is Installed

kubectl get pods -n calico-system
Expected: Calico pods in Running state
If fails: This guide requires Calico to be pre-installed. Refer to Calico installation docs.
Check 3: Calico Version

kubectl get deployment -n calico-system calico-apiserver -o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || kubectl get daemonset -n calico-system calico-node -o jsonpath='{.spec.template.spec.containers[0].image}'
Expected: Image tag showing v3.28+ (v3.31+ recommended)
If older: Consider upgrading Calico for best compatibility

Step 0: Install Required Tools

Install kubectl (if not present)

Check if kubectl exists:
command -v kubectl >/dev/null 2>&1 && echo "kubectl installed" || echo "kubectl missing"
If missing, install:
# Linux x86_64
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
mkdir -p ~/.local/bin
mv kubectl ~/.local/bin/
export PATH="$HOME/.local/bin:$PATH"
Verify:
kubectl version --client
Install istioctl (if not present)

Check if istioctl exists:
command -v istioctl >/dev/null 2>&1 && echo "istioctl installed" || echo "istioctl missing"
If missing, install Istio 1.28.1:
cd /tmp
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.28.1 sh -
chmod +x istio-1.28.1/bin/istioctl
mkdir -p ~/.local/bin
cp istio-1.28.1/bin/istioctl ~/.local/bin/
export PATH="$HOME/.local/bin:$PATH"
Verify:
istioctl version
Expected: Shows client version 1.28.1 (server version may not be shown if Istio not yet installed)
Install calicoctl (optional, not required for this guide)

Check if calicoctl exists:
command -v calicoctl >/dev/null 2>&1 && echo "calicoctl installed" || echo "calicoctl missing"
If missing and you want it:
cd /tmp
curl -L https://github.com/projectcalico/calico/releases/download/v3.31.2/calicoctl-linux-amd64 -o calicoctl
chmod +x calicoctl
mkdir -p ~/.local/bin
mv calicoctl ~/.local/bin/
export PATH="$HOME/.local/bin:$PATH"
Verify:
calicoctl version

Step 1: Enable Calico Policy Sync API

Enable the Policy Sync API in Felix to allow Dikastes to communicate with Felix:
kubectl patch felixconfiguration default --type merge -p '{"spec":{"policySyncPathPrefix":"/var/run/nodeagent"}}'
Expected: felixconfiguration.crd.projectcalico.org/default patched
Verify the setting:
kubectl get felixconfiguration default -o yaml | grep policySyncPathPrefix
Expected: policySyncPathPrefix: /var/run/nodeagent
If FelixConfiguration doesn't exist: Create it first:
cat <<EOF | kubectl apply -f -
apiVersion: projectcalico.org/v3
kind: FelixConfiguration
metadata:
  name: default
spec:
  policySyncPathPrefix: /var/run/nodeagent
EOF

Step 2: Install Calico CSI Driver

The CSI driver is required to mount the Felix Policy Sync socket into Dikastes containers.
Check if CSI driver is already installed:
kubectl get daemonset -n calico-system csi-node-driver 2>/dev/null && echo "CSI driver exists" || echo "CSI driver missing"
If missing, install it:
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.31.2/manifests/csi-driver.yaml
Expected:
serviceaccount/csi-node-driver created
clusterrole.rbac.authorization.k8s.io/csi-node-driver created
clusterrolebinding.rbac.authorization.k8s.io/csi-node-driver created
daemonset.apps/csi-node-driver created

Wait for CSI driver to be ready (required before proceeding):
kubectl wait --for=condition=ready pod -n calico-system -l k8s-app=csi-node-driver --timeout=120s
Expected: All CSI driver pods ready
Verify CSI driver is running on all nodes:
kubectl get pods -n calico-system -l k8s-app=csi-node-driver -o wide
Expected: One pod per node in Running state

Step 3: Check if Istio is Already Installed

kubectl get namespace istio-system 2>/dev/null && echo "istio-system namespace exists" || echo "istio-system namespace missing"
If istio-system exists, check what's installed:
kubectl get pods -n istio-system
If Istio is already installed:

Verify version is compatible (1.18+): istioctl version
If version is 1.18+, you can proceed to Step 5 (skip fresh installation)
If version is older, consider upgrading Istio first

If Istio is NOT installed: Proceed to Step 4

Step 4: Install Istio with Dikastes Templates

This step installs Istio WITH the Dikastes injection templates in a single operation.
Create the IstioOperator manifest with embedded Dikastes templates:
cat <<'EOF' > /tmp/istio-operator-dikastes.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-with-dikastes
  namespace: istio-system
spec:
  profile: minimal

  values:
    sidecarInjectorWebhook:
      templates:
        # Template for workload pods (non-root)
        dikastes: |
          spec:
            containers:
            - name: dikastes
              image: quay.io/calico/dikastes:v3.31.2
              args:
              - server
              - -l
              - /var/run/dikastes/dikastes.sock
              - -d
              - /var/run/felix/nodeagent/socket
              securityContext:
                allowPrivilegeEscalation: false
                runAsGroup: 999
                runAsNonRoot: true
                runAsUser: 999
              livenessProbe:
                exec:
                  command:
                  - /healthz
                  - liveness
                initialDelaySeconds: 3
                periodSeconds: 3
              readinessProbe:
                exec:
                  command:
                  - /healthz
                  - readiness
                initialDelaySeconds: 3
                periodSeconds: 3
              volumeMounts:
              - mountPath: /var/run/dikastes
                name: dikastes-sock
              - mountPath: /var/run/felix
                name: felix-sync
            # CRITICAL: Patch istio-proxy to mount dikastes socket
            # This allows Envoy to communicate with Dikastes via Unix socket
            initContainers:
            - name: istio-proxy
              volumeMounts:
              - mountPath: /var/run/dikastes
                name: dikastes-sock
            volumes:
            - name: dikastes-sock
              emptyDir:
                medium: Memory
            - name: felix-sync
              csi:
                driver: csi.tigera.io

        # Template for gateway pods (runs as root)
        dikastes-gateway: |
          spec:
            containers:
            - name: dikastes
              image: quay.io/calico/dikastes:v3.31.2
              args:
              - server
              - -l
              - /var/run/dikastes/dikastes.sock
              - -d
              - /var/run/felix/nodeagent/socket
              securityContext:
                allowPrivilegeEscalation: false
                runAsGroup: 0
                runAsNonRoot: false
                runAsUser: 0
              livenessProbe:
                exec:
                  command:
                  - /healthz
                  - liveness
                initialDelaySeconds: 3
                periodSeconds: 3
              readinessProbe:
                exec:
                  command:
                  - /healthz
                  - readiness
                initialDelaySeconds: 3
                periodSeconds: 3
              volumeMounts:
              - mountPath: /var/run/dikastes
                name: dikastes-sock
              - mountPath: /var/run/felix
                name: felix-sync
            # CRITICAL: Patch istio-proxy to mount dikastes socket
            initContainers:
            - name: istio-proxy
              volumeMounts:
              - mountPath: /var/run/dikastes
                name: dikastes-sock
            volumes:
            - name: dikastes-sock
              emptyDir:
                medium: Memory
            - name: felix-sync
              csi:
                driver: csi.tigera.io
EOF
Install Istio with the Dikastes templates:
istioctl install -f /tmp/istio-operator-dikastes.yaml -y
Expected:
✔ Istio core installed
✔ Istiod installed
✔ Installation complete

Wait for Istio control plane to be ready:
kubectl wait --for=condition=ready pod -n istio-system -l app=istiod --timeout=300s
Expected: pod/istiod-xxx condition met

Step 5: Update Existing Istio with Dikastes Templates (Alternative to Step 4)

Use this step ONLY if Istio was already installed (you skipped Step 4).
If Istio is already running, you can update it to add Dikastes templates:
# Create the same manifest as in Step 4
cat <<'EOF' > /tmp/istio-operator-dikastes.yaml
# [Same content as Step 4 - full IstioOperator YAML]
EOF

# Update existing Istio installation
istioctl install -f /tmp/istio-operator-dikastes.yaml -y
This will merge the Dikastes templates into your existing Istio configuration.

Step 6: Verify Dikastes Templates are Loaded

Check that the templates were loaded into the sidecar injector:
kubectl get configmap -n istio-system istio-sidecar-injector -o yaml | grep "dikastes:" -A 5
Expected: You should see both dikastes: and dikastes-gateway: template definitions
Count the templates:
kubectl get configmap -n istio-system istio-sidecar-injector -o yaml | grep -c "dikastes:"
Expected: 2 (one for each template)
If templates are missing: Re-run Step 4 or Step 5, ensuring no errors occurred

Step 7: Deploy Envoy Authorization Filter

Deploy the EnvoyFilter, ServiceEntry, and DestinationRule for Dikastes integration:
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.31.2/manifests/alp/istio-app-layer-policy-envoy-v3.yaml
Expected:
serviceentry.networking.istio.io/dikastes created
destinationrule.networking.istio.io/dikastes-dr created
envoyfilter.networking.istio.io/ext-authz created

Verify the resources were created:
kubectl get serviceentry,destinationrule,envoyfilter -n istio-system
Expected: Should show dikastes, dikastes-dr, and ext-authz

Step 8: Create Test Namespace

Create a namespace for testing with Istio injection enabled:
kubectl create namespace bookinfo --dry-run=client -o yaml | kubectl apply -f -
kubectl label namespace bookinfo istio-injection=enabled --overwrite
Expected:
namespace/bookinfo created (or unchanged)
namespace/bookinfo labeled

Verify the label:
kubectl get namespace bookinfo -o jsonpath='{.metadata.labels.istio-injection}'
Expected: enabled

Step 9: Deploy Test Application

Deploy a test application with Dikastes sidecar injection:
cat <<'EOF' | kubectl apply -f -
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
  namespace: bookinfo
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: bookinfo
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  namespace: bookinfo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
      annotations:
        # KEY ANNOTATION: Inject both Istio sidecar AND Dikastes sidecar
        inject.istio.io/templates: sidecar,dikastes
    spec:
      serviceAccountName: httpbin
      containers:
      - name: httpbin
        image: kennethreitz/httpbin
        ports:
        - containerPort: 80
---
# Client pod for testing (no Dikastes, just Istio proxy)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: client
  namespace: bookinfo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: client
  namespace: bookinfo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      labels:
        app: client
      annotations:
        inject.istio.io/templates: sidecar
    spec:
      serviceAccountName: client
      containers:
      - name: client
        image: curlimages/curl:latest
        command: ["/bin/sh"]
        args: ["-c", "while true; do sleep 30; done"]
EOF
Expected:
serviceaccount/httpbin created
service/httpbin created
deployment.apps/httpbin created
serviceaccount/client created
deployment.apps/client created


Step 10: Wait for Pods to be Ready

Wait for the httpbin pod to be ready (this may take 1-3 minutes):
kubectl wait --for=condition=ready pod -l app=httpbin -n bookinfo --timeout=180s
Expected: pod/httpbin-xxx condition met
Wait for the client pod:
kubectl wait --for=condition=ready pod -l app=client -n bookinfo --timeout=180s
Expected: pod/client-xxx condition met
Check all pods are running:
kubectl get pods -n bookinfo
Expected: Both httpbin and client pods showing 3/3 or 2/2 Ready

Step 11: Verify Dikastes Injection

Check that Dikastes container was injected into the httpbin pod:
kubectl get pod -l app=httpbin -n bookinfo -o jsonpath='{.items[0].spec.containers[*].name}'
Expected output: dikastes httpbin
This confirms:

✅ Dikastes container is present
✅ Application container (httpbin) is present
✅ Istio-proxy is running (as native sidecar init container)

Check the pod has 3/3 containers ready:
kubectl get pods -n bookinfo -l app=httpbin
Expected: READY column shows 3/3
The three containers are:

dikastes - Calico L7 policy enforcer (regular container)
httpbin - Application container (regular container)
istio-proxy - Envoy sidecar (native Kubernetes sidecar via init container)


Step 12: Verify Dikastes Connectivity to Felix

Check Dikastes logs for successful connection to Felix Policy Sync API:
kubectl logs -n bookinfo -l app=httpbin -c dikastes --tail=50
Expected output should include:
Successfully connected to Policy Sync server
Starting synchronization with Policy Sync server

If you see connection errors:

Verify CSI driver is running: kubectl get pods -n calico-system -l k8s-app=csi-node-driver
Verify Felix Policy Sync API is enabled: kubectl get felixconfiguration default -o yaml | grep policySyncPathPrefix


Step 13: Verify Envoy-Dikastes Communication

Check Envoy's connection to Dikastes via Unix socket:
POD_NAME=$(kubectl get pod -n bookinfo -l app=httpbin -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n bookinfo $POD_NAME -c istio-proxy -- curl -s localhost:15000/clusters | grep dikastes
Expected output should include:
dikastes::added_via_api::true
dikastes::health_flags::healthy
dikastes::cx_active::1
dikastes::cx_connect_fail::0
dikastes::rq_success::[number]

Critical indicators:

✅ cx_connect_fail::0 - No connection failures
✅ cx_active::1 - Active connection to Dikastes
✅ rq_success - Successful requests to Dikastes

If you see cx_connect_fail > 0:

This indicates Envoy cannot reach Dikastes Unix socket
Verify the istio-proxy volumeMount is configured (check Step 4 manifest)
Delete and recreate the pod: kubectl delete pod -n bookinfo -l app=httpbin


Step 14: Test L7 Authorization

Test that Dikastes is actively enforcing authorization:
kubectl exec -n bookinfo deploy/client -- curl -s -o /dev/null -w "%{http_code}\n" http://httpbin:8000/get
Expected output: 403
What this means:

✅ Request reached httpbin service
✅ Envoy intercepted the request
✅ Envoy called Dikastes for authorization
✅ Dikastes enforced default-deny policy (no allow policy configured)
✅ 403 Forbidden returned

If you see 200 instead of 403:

Dikastes may not be processing requests
Check Envoy-Dikastes connection (Step 13)
Check Dikastes logs (Step 12)

If you see connection errors:

Check service exists: kubectl get svc -n bookinfo httpbin
Check Istio injection: kubectl get pod -n bookinfo -l app=client -o jsonpath='{.spec.containers[*].name}' (should include istio-proxy)


Step 15: Enable Debug Logging (Optional)

For detailed visibility into authorization decisions, enable debug logging:
Update the Dikastes deployment to add --debug flag:
cat <<'EOF' > /tmp/istio-operator-dikastes-debug.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-with-dikastes
  namespace: istio-system
spec:
  profile: minimal
  values:
    sidecarInjectorWebhook:
      templates:
        dikastes: |
          spec:
            containers:
            - name: dikastes
              image: quay.io/calico/dikastes:v3.31.2
              args:
              - server
              - -l
              - /var/run/dikastes/dikastes.sock
              - -d
              - /var/run/felix/nodeagent/socket
              - --debug
              securityContext:
                allowPrivilegeEscalation: false
                runAsGroup: 999
                runAsNonRoot: true
                runAsUser: 999
              livenessProbe:
                exec:
                  command: ["/healthz", "liveness"]
                initialDelaySeconds: 3
                periodSeconds: 3
              readinessProbe:
                exec:
                  command: ["/healthz", "readiness"]
                initialDelaySeconds: 3
                periodSeconds: 3
              volumeMounts:
              - mountPath: /var/run/dikastes
                name: dikastes-sock
              - mountPath: /var/run/felix
                name: felix-sync
            initContainers:
            - name: istio-proxy
              volumeMounts:
              - mountPath: /var/run/dikastes
                name: dikastes-sock
            volumes:
            - name: dikastes-sock
              emptyDir:
                medium: Memory
            - name: felix-sync
              csi:
                driver: csi.tigera.io
        dikastes-gateway: |
          spec:
            containers:
            - name: dikastes
              image: quay.io/calico/dikastes:v3.31.2
              args:
              - server
              - -l
              - /var/run/dikastes/dikastes.sock
              - -d
              - /var/run/felix/nodeagent/socket
              - --debug
              securityContext:
                allowPrivilegeEscalation: false
                runAsGroup: 0
                runAsNonRoot: false
                runAsUser: 0
              livenessProbe:
                exec:
                  command: ["/healthz", "liveness"]
                initialDelaySeconds: 3
                periodSeconds: 3
              readinessProbe:
                exec:
                  command: ["/healthz", "readiness"]
                initialDelaySeconds: 3
                periodSeconds: 3
              volumeMounts:
              - mountPath: /var/run/dikastes
                name: dikastes-sock
              - mountPath: /var/run/felix
                name: felix-sync
            initContainers:
            - name: istio-proxy
              volumeMounts:
              - mountPath: /var/run/dikastes
                name: dikastes-sock
            volumes:
            - name: dikastes-sock
              emptyDir:
                medium: Memory
            - name: felix-sync
              csi:
                driver: csi.tigera.io
EOF

istioctl install -f /tmp/istio-operator-dikastes-debug.yaml -y
Delete the httpbin pod to pick up new injection template:
kubectl delete pod -n bookinfo -l app=httpbin
kubectl wait --for=condition=ready pod -l app=httpbin -n bookinfo --timeout=180s
Now check logs to see detailed authorization decisions:
kubectl logs -n bookinfo -l app=httpbin -c dikastes --tail=100 | grep -A 20 "Check start"
Expected: Detailed logs showing source identity, destination identity, HTTP method, path, and policy decision

Success Criteria Summary

Your integration is successful if all these conditions are met:

✅ CSI driver is running on all nodes
✅ Felix Policy Sync API is enabled
✅ Istio is installed with Dikastes templates loaded
✅ EnvoyFilter, ServiceEntry, and DestinationRule are deployed
✅ Test pods are running (3/3 containers ready for httpbin)
✅ Dikastes container is injected into httpbin pod
✅ Dikastes logs show "Successfully connected to Policy Sync server"
✅ Envoy shows cx_connect_fail::0 and cx_active::1 for dikastes cluster
✅ Test request returns 403 (default-deny working)


Troubleshooting

Issue: CSI driver pods not running

Check:
kubectl get pods -n calico-system -l k8s-app=csi-node-driver
kubectl describe pod -n calico-system -l k8s-app=csi-node-driver
Resolution:

Check node taints and tolerations
Verify calico-system namespace exists
Check CSI driver manifest applied correctly

Issue: Dikastes container not injected

Check namespace label:
kubectl get namespace bookinfo -o yaml | grep istio-injection
Expected: istio-injection: enabled
Check pod annotation:
kubectl get pod -n bookinfo -l app=httpbin -o yaml | grep "inject.istio.io/templates"
Expected: inject.istio.io/templates: sidecar,dikastes
Resolution:

Ensure namespace has istio-injection=enabled label
Ensure pod template has the annotation
Delete pod to re-trigger injection

Issue: Pod stuck in PodInitializing

Check events:
kubectl describe pod -n bookinfo -l app=httpbin
Common causes:

CSI driver not running
Felix Policy Sync socket not available
Image pull issues

Check Dikastes logs:
kubectl logs -n bookinfo -l app=httpbin -c dikastes
Issue: Envoy cannot connect to Dikastes (cx_connect_fail > 0)

Root cause: istio-proxy doesn't have dikastes-sock volume mounted
Resolution:

Verify IstioOperator manifest includes initContainers section (Step 4)
Re-apply the IstioOperator: istioctl install -f /tmp/istio-operator-dikastes.yaml -y
Delete the pod: kubectl delete pod -n bookinfo -l app=httpbin
Wait for new pod: kubectl wait --for=condition=ready pod -l app=httpbin -n bookinfo --timeout=180s
Verify fix: Check pod spec has volumeMount:

kubectl get pod -n bookinfo -l app=httpbin -o json | jq '.items[0].spec.initContainers[] | select(.name=="istio-proxy") | .volumeMounts[] | select(.name=="dikastes-sock")'
Issue: Test request returns 200 instead of 403

Possible causes:

Dikastes not processing requests (check Envoy connection)
An allow policy was created
EnvoyFilter not applied

Check EnvoyFilter:
kubectl get envoyfilter -n istio-system ext-authz -o yaml
Expected: Should show ext_authz configuration pointing to dikastes

Next Steps

After successful integration:

Create Calico Network Policies: Define ApplicationLayerPolicy resources (syntax depends on Calico distribution)
Test Allow Policies: Create policies that explicitly allow specific traffic
Test HTTP Method Filtering: Restrict based on GET, POST, PUT, DELETE
Test Path-Based Authorization: Allow/deny based on URL paths
Test Identity-Based Access: Use ServiceAccount identities for authorization


Architecture Reference

Communication Flow

HTTP Request → Envoy → ext_authz → Dikastes → Felix Policy Sync API → Policy Decision

Socket Paths


Envoy ↔ Dikastes: /var/run/dikastes/dikastes.sock (emptyDir volume)
Dikastes ↔ Felix: /var/run/felix/nodeagent/socket (CSI-mounted volume)

Key Components


Dikastes: Calico sidecar container that implements Envoy external authorization API
Felix Policy Sync API: Calico component that provides policy decisions
CSI Driver: Mounts Felix socket into pod containers
EnvoyFilter: Configures Envoy's ext_authz filter to call Dikastes
ServiceEntry: Defines dikastes.calico.cluster.local for Unix socket communication


Version Information

Tested with:

Calico v3.31.2
Istio v1.28.1
Kubernetes v1.29+

Compatibility:

Istio 1.18+ (IstioOperator approach)
Calico 3.28+ (CSI driver support)
Kubernetes 1.29+ (native sidecar support)


Last Updated: 2025-12-06
Guide Version: 1.0.0
Target Audience: AI Code Agents
No results found