Istio mTLS with SPIRE identities

7 min read · Advanced

This guide shows how to use Edera’s SPIRE-based workload identity feature Istio’s SPIRE support in sidecar mode, using the SPIFFE identities that SPIRE issues to each pod for pod-to-pod mTLS and Istio AuthorizationPolicy configuration.

⚠️

SPIRE-based workload identity in Edera is still a preview.

How it works

Istio’s sidecar proxy can take its workload certificate from a SPIRE agent instead of from the Istio CA. It looks for the agent’s Workload API socket at a fixed path, /var/run/secrets/workload-spiffe-uds/.

On a node with the upstream SPIRE agent, the SPIFFE CSI driver puts the node agent’s socket at that path. An Edera pod runs in its own zone, and the zone has its own SPIRE agent. So Edera pods use a separate Istio injection template that links the zone agent’s socket to the path the proxy reads.

Both templates produce the same identity for the same service account, so Istio sees one trust domain and one set of principals.

Prerequisites

SPIRE set up for Edera pods. Complete steps 1 to 5 of SPIRE workload identity on Kubernetes.
If the mesh has non-Edera pods, keep the spire chart’s spire-agent and spiffe-csi-driver enabled. Both are on by default.
Istio is not installed yet, or you can change the istiod Helm values.
kubectl, helm, jq, and openssl on your workstation.

This guide uses the variables from the SPIRE guide, TRUST_DOMAIN and NAMESPACE.

This guide was tested with these versions:

Component	Version
Istio `base` and `istiod` charts	1.30.5
`spire` chart	0.30.2
Edera attestor plugins	`v0.0.13`

Step 1: Install Istio with SPIRE injection templates

The zone’s SPIRE agent shares the pod’s network. Istio’s traffic capture would send the agent’s own connections to the SPIRE server into the proxy, and the proxy cannot start until that agent gives it a certificate. Exclude the SPIRE server’s address from capture:

export SPIRE_SERVER_IP=$(kubectl -n "$NAMESPACE" get service spire-server -o jsonpath='{.spec.clusterIP}')

Write the istiod values. Set the mesh trust domain to the SPIRE trust domain, and add two injection templates:

spire, the upstream template. The proxy reads the node agent’s socket through the SPIFFE CSI driver.
spire-edera, for Edera pods. An init container waits for the zone agent’s socket, then links it into the directory the proxy already mounts.

cat > istiod-values.yaml <<EOF
meshConfig:
  trustDomain: ${TRUST_DOMAIN}
global:
  proxy:
    excludeIPRanges: ${SPIRE_SERVER_IP}/32
sidecarInjectorWebhook:
  templates:
    spire: |
      spec:
        containers:
        - name: istio-proxy
          env:
          - name: WORKLOAD_IDENTITY_SOCKET_FILE
            value: spire-agent.sock
        volumes:
        - name: workload-socket
          csi:
            driver: csi.spiffe.io
            readOnly: true
    spire-edera: |
      spec:
        initContainers:
        - name: edera-spire-socket
          image: busybox:1.37.0
          command:
          - sh
          - -c
          - until [ -S /shared/zone/sockets/agent.sock ]; do sleep 1; done; ln -sf /shared/zone/sockets/agent.sock /var/run/secrets/workload-spiffe-uds/spire-agent.sock
          volumeMounts:
          - name: workload-socket
            mountPath: /var/run/secrets/workload-spiffe-uds
        containers:
        - name: istio-proxy
          env:
          - name: WORKLOAD_IDENTITY_SOCKET_FILE
            value: spire-agent.sock
EOF

Both templates name the socket spire-agent.sock. With a name other than Istio’s default, a proxy that cannot reach SPIRE stops with an error. With the default name, it falls back to the Istio CA without telling you.

Install Istio:

helm upgrade --install istio-base base \
  --repo https://istio-release.storage.googleapis.com/charts --version 1.30.5 \
  --namespace istio-system --create-namespace --wait
helm upgrade --install istiod istiod \
  --repo https://istio-release.storage.googleapis.com/charts --version 1.30.5 \
  --namespace istio-system --values istiod-values.yaml --wait

Step 2: Deploy the workloads

Create a namespace with sidecar injection:

kubectl create namespace mesh-demo
kubectl label namespace mesh-demo istio-injection=enabled

Every meshed pod needs these annotations:

Annotation	Pods	Why
`sidecar.istio.io/nativeSidecar: "false"`	All	The templates patch `istio-proxy` as a regular container, and a native sidecar would start before `spire-edera` links the socket.
`inject.istio.io/templates: sidecar,spire-edera`	Edera	The proxy reads the zone agent’s socket.
`dev.edera/spire-access: "true"`	Edera	Puts a SPIRE agent in the zone and gives the containers its socket.
`inject.istio.io/templates: sidecar,spire`	Not Edera	The proxy reads the node agent’s socket.

Deploy an httpbin server as an Edera pod:

kubectl apply -n mesh-demo -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
      annotations:
        sidecar.istio.io/nativeSidecar: "false"
        inject.istio.io/templates: sidecar,spire-edera
        dev.edera/spire-access: "true"
    spec:
      runtimeClassName: edera
      serviceAccountName: httpbin
      containers:
      - name: main
        image: mccutchen/go-httpbin:v2.15.0
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
spec:
  selector:
    app: httpbin
  ports:
  - name: http
    port: 8000
    targetPort: 8080
EOF

Deploy a client as a non-Edera pod:

kubectl apply -n mesh-demo -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: client
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: client
spec:
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      labels:
        app: client
      annotations:
        sidecar.istio.io/nativeSidecar: "false"
        inject.istio.io/templates: sidecar,spire
    spec:
      serviceAccountName: client
      containers:
      - name: main
        image: curlimages/curl:8.16.0
        args: ["sleep", "infinity"]
EOF
kubectl -n mesh-demo rollout status deployment/httpbin deployment/client

Step 3: Require mTLS and authorize callers

Require mTLS in the namespace, and allow only the client service account to call it:

kubectl apply -n mesh-demo -f - <<EOF
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: STRICT
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: clients-only
spec:
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["${TRUST_DOMAIN}/ns/mesh-demo/sa/client"]
EOF

Istio matches principals against the SPIFFE ID in the peer’s certificate, which SPIRE issued.

Verification

Check that the Edera pod’s proxy serves an SVID for its own service account, signed by SPIRE and not by the Istio CA:

POD=$(kubectl -n mesh-demo get pod -l app=httpbin -o jsonpath='{.items[0].metadata.name}')
kubectl -n mesh-demo exec "$POD" -c istio-proxy -- pilot-agent request GET config_dump 2>/dev/null \
  | jq -r '.configs[] | select(."@type" | test("SecretsConfigDump"))
      | .dynamic_active_secrets[] | select(.name == "default")
      | .secret.tls_certificate.certificate_chain.inline_bytes' \
  | base64 -d > svid.pem
openssl x509 -in svid.pem -noout -ext subjectAltName
# URI:spiffe://example.org/ns/mesh-demo/sa/httpbin

kubectl -n "$NAMESPACE" exec spire-server-0 -c spire-server -- \
  /opt/spire/bin/spire-server bundle show > spire-bundle.pem
openssl verify -CAfile spire-bundle.pem -untrusted svid.pem svid.pem
# svid.pem: OK

kubectl -n mesh-demo get configmap istio-ca-root-cert -o jsonpath='{.data.root-cert\.pem}' > istio-root.pem
openssl verify -CAfile istio-root.pem -untrusted svid.pem svid.pem
# error ... unable to get local issuer certificate

Run the same check on the client pod. Its SVID also chains to the SPIRE bundle.

Call the Edera server from the non-Edera client. The server sees the client’s SPIFFE ID in the client certificate:

kubectl -n mesh-demo exec deploy/client -c main -- curl -s http://httpbin:8000/headers \
  | jq -r '.headers["X-Forwarded-Client-Cert"][0]'
# By=spiffe://example.org/ns/mesh-demo/sa/httpbin;Hash=...;URI=spiffe://example.org/ns/mesh-demo/sa/client

A caller with another service account is refused by Istio:

kubectl -n mesh-demo run stranger --image=curlimages/curl:8.16.0 \
  --annotations=sidecar.istio.io/nativeSidecar=false \
  --annotations=inject.istio.io/templates=sidecar,spire \
  --command -- sleep infinity
kubectl -n mesh-demo wait pod/stranger --for=condition=Ready
kubectl -n mesh-demo exec stranger -c stranger -- curl -s -w '\n%{http_code}\n' http://httpbin:8000/headers
# RBAC: access denied
# 403

Limitations

Sidecar mode only. Istio ambient mode does not work with Edera pods. Its node proxy, ztunnel, gets identities from the node’s SPIRE agent by process ID, and a process in a zone has no host process ID. Linkerd uses its own CA for pods in the cluster, and Cilium mutual authentication holds identities in the node’s Cilium agent, so neither uses the identities that Edera issues.
No native sidecars. Set sidecar.istio.io/nativeSidecar: "false" on every pod that uses these templates.
Egress policy. The zone agent connects to the SPIRE server from the pod’s address. A network policy on an Edera pod must allow egress to the SPIRE server.
Proxy restarts. If istio-proxy restarts while the zone agent is down, the proxy deletes the socket link and the pod’s init containers do not run again. Delete the pod to recover.

Troubleshooting

Symptom	Cause
The proxy logs `agent configured for non-default SDS socket path ... but no socket found`	The socket link is missing. The Edera pod lacks `dev.edera/spire-access: "true"` or `inject.istio.io/templates: sidecar,spire-edera`, or the proxy restarted. Delete the pod.
The proxy logs `workload is not authorized for the requested identities ["default"]`	The zone agent has no entry for the pod. Check that `excludeIPRanges` has the SPIRE server’s current ClusterIP, and that the pod has an entry under its zone agent, as in Verification.
Pod creation fails with `Duplicate value: "istio-proxy"`	The pod uses a native sidecar. Set `sidecar.istio.io/nativeSidecar: "false"`.
The SVID chains to `istio-root.pem`	The proxy uses the Istio CA. The pod lacks the template annotation, or the template does not set `WORKLOAD_IDENTITY_SOCKET_FILE`.
Calls return `RBAC: access denied` for an allowed client	The principal does not match. Check that `meshConfig.trustDomain` equals the SPIRE trust domain.