Software Supply Chain Security: Ephemeral CI/CD Runners, Cosign & SLSA Compliance

1. Threat Modeling the Modern CI/CD Pipeline: SolarWinds, Dependency Confusion & Codecov

In modern cloud-native architectures, security perimeters have shifted from perimeter network firewalls to the software delivery pipeline. While production Kubernetes clusters enforce strict runtime policies (mTLS, RBAC, NetworkPolicies), the build-and-release infrastructure often remains an under-defended attack vector.

Attackers exploit supply chain pipelines across three discrete vulnerability surfaces:

• Source Integrity Compromise: Unauthorized commits injected via stolen developer SSH/GPG keys, branch protection bypasses, or untracked dependency confusion packages.
• Build Pipeline Tampering (SolarWinds Attack Vector): Malicious payloads inserted into build artifacts directly on persistent CI build worker nodes during binary compilation, bypassing source code review.
• Artifact Registry Injection: Compromised registry credentials overwriting immutable Docker image tags (e.g., rewriting v1.2.0 to point to a trojanized container digest).

2. The SLSA v1.0 Framework: Levels, Build Integrity & Provenance Attestations

The Supply-chain Levels for Software Artifacts (SLSA v1.0) framework, developed under OpenSSF and Google, establishes an industry-standard security specification to guarantee artifact authenticity and non-tamperability across build systems.

3. Ephemeral Isolated Runners: Kubernetes Autoscaling vs. Persistent Host Contamination

Legacy CI/CD infrastructures utilizing persistent virtual machines or static bare-metal build agents create dangerous state contamination. A single malicious dependency pulling an arbitrary post-install script (e.g., npm postinstall) can plant a persistent daemon, modify compiler toolchains, or scrape environment secrets stored on disk across subsequent builds.

Achieving SLSA Level 3 requires 100% Ephemeral Build Runners (e.g., Actions Runner Controller - ARC on Kubernetes):

• Single-Job Lifecycle: A dedicated runner Pod is spawned dynamically via Kubernetes API when a job is queued.
• Total Memory & Disk Teardown: Once the build concludes, the container runtime terminates the Pod and permanently destroys the rootfs and ephemeral volumes, leaving zero residual state for future jobs.
• Rootless Container Isolation: Builds execute without Docker-in-Docker (dind) root privileges using container builders like Kaniko or Buildah.

4. Cryptographic Artifact Signing with Sigstore Cosign: Keyless OIDC & Rekor Transparency

Traditional signing systems rely on long-lived private PGP or RSA keys stored inside CI/CD secrets (e.g., COSIGN_PRIVATE_KEY). If these credentials leak, attackers can sign arbitrary malicious binaries indefinitely.

Sigstore Keyless Signing eliminates long-lived secrets using OpenID Connect (OIDC) and short-lived certificates:

1. OIDC Identity Token: The CI/CD runner requests an OIDC identity token from the cloud host (e.g., GitHub Actions OIDC provider) certifying the exact repository, workflow path, commit SHA, and triggering actor.
2. Fulcio Certificate Authority: Cosign submits the OIDC token to Fulcio (an automated Certificate Authority), which issues an X.509 code-signing certificate valid for only 10 minutes.
3. Artifact Signing & Rekor Ledger: Cosign signs the container image digest and records the cryptographic signature, certificate, and hash directly into Rekor—an immutable, append-only cryptographic transparency log.
4. Zero-Secret Key Verification: Admission controllers in production Kubernetes (Kyverno or OPA Gatekeeper) verify the signature against Rekor transparency records without requiring any pre-shared secret keys.

5. Production Incident Postmortem: Stolen Build Credentials & Unauthorized Image Tag Mutability

Incident Summary

An enterprise SaaS platform discovered unauthorized cryptomining containers running in their production Kubernetes cluster. The container image was tagged auth-gateway:v2.1.4 and pulled from their private registry, matching an official deployment manifest.

Root-Cause Investigation Sequence

1. Credential Exfiltration: A developer's compromised laptop leaked personal container registry credentials with write permissions.
2. Tag Overwrite (Immutability Failure): The attacker built a trojanized container image locally and pushed it directly to the registry, overwriting the existing auth-gateway:v2.1.4 tag.
3. Absence of Signature Verification: Kubernetes pulled the image by tag rather than immutable SHA-256 digest, and the cluster lacked an admission controller enforcing Sigstore Cosign signature verification.
4. Remediation: Enforced registry tag immutability, mandated deployment strictly via image digests (image@sha256:...), and deployed Kyverno admission controllers to reject any unsigned container payloads.

6. Production GitHub Actions Workflow & In-Toto Attestation Blueprint

Deploy the following hardened workflow to automate SLSA Level 3 keyless signing and in-toto provenance generation:

name: Secure Build & Keyless Signing Pipeline

on:
  push:
    tags: [ 'v*.*.*' ]

permissions:
  contents: read
  packages: write
  id-token: write # MANDATORY for Sigstore Keyless OIDC Token Generation

jobs:
  build-and-sign:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Source Code
        uses: actions/checkout@v4

      - name: Install Sigstore Cosign
        uses: sigstore/cosign-installer@v3.5.0

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and Push Immutable Container Image
        id: build-push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.ref_name }}
          outputs: type=image,name=target,annotation-index.org.opencontainers.image.description=Production Release

      - name: Sign Container Image Digest (Keyless)
        env:
          IMAGE_DIGEST: ${{ steps.build-push.outputs.digest }}
        run: |
          cosign sign --yes \
            "ghcr.io/${{ github.repository }}@${IMAGE_DIGEST}"

      - name: Generate and Sign SLSA v1.0 In-Toto Provenance Attestation
        env:
          IMAGE_DIGEST: ${{ steps.build-push.outputs.digest }}
        run: |
          cosign attest --yes \
            --type slsaprovenance1 \
            --predicate <(echo '{"builder":{"id":"https://github.com/actions/runner"}}') \
            "ghcr.io/${{ github.repository }}@${IMAGE_DIGEST}"

7. Failure Mode and Effects Analysis (FMEA Matrix) for Supply Chain Pipelines

8. Engineering FAQ & OpenSSF / CNCF Supply Chain Standards Reference

Q1: What is the exact difference between an Artifact Signature and an In-Toto Attestation?

Specification: in-toto Attestation Framework & Sigstore Specification.
Technical Explanation: An Artifact Signature (e.g., cosign sign) simply asserts: "The entity identified by this certificate verified the SHA-256 digest of this binary at this timestamp." An In-Toto Attestation (e.g., cosign attest) is a cryptographically signed metadata statement containing a structured predicate. It asserts deep provenance details: the exact source repository commit SHA, the compiler flags, the builder URI, the environment variables, and the Software Bill of Materials (SBOM), proving not just who signed it, but exactly how and where the software was constructed.

Q2: Why is Rootless Container building critical inside CI/CD runners?

Technical Explanation: Standard Docker build commands require mounting the host Docker daemon socket (/var/run/docker.sock) inside the runner. Any build process that executes arbitrary code (such as third-party build plugins) with root access to docker.sock can escape the container sandbox, execute code on the host node, and compromise all other workloads sharing the physical server. Rootless builders (such as Kaniko, Buildah, or Makisu) compile OCI images entirely in user space without host daemon privileges.