Tekton – Supply Chain Security

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Authentication for Chains

Authentication must be set up to take advantage of the following features in Chains:

Pushing signatures to an OCI registry after signing an image
Using Fulcio to get Signing Certificates when utilizing Keyless signing.

This doc will cover how to set this up!

Authenticating to an OCI Registry

To push to an OCI registry, the Chains controller will look for credentials in two places. The first place is in the pod executing your Task and the second place is in the service account configured to run your Task.

First we’ll cover creating the credentials.

Set the namespace and name of the Kubernetes service account:

export NAMESPACE=<your namespace>
export SERVICE_ACCOUNT_NAME=<service account name>

Create a Secret based on existing credentials

If you already ran docker login, you can copy the credentials stored in config.json into Kubernetes.

Note: Make sure that any external credentials store, such as the native keychain of the operating system, is not used to store the credentials and the config.json is of the format:
{
  "auths": {
    "<registry>": {
      "auth": "redacted"
    }
  }
}

Create a secret with config.json:

kubectl create secret generic docker-registry \
    --from-file=.dockerconfigjson=<path/to/.docker/config.json> \
    --type=kubernetes.io/dockerconfigjson \
    -n $NAMESPACE

More details around creating this secret can be found here.

Create a Secret by providing credentials on the command line

First, you will need access to credentials for your registry (they are in a file called credentials.json in this example). Next, create a Docker config type Kubernetes secret, which will contain the credentials required to push signatures:

Then, create a .dockerconfig type secret:

kubectl create secret docker-registry registry-credentials \
  --docker-server=gcr.io \
  --docker-username=_json_key \
  --docker-email=someemail@something.com \
  --docker-password="$(cat credentials.json)" \
  -n $NAMESPACE

More details around creating this secret can be found here.

Setup credentials using the pod

Tekton supports specifying a Pod template to customize the Pod running your Task. You must supply the Pod template when starting your Task with the cli or embeddng it into your TaskRun.

An example TaskRun configured with the registry-credentials secret.

---
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: mytaskrun
  namespace: default
spec:
  taskRef:
    name: mytask
  podTemplate:
    imagePullSecrets:
    - name: registry-credentials

More details on how to modify the podTemplate for a taskRun can be found here.

Setup credentials using the service account

Finally, give the service account access to the secret above:

kubectl patch serviceaccount $SERVICE_ACCOUNT_NAME \
  -p "{\"secrets\": [{\"name\": \"registry-credentials\"}]}" -n $NAMESPACE

Now, Chains has push permissions for any TaskRuns running under the service account $SERVICE_ACCOUNT_NAME.

The secrets in the imagePullSecrets attribute of the ServiceAccount are also taken into account. However, other Tekton components may not do so. The secrets attribute is the recommended approach.

Authenticating to Fulcio for Keyless signing

The default deployment will work against public Fulcio assuming it is installed into an EKS or GKE cluster. You will just need to add the following to chains-config ConfigMap data section in the tekton-chains namespace:

  "signers.x509.fulcio.enabled": "true"

Specifying a custom Fulcio endpoint

If you are running your own instance of Fulcio, you need to further configure Fulcio for this. You need to additionally point Chains to your fulcio instance by adding this to chains-config. In this case, it’s a local k8s service, but you will need to change the URL to point to your Fulcio instance.

  "signers.x509.fulcio.address": "http://fulcio.fulcio-system.svc"

Specifying Spiffe as authentication provider

If you are using Spiffe to authenticate to Fulcio, you will need to configure your Chains Deployment to fetch the SVID from the Spire agent. This requires mounting the Agent socket, specifying an environmental variable (if not using the default of /tmp/spire-agent/public/api.sock).

For VolumeMount, replace the k8s SA token, or add if you use it for something else the following to tekton-chains-controller container volumeMounts section:

        - name: spiffe-workload-api
          mountPath: /run/spire/sockets/agent.sock
          readOnly: true

Specify (if necessary) the non-default Agent socket, by adding the following to the tekton-chains-controller env section:

        - name: SPIFFE_ENDPOINT_SOCKET
          value: "/run/spire/sockets/agent.sock"

And finally, adding the volume for the Spiffe workload API by adding this to deployment volumes section:

      - name: spiffe-workload-api
        hostPath:
          path: /run/spire/sockets/agent.sock

Last but not least, thanks to spiffe-csi, which is a CSI (Container Storage Interface) driver for Kubernetes that facilitates injection of the SPIFFE Workload API , there is alternative way of retrieving the Agent socket to your Pods without having to mount the hostPath. You can read more about it here. Once you have installed spiffe-csi-driver into your cluster by following the installation steps in the GitHub repository, the only thing that you have to do is add the following code snippet to deployment volumes and volumeMounts sections:

     ...
        volumeMounts:
          - name: spiffe-workload-api
            mountPath: /spiffe-workload-api
            readOnly: true

     ...
      volumes:
        - name: spiffe-workload-api
          csi:
            driver: "csi.spiffe.io"
            readOnly: true

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Chains Configuration

Chains works by observing TaskRun and PipelineRun executions, capturing relevant information, and storing it in a cryptographically-signed format.

TaskRuns and PipelineRuns can indicate inputs and outputs which are then captured and surfaced in the Chains payload formats, where relevant. Chains uses the Results to hint at the correct inputs and outputs. Check out slsa-provenance.md for more details.

Chains Configuration

Chains uses a ConfigMap called chains-config in the tekton-chains namespace for configuration. Supported keys include:

TaskRun Configuration

Key	Description	Supported Values	Default
`artifacts.taskrun.format`	The format to store `TaskRun` payloads in.	`in-toto`, `slsa/v1`, `slsa/v2alpha3`, `slsa/v2alpha4`	`in-toto`
`artifacts.taskrun.storage`	The storage backend to store `TaskRun` signatures in. Multiple backends can be specified with comma-separated list (“tekton,oci”). To disable the `TaskRun` artifact input an empty string ("").	`tekton`, `oci`, `gcs`, `docdb`, `grafeas`, `archivista`	`tekton`
`artifacts.taskrun.signer`	The signature backend to sign `TaskRun` payloads with. Use `none` to disable signing while still storing provenance.	`x509`, `kms`, `none`	`x509`

NOTE:

slsa/v1 is an alias of in-toto for backwards compatibility.

slsa/v2alpha3 corresponds to the slsav1.0 spec. and uses latest v1 Tekton Objects. Recommended format for new chains users who want the slsav1.0 spec.

slsa/v2alpha4 corresponds to the slsav1.0 spec. and uses latest v1 Tekton Objects. It reads type-hinted results from StepActions. Recommended format for new chains users who want the slsav1.0 spec.

PipelineRun Configuration

Key	Description	Supported Values	Default
`artifacts.pipelinerun.format`	The format to store `PipelineRun` payloads in.	`in-toto`, `slsa/v1`, `slsa/v2alpha3`, `slsa/v2alpha4`	`in-toto`
`artifacts.pipelinerun.storage`	The storage backend to store `PipelineRun` signatures in. Multiple backends can be specified with comma-separated list (“tekton,oci”). To disable the `PipelineRun` artifact input an empty string ("").	`tekton`, `oci`, `gcs`, `docdb`, `grafeas`, `archivista`	`tekton`
`artifacts.pipelinerun.signer`	The signature backend to sign `PipelineRun` payloads with. Use `none` to disable signing while still storing provenance.	`x509`, `kms`, `none`	`x509`
`artifacts.pipelinerun.enable-deep-inspection`	This boolean option will configure whether Chains should inspect child taskruns in order to capture inputs/outputs within a pipelinerun. `"false"` means that Chains only checks pipeline level results, whereas `"true"` means Chains inspects both pipeline level and task level results.	`"true"`, `"false"`	`"false"`

NOTE:

For grafeas storage backend, currently we only support Container Analysis. We will make grafeas server address configurabe within a short time.

slsa/v1 is an alias of in-toto for backwards compatibility.

slsa/v2alpha3 corresponds to the slsav1.0 spec. and uses latest v1 Tekton Objects. Recommended format for new chains users who want the slsav1.0 spec.

slsa/v2alpha4 corresponds to the slsav1.0 spec. and uses latest v1 Tekton Objects. It reads type-hinted results from StepActions when artifacts.pipelinerun.enable-deep-inspection is set to true. Recommended format for new chains users who want the slsav1.0 spec.

OCI Configuration

Key	Description	Supported Values	Default
`artifacts.oci.format`	The format to store `OCI` payloads in.	`simplesigning`	`simplesigning`
`artifacts.oci.storage`	The storage backend to store `OCI` signatures in. Multiple backends can be specified with comma-separated list (“oci,tekton”). To disable the `OCI` artifact input an empty string ("").	`tekton`, `oci`, `gcs`, `docdb`, `grafeas`	`oci`
`artifacts.oci.signer`	The signature backend to sign `OCI` payloads with. Use `none` to skip signing of OCI artifacts while still allowing provenance generation and attestation signing (see note below).	`x509`, `kms`, `none`	`x509`

Note: When artifacts.oci.signer is set to none, only OCI image signing is disabled; attestations are still generated and pushed as configured. To push attestations to registries, set artifacts.taskrun.storage and/or artifacts.pipelinerun.storage to include oci. Attestations will still be pushed to the same location determined by type hinting (IMAGE_URL/IMAGE_DIGEST results) or storage.oci.repository if configured.

KMS Configuration

Key	Description	Supported Values	Default
`signers.kms.kmsref`	The URI reference to a KMS service to use in `KMS` signers.	Supported schemes: `gcpkms://`, `awskms://`, `azurekms://`, `hashivault://`. See https://docs.sigstore.dev/cosign/kms_support for more details.

Storage Configuration

Key	Description	Supported Values	Default
`storage.gcs.bucket`	The GCS bucket for storage
`storage.oci.repository`	The OCI repo to store OCI signatures and attestation in	If left undefined and one of `artifacts.{oci,taskrun}.storage` includes `oci` storage, attestations will be stored alongside the stored OCI artifact itself. (example on GCP) Defining this value results in the OCI bundle stored in the designated location instead of alongside the image. See cosign documentation for additional information.
`storage.oci.repository.insecure`	Whether to use insecure connection when connecting to the OCI repository	`true`, `false`	`false`
`storage.docdb.url`	The go-cloud URI reference to a docstore collection	`firestore://projects/[PROJECT]/databases/(default)/documents/[COLLECTION]?name_field=name`
`storage.docdb.mongo-server-url` (optional)	The value of MONGO_SERVER_URL env var with the MongoDB connection URI	Example: `mongodb://[USER]:[PASSWORD]@[HOST]:[PORT]/[DATABASE]`
`storage.docdb.mongo-server-url-dir` (optional)	The path of the directory that contains the file named MONGO_SERVER_URL that stores the value of MONGO_SERVER_URL env var	If the file `/mnt/mongo-creds-secret/MONGO_SERVER_URL` has the value of MONGO_SERVER_URL, then set `storage.docdb.mongo-server-url-dir: /mnt/mongo-creds-secret`
`storage.docdb.mongo-server-url-path` (optional)	The path of the file that contains the value of mongo server url	If the file `/mnt/mongo-creds-secret/mongo-server-url` has the value, then set `storage.docdb.mongo-server-url-path: /mnt/mongo-creds-secret/mongo-server-url`
`storage.grafeas.projectid`	The project of where grafeas server is located for storing occurrences
`storage.grafeas.noteid` (optional)	This field will be used as the prefix part of the note name that will be created. The value of this field must be a string without spaces. (See more details below.)
`storage.grafeas.notehint` (optional)	This field is used to set the human_readable_name field in the Grafeas ATTESTATION note. If it is not provided, the default `This attestation note was generated by Tekton Chains` will be used.
`storage.archivista.url`	The URL endpoint for the Archivista service.	A valid HTTPS URL pointing to your Archivista instance (e.g. `https://archivista.testifysec.io`).	None

[!WARNING] Security Considerations for storage.oci.repository.insecure

The storage.oci.repository.insecure flag allows connecting to OCI registries without TLS certificate verification. This feature is designed to ease developer overhead during testing and development where setting up HTTPS might be cumbersome.

Security Risks:

Production Environment Risk: Enabling this flag in production environments can lead to serious security compromises. Administrators must ensure this flag is only enabled for development and testing purposes.

Man-in-the-Middle Attacks: Skipping TLS certificate verification makes the connection vulnerable to man-in-the-middle attacks where provenance could be tampered with.

SLSA Guarantees Violation: Tampered provenance can lead to violation of SLSA (Supply chain Levels for Software Artifacts) guarantees that Tekton Chains promises to provide.

Recommendation: Only use storage.oci.repository.insecure: true in development or test environments. For production deployments, always use secure HTTPS connections with valid TLS certificates (storage.oci.repository.insecure: false, which is the default).

docstore

You can read about the go-cloud docstore URI format here. Tekton Chains supports the following docstore services:

firestore
dynamodb
mongo

MongoDB

You can provide MongoDB connection through different options

Using MONGO_SERVER_URL Environment Variable
- User can set the MongoDB connection URL in the MONGO_SERVER_URL env var in the Chains deployment
Using storage.docdb.mongo-server-url field in the chains-config configmap
- Alternatively, you can set the connection URL using the storage.docdb.mongo-server-url field in the chains-config configmap
- This field overrides the MONGO_SERVER_URL env var
Using storage.docdb.mongo-server-url-dir field
- Another option is to set storage.docdb.mongo-server-url-dir, which points to a directory containing a file named MONGO_SERVER_URL
- The directory path setting takes precedence over both storage.docdb.mongo-server-url and the MONGO_SERVER_URL env var
- For instance, if /mnt/mongo-creds-secret/MONGO_SERVER_URL contains the MongoDB URL, set storage.docdb.mongo-server-url-dir: /mnt/mongo-creds-secret
Using storage.docdb.mongo-server-url-path field
- You can use storage.docdb.mongo-server-url-path field in chains-config configmap to directly reference the file containing the MongoDB URL
- This field overrides all others (mongo-server-url-dir, mongo-server-url, and MONGO_SERVER_URL env var)
- For instance, if /mnt/mongo-creds-secret/mongo-server-url contains the MongoDB URL, then set storage.docdb.mongo-server-url-path: /mnt/mongo-creds-secret/mongo-server-url

NOTE :-

When using storage.docdb.mongo-server-url-dir or storage.docdb.mongo-server-url-path field, store the value of mongo server url in a secret and mount the secret. When the secret is updated, the new value will be fetched by Tekton Chains controller
Also using storage.docdb.mongo-server-url-dir or storage.docdb.mongo-server-url-path field are recommended, using storage.docdb.mongo-server-url should be avoided since credentials are stored in a ConfigMap instead of a secret

Grafeas

You can read more about Grafeas notes and occurrences here. To create occurrences, we have to create notes first that are used to link occurrences. Two types of occurrences will be created: ATTESTATION Occurrence and BUILD Occrrence. The configurable noteid is used as the prefix of the note name. Under the hood, the suffix -simplesigning will be appended for the ATTESTATION note, and the suffix -intoto will be appended for the BUILD note. If the noteid field is not configured, tekton-<NAMESPACE> will be used as the prefix.

In-toto Configuration

Key	Description	Supported Values	Default
`builder.id`	The builder ID to set for in-toto attestations		`https://tekton.dev/chains/v2`
`builddefinition.buildtype`	The buildType for in-toto attestations	`https://tekton.dev/chains/v2/slsa`, `https://tekton.dev/chains/v2/slsa-tekton`	`https://tekton.dev/chains/v2/slsa`

NOTE: Considerations for the builddefinition.buildtype parameter:

It is only valid for slsa/v2alpha3 configurations (see TaskRun or PipelineRun configuration).

The parameter can take one of two values:

https://tekton.dev/chains/v2/slsa: This buildType strictly conforms to the slsav1.0 spec.

https://tekton.dev/chains/v2/slsa-tekton: This buildType also conforms to the slsav1.0 spec, but adds additional information specific to Tekton. This information includes the PipelinRun/TaskRun labels and annotations as internalParameters. It also includes capturing each pipeline task in a PipelinRun under resolvedDependencies.

Sigstore Features Configuration

Transparency Log

Key	Description	Supported Values	Default
`transparency.enabled`	Whether to enable automatic binary transparency uploads.	`true`, `false`, `manual`	`false`
`transparency.url`	The URL to upload binary transparency attestations to, if enabled.		`https://rekor.sigstore.dev`

Note: If transparency.enabled is set to manual, then only TaskRuns and PipelineRuns with the following annotation will be uploaded to the transparency log:

chains.tekton.dev/transparency-upload: "true"

Keyless Signing with Fulcio

Key	Description	Supported Values	Default
`signers.x509.fulcio.enabled`	Whether to enable automatic certificates from fulcio.	`true`, `false`	`false`
`signers.x509.fulcio.address`	Fulcio address to request certificate from, if enabled		`https://fulcio.sigstore.dev`
`signers.x509.fulcio.issuer`	Expected OIDC issuer.		`https://oauth2.sigstore.dev/auth`
`signers.x509.fulcio.provider`	Provider to request ID Token from	`google`, `spiffe`, `github`, `filesystem`	Unset, each provider will be attempted.
`signers.x509.identity.token.file`	Path to file containing ID Token.
`signers.x509.tuf.mirror.url`	TUF server URL. $TUF_URL/root.json is expected to be present.		`https://sigstore-tuf-root.storage.googleapis.com`

KMS OIDC and Spire Configuration

Key	Description	Supported Values	Default
`signers.kms.auth.address`	URI of KMS server (e.g. the value of `VAULT_ADDR`)
`signers.kms.auth.token`	Auth token KMS server (e.g. the value of `VAULT_TOKEN`)
`signers.kms.auth.token-path`	Path to store KMS server Auth token (e.g. `/etc/kms-secrets`)
`signers.kms.auth.oidc.path`	Path used for OIDC authentication (e.g. `jwt` for Vault)
`signers.kms.auth.oidc.role`	Role used for OIDC authentication
`signers.kms.auth.spire.sock`	URI of the Spire socket used for KMS token (e.g. `unix:///tmp/spire-agent/public/api.sock`)
`signers.kms.auth.spire.audience`	Audience for requesting a SVID from Spire

NOTE:

If signers.kms.auth.token-path is set, create a secret and ensure the Chains deployment mounts this secret to the path specified by signers.kms.auth.token-path.

[!IMPORTANT] To project the latest token values without needing to recreate the pod, avoid using subPath in volume mount.

Visual Guide: ConfigMap Configuration Options

Refer the diagram below to explore the pictorial representation of signing and storage configuration options, and their usage in the context of chains artifacts.

Namespaces Restrictions in Chains Controller

This feature allows you to specify a list of namespaces for the controller to monitor, providing granular control over its operation. If no namespaces are specified, the controller defaults to monitoring all namespaces.

Usage

To restrict the Chains Controller to specific namespaces, pass a comma-separated list of namespaces as an argument to the controller using the –namespace flag.

Example

To restrict the controller to the dev and test namespaces, you would start the controller with the following argument:

--namespace=dev,test

In this example, the controller will only monitor resources (pipelinesruns and taskruns) within the dev and test namespaces.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Signing Artifacts

Signing Secrets

To get started signing things in Chains, you will need to generate a keypair and instruct Chains to sign with it via a Kubernetes secret. Chains expects a private key, and password if the key is encrypted, to exist in a Kubernetes secret signing-secrets in the tekton-chains namespace.

Chains supports a few different signature schemes, including x509 and KMS systems.

This doc explains how to generate keys and configure Chains for each type. Note, only one of the following keys needs to be set up for Chains to work:

x509

For x509, Chains expects the private key to be stored in a secret called signing-secrets with the following structure:

x509.pem (the private key)

Chains also has the following requirements:

The private key to be stored as an unencrypted PKCS8 PEM file (BEGIN PRIVATE KEY)
The key is of type ed25519 or ecdsa

Cosign

For cosign, Chains expects the encrypted private key to be stored in a secret called signing-secrets with the following structure:

cosign.key (the cosign-generated private key)
cosign.password (the password to decrypt the private key)

Chains also has the following requirements:

The private key must be stored as an encrypted PEM file of type ENCRYPTED COSIGN PRIVATE KEY

Generate cosign Keypair

To create a cosign keypair, cosign.key and cosign.pub, install cosign and run the following:

cosign generate-key-pair k8s://tekton-chains/signing-secrets

Cosign will prompt you for a password, and create the Kubernetes secret for you.

KMS

Chains uses a “go-cloud” URI like scheme for KMS references. Chains supports GCP KMS and Hashicorp Vault today, but we would love to add support for more.

You can configure Chains to use a specific KMS key using the signers.kms.kmsref config key in chains-config.

For GCP, this should have the structure of gcpkms://projects/<project>/locations/<location>/keyRings/<keyring>/cryptoKeys/<key> where location, keyring, and key are filled in appropriately.

For Vault, this should have the structure of hashivault://<keyname>, where the keyname is filled out appropriately.

For AWS, this should have the structure of awskms://[ENDPOINT]/[ID/ALIAS/ARN] (endpoint optional).

For Azure, this should have the structure of azurekms://[VAULT_NAME][VAULT_URL]/[KEY_NAME].

Authentication

Most likely, you will need to set up some additional authentication so that the chains-controller deployment has access to your KMS service for signing.

For Vault, if you use Token-based authentication, store the token as a secret. Mount this secret to a specific path within the tekton-chains-controller container. Specify the mounted path as the value for the chains-config config map key signers.kms.auth.token-path. This approach can also be applied to other KMS providers that support token-based authentication. Note that the existing configuration option signers.kms.auth.token will still work. If both values are set, signers.kms.auth.token-path will take precedence.

For GCP/GKE, we suggest enabling Workload Identity, and giving your service account Cloud KMS Admin permissions. Other Service Account techniques would work as well.

Troubleshooting

If your signing secrets is already populated, you may get the following error:

Error from server (AlreadyExists): secrets "signing-secrets" already exists

Simply delete the secret and then recreate as described above:

kubectl delete secret signing-secrets -n tekton-chains

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

SLSA Provenance

Goal

This doc includes instructions for how to configure a Tekton Pipeline/Task so that Tekton Chains can generate SLSA provenances properly.

Glossary

SLSA: SLSA stands for Supply-chain Levels for Software Artifacts, or SLSA (“salsa”). It’s a security framework, a checklist of standards and controls to prevent tampering, improve integrity, and secure packages and infrastructure. It’s how you get from “safe enough” to being as resilient as possible, at any link in the chain. (source)
Attestation (in-toto attestation): An in-toto attestation is authenticated metadata about one or more software artifacts. The intended consumers are automated policy engines, such as in-toto-verify and Binary Authorization. There are a variety of attestations, and the type of attestation is determined by the predicate.
SLSA Provenance: SLSA Provenance is an attestation that a build platform generated to describe how an artifact or set of artifacts was produced.
Pipeline-level provenance: Provenance that Tekton Chains generates to cover the whole picture of the PipelineRun execution.
Task-level provenance: Provenance that Tekton Chains generates to only include the details of a particular TaskRun execution. It’s particularly needed for a standalone TaskRun that is not spawned by a PipelineRun. By contrast, if it’s a child TaskRun of a PipelineRun, Task-level provenance will miss the details of other TaskRuns within that Pipeline.
Input Artifacts: A canonical term used in this doc to refer to the artifacts that influenced the build process such as source code repository, dependencies and so on. It’s mapped to resolvedDependences field in SLSA v1.0, and mapped to materials field in SLSA v0.1 & v0.2.
Output Artifacs: A canonical term used in this doc to refer to the artifacts that the build process produced i.e. an OCI image. This is mapped to Subjects field in all SLSA versions.
Results: Results are Tekton API fields that authors can use to emit some information after a TaskRun/PipelineRun is complete. Results can be used to pass along information to different tasks within a pipeline or aggregate different task results to a pipeline result. Check out Tekton official doc more information. Note: API result field is completely different from Tekton Results Operator.
Type hinting: Refer to specially named results/params that aim to enable Tekton Chains to understand the input artifacts and outputs of a PipelineRun/TaskRun.

How does Tekton Chains work?

Tekton Chains works by reconciling the run of a task or a pipeline. Once the run is observed as completed, Tekton Chains will take a snapshot of the completed TaskRun/PipelineRun, and start its core works in the order of formatting (generate provenance json) -> signing (sign the payload using the key configured by user) -> uploading (upload the provenance and its signature to the storage configured by user).

How to configure Tekton Chains

Tekton Chains supports both SLSA v0.2 and v1.0 provenance for both task-level and pipeline-level provenance.

The following shows the mapping between slsa version and formatter name.

SLSA Version	Formatter Name
v1.0	`slsa/v2alpha3` and `slsa/v2alpha4`
v0.2	`slsa/v1` or `in-toto`

To configure Task-level provenance version

kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.taskrun.format": "slsa/v1"}}'

To configure Pipeline-level provenance version

kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.pipelinerun.format": "slsa/v1"}}'

Note:
While Chains is able to generate both task-level and pipeline-level provenance at the same time, it’s not recommended to upload both to a storage backend because it would be confusing to have 2 different provenances for the same artifact.
To disable Task-level provenance, simply config empty string "" as the storage backend
kubectl patch configmap chains-config -n tekton-chains -p='> {"data":{"artifacts.taskrun.storage": ""}}'
To disable Pipeline-level provenance, simply config empty string "" as the storage backend
kubectl patch configmap chains-config -n tekton-chains -p='> {"data":{"artifacts.pipelinerun.storage": ""}}'

How to configure a Task or Pipeline

As mentioned in the Glossary, SLSA provenance describes the build process of a particular artifact being produced. While Tekton Chains is able to capture the build process regardless of how the pipeline was configured, it is mandatory to signal Chains what the output and input artifacts are in the pipeline config. The way to do that is through the type hinting.

Task-level Provenance: The type hinting carrying the references of input/output artifacts should be defined in the TaskSpec.
Pipeline-level Provenance: The type hinting carrying the references of input/output artifacts can be defined either:
- in the PipelineSpec
- in the TaskSpec. However, in this case, the feature flag artifacts.pipelinerun.enable-deep-inspection must be enabled to instruct Chains to dive deep into each child TaskRuns to look for type-hinting.

Type Hinting

Type hinting is a way to let Chains to understand the input and output artifacts throughout a PipelineRun. Chains expects different type hinting names for inputs and outputs. However, both input and output artifacts should have the uri and digest components, which is the common thing in the following type hinting list.

Input Artifacts

Input artifacts can be defined either in params or results using one of following options.

It’s worth noting that the value for the digest component needs to be precise commit SHA. It can’t be other mutable references i.e. tag, branch name and so on.

Git Results

In this approach, one can define the url of the source code repository and the precise commit sha digest in type hinting exactly named as CHAINS-GIT_URL and CHAINS-GIT_COMMIT respectively.

Example TaskRun

apiVersion: tekton.dev/v1
kind: TaskRun
metadata:
  name: git-clone
spec:
  taskSpec:
    params:
      - name: url
        description: Repository URL to clone from.
        type: string
        default: "https://github.com/tektoncd/pipeline"
      - name: revision
        description: Revision to checkout. (branch, tag, sha, ref, etc...)
        type: string
        default: "main"
    results:
      - name: CHAINS-GIT_URL
        type: string
        description: The precise URL that was fetched by this Task.
      - name: CHAINS-GIT_COMMIT
        type: string
        description: The precise commit SHA that was fetched by this Task.
    steps:
      - name: dummy-clone
        image: bash:latest
        script: |
          #!/usr/bin/env bash
          echo -n "https://github.com/tektoncd/pipeline" | tee $(results.CHAINS-GIT_URL.path)
          echo -n "7f2f46e1b97df36b2b82d1b1d87c81b8b3d21601" | tee $(results.CHAINS-GIT_COMMIT.path)

Note: This can be either params or results.

Param: To surface a git URL/commit as the input artifact, add a parameter named CHAINS-GIT_COMMIT and CHAINS-GIT_URL. The value of these parameters should be fed by some VCS task (e.g like this task) so that the reported url and revision are guaranteed to be the one that was fetched. A PipeLine example where another task checkout has URL/commit as task results:
```
- name: build
  params:
    - name: CHAINS-GIT_COMMIT
      value: "$(tasks.checkout.results.commit)"
    - name: CHAINS-GIT_URL
      value: "$(tasks.checkout.results.url)"
```

Result: Alternatively, CHAINS-GIT_COMMIT and CHAINS-GIT_URL can be results instead. Another Pipeline example where results are used:

spec:
  results:
    - description: Repository URL used for buiding the image.
      name: CHAINS-GIT_URL
      value: $(tasks.checkout.results.url)
    - description: Repository commit used for building the image.
      name: CHAINS-GIT_COMMIT
      value: $(tasks.checkout.results.commit)
  tasks:
    - name: checkout

`*ARTIFACT_INPUTS`

Note:
* indicates any expression

In this approach, one can group the url of the source code repository and the precise commit sha into a single object type hinting. The object type hinting only needs to have the suffix ARTIFACT_INPUTS and have the 2 keys exactly named as uri and digest. This is particularly useful if there are multiple input artifacts. For example, one object type hinting can be first_ARTIFACT_INPUTS and another one is second_ARTIFACT_INPUTS.

Note:

The digest component must be in the format of cryptographic hash algorithm name + : + a valid hex value i.e. “sha1:7f2f46e1b97df36b2b82d1b1d87c81b8b3d21601”.

Example TaskRun

apiVersion: tekton.dev/v1
kind: TaskRun
metadata:
  name: git-clone
spec:
  taskSpec:
    params:
      - name: url
        description: Repository URL to clone from.
        type: string
        default: "https://github.com/tektoncd/pipeline"
      - name: revision
        description: Revision to checkout. (branch, tag, sha, ref, etc...)
        type: string
        default: "main"
    results:
      - name: source_repo_ARTIFACT_INPUTS
        description: The source code repo artifact
        type: object
        properties:
          uri: {}
          digest: {}
    steps:
      - name: dummy-clone
        image: bash:latest
        script: |
          #!/usr/bin/env bash
          echo -n "{\"uri\":\"https://github.com/tektoncd/pipeline\", \"digest\":\"sha1:7f2f46e1b97df36b2b82d1b1d87c81b8b3d21601\"}" > $(results.source_repo_ARTIFACT_INPUTS.path)

Output Artifacts

Output artifacts should be defined in results only, using one of following options.

`IMAGE_URL` / `IMAGE_DIGEST`

In this approach, one can write the url and digest of an output OCI artifact into 2 results that have same prefix, but the one for url has suffix IMAGE_URL and the one for digest has suffix IMAGE_DIGEST.

Note:

The IMAGE_URL component must be a valid container repository URL.

The IMAGE_DIGEST component must be in the format of cryptographic hash algorithm name + : + a valid hex value i.e. “sha256:586789aa031fafc7d78a5393cdc772e0b55107ea54bb8bcf3f2cdac6c6da51ee”

Example TaskRun

apiVersion: tekton.dev/v1
kind: TaskRun
metadata:
  name: image-build
spec:
  taskSpec:
    results:
      - name: first-image-IMAGE_URL
        type: string
        description: The precise URL of the OCI image built.
      - name: first-image-IMAGE_DIGEST
        type: string
        description: The algorithm and digest of the OCI image built.
    steps:
      - name: dummy-build
        image: bash:latest
        script: |
          #!/usr/bin/env bash
          echo -n "gcr.io/foo/bar" | tee $(results.first-image-IMAGE_URL.path)
          echo -n "sha256:586789aa031fafc7d78a5393cdc772e0b55107ea54bb8bcf3f2cdac6c6da51ee" | tee $(results.first-image-IMAGE_DIGEST.path)

`IMAGES`

Multiple images can also be specified by using a single IMAGES Result. The value of the IMAGES result is a list of images, each qualified by digest. The list of images can be separated by commas or by newlines.

Example TaskRun

apiVersion: tekton.dev/v1
kind: TaskRun
metadata:
  name: image-build
spec:
  taskSpec:
    results:
      - name: IMAGES
        description: The multiple image artifacts
        type: string
    steps:
      - name: dummy-build
        image: bash:latest
        script: |
          #!/usr/bin/env bash
          echo -n "img1@sha256:digest1, img2@sha256:digest2" | tee $(results.IMAGES.path)

`ARTIFACT_URI` / `ARTIFACT_DIGEST`

Similar to option 1 - IMAGE_URL and IMAGE_DIGEST, but just with different names.

`*ARTIFACT_OUTPUTS`

In this approach, one can group the url and digest of the output artifact a single object result. The object result only needs to have the suffix ARTIFACT_OUTPUTS and have the 2 keys exactly named as uri and digest. This is particularly useful if there are multiple artifacts produced throughout a task. For example, one object type hinting can be first_ARTIFACT_OUTPUTS and another one is second_ARTIFACT_OUTPUTS.

Note:

The digest component must be in the format of cryptographic hash algorithm name + : + a valid hex value i.e. “sha256:586789aa031fafc7d78a5393cdc772e0b55107ea54bb8bcf3f2cdac6c6da51ee”.

Example TaskRun

apiVersion: tekton.dev/v1
kind: TaskRun
metadata:
  name: image-build
spec:
  taskSpec:
    results:
      - name: first-ARTIFACT_OUTPUTS
        description: The first artifact built
        type: object
        properties:
          uri: {}
          digest: {}
    steps:
      - name: dummy-build
        image: bash:latest
        script: |
          #!/usr/bin/env bash
          echo -n "{\"uri\":\"gcr.io/foo/bar\", \"digest\":\"sha256:586789aa031fafc7d78a5393cdc772e0b55107ea54bb8bcf3f2cdac6c6da51ee\"}" > $(results.first-ARTIFACT_OUTPUTS.path)

`v2alpha4` formatter

Starting with version v2alpha4, the type-hinted object results value now can include a new boolean flag called isBuildArtifact. When set to true, this flag indicates the output artifact should be considered as subject in the executed TaskRun/PipelineRun.

The isBuildArtifact can be set in results whose type-hint uses the *ARTIFACT_OUTPUTS format. Results using the IMAGES and *IMAGE_URL / *IMAGE_DIGEST type-hint format will still be considered as subject automatically; all other results will be classified as byProduct

For instance, in the following TaskRun:

apiVersion: tekton.dev/v1
kind: TaskRun
metadata:
  name: image-build
spec:
  taskSpec:
    results:
      - name: first-ARTIFACT_OUTPUTS
        description: The first artifact built
        type: object
        properties:
          uri: {}
          digest: {}
      
      - name: second-ARTIFACT_OUTPUTS
        description: The second artifact built
        type: object
        properties:
          uri: {}
          digest: {}
          isBuildArtifact: {}
      
      - name: third-IMAGE_URL
        type: string
      - name: third-IMAGE_DIGEST
        type: string

      - name: IMAGES
        type: string
    steps:
      - name: dummy-build
        image: bash:latest
        script: |
          echo -n "{\"uri\":\"gcr.io/foo/img1\", \"digest\":\"sha256:586789aa031fafc7d78a5393cdc772e0b55107ea54bb8bcf3f2cdac6c6da51ee\"}" > $(results.first-ARTIFACT_OUTPUTS.path)

          echo -n "{\"uri\":\"gcr.io/foo/img2\", \"digest\":\"sha256:05f95b26ed10668b7183c1e2da98610e91372fa9f510046d4ce5812addad86b5\", \"isBuildArtifact\":\"true\"}" > $(results.second-ARTIFACT_OUTPUTS.path)

          echo -n "gcr.io/foo/bar" | tee $(results.third-IMAGE_URL.path)
          echo -n "sha256:05f95b26ed10668b7183c1e2da98610e91372fa9f510046d4ce5812addad86b6" | tee $(results.third-IMAGE_DIGEST.path)

          echo -n "gcr.io/test/img3@sha256:2996854378975c2f8011ddf0526975d1aaf1790b404da7aad4bf25293055bc8b, gcr.io/test/img4@sha256:ef334b5d9704da9b325ed6d4e3e5327863847e2da6d43f81831fd1decbdb2213" | tee $(results.IMAGES.path)

second-ARTIFACT_OUTPUTS, third-IMAGE_URL/third-IMAGE_DIGEST, and IMAGES will be considered as subject. first-ARTIFACT_OUTPUTS doesn’t specify isBuildArtifact: true so it is not count as subject.

Chains’ v2alpha4 formatter now automatically reads type-hinted results from StepActions associated to the executed TaskRun/PipelineRun; users no longer need to manually surface these results from the StepActions when the appropriate type hints are in place. PipelineRuns require artifacts.pipelinerun.enable-deep-inspection: true for this functionality to work. For instance, with the following TaskRun:

apiVersion: tekton.dev/v1alpha1
kind: StepAction
metadata:
  name: img-builder
spec:
  image: busybox:glibc

  results:
    - name: first-ARTIFACT_OUTPUTS
      description: The first artifact built
      type: object
      properties:
        uri: {}
        digest: {}
        isBuildArtifact: {}

    - name: second-IMAGE_URL
      type: string
    - name: second-IMAGE_DIGEST
      type: string

  script: |
    echo -n "{\"uri\":\"gcr.io/foo/img1\", \"digest\":\"sha256:586789aa031fafc7d78a5393cdc772e0b55107ea54bb8bcf3f2cdac6c6da51ee\", \"isBuildArtifact\": \"true\" }" > $(step.results.first-ARTIFACT_OUTPUTS.path)

    echo -n "gcr.io/foo/bar" > $(step.results.second-IMAGE_URL.path)
    echo -n "sha256:05f95b26ed10668b7183c1e2da98610e91372fa9f510046d4ce5812addad86b6" > $(step.results.second-IMAGE_DIGEST.path)    
---
apiVersion: tekton.dev/v1
kind: TaskRun
metadata:
  name: taskrun
spec:
  taskSpec:
    steps:
      - name: action-runner
        ref:
          name: img-builder

Chains Will read first-ARTIFACT_OUTPUTS and second-IMAGE_URL/second-IMAGE_DIGEST from the StepAction and classify them as a subject.

Besides inputs/outputs

Tekton Chains is also able to capture the feature flags being used for Tekton Pipelines controller and the origin of the build configuration file with immutable references such as task.yaml and pipeline.yaml. However, those fields in Tekton Pipelines are gated by a dedicated feature flag. Therefore, the feature flag needs to be enabled to let Tekton Pipelines controller to populate these fields.

kubectl patch -n tekton-pipelines configmap feature-flags -p '{"data":{"enable-provenance-in-status":"true"}}'

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Sigstore

Transparency Log Support

Chains supports automatic binary uploads to a transparency log and defaults to using Rekor. If enabled, all signatures and attestations will be logged. The entry ID will be appended as an annotation on a TaskRun or a PipelineRun once Chains has uploaded it:

chains.tekton.dev/transparency: https://rekor.sigstore.dev/7599

Enabling Transparency Log Support

To enable, run:

kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"transparency.enabled": "true"}}'

Right now, Chains default to storing entries in the public Rekor instance (https://rekor.sigstore.dev). To customize where entries are stored, run:

kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"transparency.url": "<YOUR URL>"}}'

Keyless Signing Mode

Chains also supports a keyless signing mode with Fulcio, Sigstore’s free root certificate authority.

In this mode, instead of setting up a signing key, Chains would request an identity token from the cluster it is running in. This identity token will be used to authorize a Fulcio certificate for a Tekton artifact (OCI image, TaskRun, or PipelineRun). This feature has been tested on GKE, EKS, and AKS, but should work on any environment that supports Cosign OIDC signing.

Once Chains has successfully requested a certificate, it will store the cert as a base64 encoded annotation on the TaskRun or PipelineRun , along with the payload and signature.

This can look like:

Annotations:  chains.tekton.dev/cert-taskrun-57e7ef8e-13fb-4d27-af6e-dc4d68f73cc4:
              chains.tekton.dev/chain-taskrun-57e7ef8e-13fb-4d27-af6e-dc4d68f73cc4:
              chains.tekton.dev/payload-taskrun-57e7ef8e-13fb-4d27-af6e-dc4d68f73cc4:
                eyJfdHlwZSI6ImJ1aWxkLWNoYWlucy01dnhycyIsInByZWRpY2F0ZVR5cGUiOiJodHRwczovL3Rla3Rvbi5kZXYvY2hhaW5zL3Byb3ZlbmFuY2UiLCJzdWJqZWN0IjpbeyJuYW1lIj...
              chains.tekton.dev/signature-taskrun-57e7ef8e-13fb-4d27-af6e-dc4d68f73cc4:
                eyJwYXlsb2FkVHlwZSI6ImFwcGxpY2F0aW9uL3ZuZC5pbi10b3RvK2pzb24iLCJwYXlsb2FkIjoiZXlKZmRIbHdaU0k2SW1KMWFXeGtMV05vWVdsdWN5MDFkbmh5Y3lJc0luQnlaV1...
              chains.tekton.dev/signed: true
              chains.tekton.dev/transparency: https://rekor.sigstore.dev/7599
              pipeline.tekton.dev/release: v0.25.0

Enabling Keyless Signing Mode

To enable singing with Fulcio, run:

kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"signers.x509.fulcio.enabled": "true"}}'

Better Way Of Navigating in Transparency Log with rekor-search-ui

The chains.tekton.dev/transparency annotation on TaskRun and PipelineRun resources holds the URL to access the transparency log entry via Rekor’s API. It is also possible to view the log entry via Rekor’s web interface.

There is already a public good instance of the Rekor Search UI provided by the Chainguard team running at https://rekor.tlog.dev, which you can use to display the details of the log entry within the transparency log.

If you want to search your entry via logIndex, you can use the following URL:

https://rekor.tlog.dev/?logIndex=735223

But the logIndex is not the only option. You could also use email, hash, commit SHA, and UUID as an option to search the transparency log entry.

For example, you can search via email:

https://rekor.tlog.dev/?email=developerguyn@gmail.com

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Experimental Features

This doc covers experimental features in Tekton Chains.

Currently, experimental features include:

PubSub Storage Backend Support

PubSub Storage Backend Support

Support for PubSub storage backend was introduced in chains. The first PubSub provider implementation is Kafka, and more may follow in the future.

Kafka

To enable the Kafka backend run:

kubectl patch configmap chains-config -n tekton-chains -p='{"data": {storage.pubsub.provider": "kafka","storage.pubsub.topic": "chains", "storage.pubsub.kafka.bootstrap.servers":"kafka-0.kafka-headless.default.svc.cluster.local:9092"}}'

Note that the storage.pubsub.kafka.bootstrap.servers value needs to be adjusted to point to the list of bootstrap servers your cluster is connected to.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Chains Getting Started Tutorial

This tutorial will guide you through:

Generating your own keypair and storing it as a Kubernetes Secret
Configuring Tekton Chains backend storage
Creating a sample TaskRun
Retrieving the signature and payload from the signed TaskRun
Verifying the signature

We will be creating a TaskRun, signing it, and storing the signature and the payload as annotations on the TaskRun itself. So, no additional authentication should be required!

For this tutorial we will use the x509 key type.

x509

To generate your own encrypted x509 keypair and save it as a Kubernetes secret, install cosign and run the following:

cosign generate-key-pair k8s://tekton-chains/signing-secrets

cosign will prompt you for a password, which will be stored in a Kubernetes secret named signing-secrets in the tekton-chains namespace.

Configuring Tekton Chains

You will need to make sure that OCI storage is disabled and that the taskrun storage and format is set to tekton.

You can set these fields by running the following command:

kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.oci.storage": "", "artifacts.taskrun.format":"in-toto", "artifacts.taskrun.storage": "tekton"}}'

Then restart the controller to ensure it picks up the changes:

kubectl delete po -n tekton-chains -l app=tekton-chains-controller

This tells Chains to use the default tekton artifact (enabled by default) and disable the OCI artifact.

To create a simple TaskRun, run:

kubectl create -f https://raw.githubusercontent.com/tektoncd/chains/main/examples/taskruns/task-output-image.yaml

The output should be similar to:

taskrun.tekton.dev/build-push-run-output-image-qbjvh created

Wait for it to finish (all the steps should be marked as Completed).

$ tkn tr describe --last

[...truncated output...]

🦶 Steps

NAME                            STATUS
∙ create-dir-builtimage-9467f   Completed
∙ git-source-sourcerepo-p2sk8   Completed
∙ build-and-push                Completed
∙ echo                          Completed
∙ image-digest-exporter-xlkn7   Completed

Next, retrieve the signature and payload from the object (they are stored as base64-encoded annotations):

export TASKRUN_UID=$(tkn tr describe --last -o  jsonpath='{.metadata.uid}')
tkn tr describe --last -o jsonpath="{.metadata.annotations.chains\.tekton\.dev/signature-taskrun-$TASKRUN_UID}" | base64 -d > sig

Finally, we can check the signature with cosign:

$ cosign verify-blob-attestation --insecure-ignore-tlog --key k8s://tekton-chains/signing-secrets --signature sig --type slsaprovenance --check-claims=false /dev/null
Verified OK

If using Cosign v1

$ cosign verify-blob --key k8s://tekton-chains/signing-secrets --signature sig sig
Verified OK

Now we have a verifiable record of the TaskRun!

What you just created

This diagram shows what you just deployed:

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Chains Signed Provenance Tutorial

This tutorial will cover how to set up Chains to sign OCI images built in Tekton, and how to automatically generate and sign in-toto attestations for each image. This tutorial will also cover how to store these attestations in a transparency log and query the log for the attestation.

This tutorial will guide you through:

Generating your own keypair and storing it as a Kubernetes Secret
Setting up authentication for your OCI registry to store images, image signatures and signed image attestations
Configuring Tekton Chains to generate and sign provenance
Building an image with kaniko in a Tekton TaskRun
Verifying the signed image and the signed provenance

Prerequisites

A Kubernetes cluster with the following installed:

Tekton Chains
Tekton Pipelines

Generate a Key Pair

First, we’ll generate an encrypted x509 keypair and save it as a Kubernetes secret. Install cosign and run the following:

cosign generate-key-pair k8s://tekton-chains/signing-secrets

cosign will prompt you for a password, which will be stored in a Kubernetes secret named signing-secrets in the tekton-chains namespace.

The public key will be written to a local file called cosign.pub.

Set up Authentication

There are two forms of authentication that need to be set up:

The Chains controller will be pushing signatures to an OCI registry using the credentials linked to your TaskRun’s service account. See our authentication doc
The Kaniko Task that will build and push the image needs push permissions for your registry.

To set up auth for the Kaniko Task, you’ll need a Kubernetes secret of a docker config.json file which contains the required auth. You can create the secret by running:

kubectl create secret generic [DOCKERCONFIG_SECRET_NAME] --from-file [PATH TO CONFIG.JSON]

Configuring Tekton Chains

You’ll need to make these changes to the Tekton Chains Config:

artifacts.taskrun.format=slsa/v1
artifacts.taskrun.storage=oci
artifacts.oci.storage=oci
transparency.enabled=true

You can set these fields by running

kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.taskrun.format": "slsa/v1"}}'
kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.taskrun.storage": "oci"}}'
kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.oci.storage": "oci"}}'
kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"transparency.enabled": "true"}}'

Note, you can continue to use the older alias of slsa/v1: in-toto above.

This tells Chains to generate an in-toto attestation and store it in the specified OCI registry. Attestations will also be stored in rekor since transparency is enabled.

Start the Kaniko Task

Great, now that the setup is done we’re finally ready to build an image with kaniko!

First, apply the Kaniko Task to your cluster:

kubectl apply -f examples/kaniko/kaniko.yaml

and set the following environment variables:

REGISTRY=[The registry you'll be pushing to]
DOCKERCONFIG_SECRET_NAME=[The name of the secret with the docker config.json]

Then, you can start the Kaniko Task with the Tekton CLI tool, tkn:

tkn task start --param IMAGE=$REGISTRY/kaniko-chains --use-param-defaults --workspace name=source,emptyDir="" --workspace name=dockerconfig,secret=$DOCKERCONFIG_SECRET_NAME kaniko-chains

You can watch the logs of this Task until they complete; if authentication is set up correctly than the final image should be pushed to $REGISTRY/kaniko-chains.

Verifying the Image and Attestation

Once the TaskRun has successfully completed, you’ll need to wait a few seconds for Chains to generate provenance and sign it.

Once you see the chains.tekton.dev/signed=true annotation on your TaskRun you know that Chains has completed the signing process and you’re ready to move on to verification:

kubectl get tr [TASKRUN_NAME] -o json | jq -r .metadata.annotations

{
  "chains.tekton.dev/signed": "true",
  ...
}

To verify the image and the attestation, we’ll use cosign again:

cosign verify --key cosign.pub $REGISTRY/kaniko-chains
cosign verify-attestation --key cosign.pub --type slsaprovenance $REGISTRY/kaniko-chains

You should see verification output for both!

Finding Provenance in Rekor

To find provenance for the image in Rekor, first get the digest of the $REGISTRY/kaniko-chains image you just built. You can look this up in the TaskRun, or pull the image to get the digest.

You can then search rekor to find all entries that match the sha256 digest of the image you just built with the rekor-cli tool:

rekor-cli search --sha [IMAGE_DIGEST]

[UUID1]
[UUID2]

The search will print out the UUIDs of matching entries. It may take a little guessing, but one of those UUIDs holds the attestation. You can see the attestation by using jq:

rekor-cli get --uuid [UUID] --format json | jq -r .Attestation | base64 --decode | jq

Congratulations! You have officially built an image, signed it, and generated signed provenance for it with Tekton Chains 🎉

What you just created

This diagram shows what you just deployed:

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Using DocDB (MongoDB) as storage for Tekton Chains signed artifacts

This tutorial will guide you through setting up MongoDB as the storage backend for Tekton Chains. We will cover deploying a MongoDB instance, securely configuring authentication using Kubernetes Secrets, and understanding the different configuration options available.

Prerequisites

Before starting, ensure you have the following installed:

Kubernetes Cluster: A local cluster like kind is sufficient.
```
kind create cluster --name chains-mongo-test
```
kubectl: The Kubernetes command-line tool.
Cosign: For generating signing keys. Installation Guide

Step 1: Install Tekton Pipelines and Chains

# Install Tekton Pipelines
kubectl apply -f  https://infra.tekton.dev/tekton-releases/pipeline/latest/release.yaml

# Install Tekton Chains
kubectl apply -f https://infra.tekton.dev/tekton-releases/chains/latest/release.yaml

Wait for the components to be ready:

kubectl wait --for=condition=Ready pods --all -n tekton-chains --timeout=60s

Step 2: Set up Signing Keys

Generate a key pair for signing artifacts and store it as a Kubernetes Secret.

# Generate key pair and create the secret
cosign generate-key-pair k8s://tekton-chains/signing-secrets


## Step 3: Deploy a MongoDB Instance

For this tutorial, we will deploy a simple MongoDB instance inside your cluster and expose it via a Service on port 27017. (27017 is the standard port for MongoDB servers.)

```shell
# Run a MongoDB pod using the standard port 27017
kubectl run mongo1 --image=mongo:6 --port=27017

# Expose the pod as a Service
kubectl expose pod mongo1 --name=mongo1-svc --port=27017 --target-port=27017

Your MongoDB connection string is now: mongodb://mongo1-svc.default:27017

Step 4: Securely Configure MongoDB Authentication

Instead of hardcoding the connection string in the Chains configuration, we will store it in a Kubernetes Secret and mount it into the Chains controller.

Create the Secret

First, write the connection string to a file:

echo -n "mongodb://mongo1-svc.default:27017" > mongo-url.txt

Create a secret named mongo-secret containing this file:

kubectl create secret generic mongo-secret \
  -n tekton-chains \
  --from-file=mongo-url=mongo-url.txt

Mount the Secret to the Chains Controller

We need to update the Chains controller deployment to mount this secret so the application can read the connection string from a file.

We will patch the deployment to:

Add a volume for the mongo-secret.
Mount that volume to /etc/secrets/mongo. (This patch assumes the controller is the first controller)

kubectl patch deployment tekton-chains-controller \
  -n tekton-chains \
  --type='json' \
  -p='[
    {
      "op": "add",
      "path": "/spec/template/spec/volumes/-",
      "value": {
        "name": "mongo-secret",
        "secret": {
          "secretName": "mongo-secret"
        }
      }
    },
    {
      "op": "add",
      "path": "/spec/template/spec/containers/0/volumeMounts/-",
      "value": {
        "name": "mongo-secret",
        "mountPath": "/etc/secrets/mongo",
        "readOnly": true
      }
    }
  ]'

Restart the controller to apply the changes:

kubectl rollout restart deploy/tekton-chains-controller -n tekton-chains

Step 5: Configure Chains to use DocDB

Now we configure Chains to use the docdb backend and point it to the mounted secret file.

kubectl patch configmap chains-config -n tekton-chains --type merge -p '{
  "data": {
    "artifacts.taskrun.storage": "docdb",
    "artifacts.pipelinerun.storage": "docdb",
    "storage.docdb.url": "mongo://tekton-chains/attestations?id_field=name",
    "storage.docdb.mongo-server-url-path": "/etc/secrets/mongo/mongo-url"
  }
}'

Explanation of Configuration:

artifacts.taskrun.storage: Tells Chains to store TaskRun artifacts in docdb.
storage.docdb.url: The Go Cloud URL for the collection. mongo:// indicates MongoDB, tekton-chains is the database, attestations is the collection, and id_field=_id sets the primary key.
storage.docdb.mongo-server-url-path: The absolute path to the file containing the connection string (which we mounted in Step 4).

Configuration Priority for MongoDB URL

Chains offers multiple ways to provide the MongoDB connection string. They are checked in the following order of priority (highest to lowest):

storage.docdb.mongo-server-url-path (Recommended)
- Description: Path to a specific file containing the connection string.
- Use Case: When mounting a specific secret key as a file (as done in this tutorial).
- Example: /etc/secrets/mongo/mongo-url
storage.docdb.mongo-server-url-dir
- Description: Path to a directory. Chains looks for a file named MONGO_SERVER_URL inside this directory.
- Use Case: When mounting a whole secret or directory where the filename is implicitly expected to be MONGO_SERVER_URL.
storage.docdb.mongo-server-url
- Description: The connection string value directly.
- Use Case: Quick testing or non-sensitive environments. Not recommended for production as it exposes credentials in the ConfigMap.
MONGO_SERVER_URL Environment Variable
- Description: An environment variable set on the controller pod.
- Use Case: Legacy configuration or 12-factor app patterns where config maps aren’t used.

Step 6: Verify the Setup

Create a TaskRun Create a simple TaskRun to generate an artifact.

kubectl create -f - <<EOF
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  generateName: simple-task-
spec:
  taskSpec:
    steps:
    - name: echo
      image: alpine
      script: |
        echo "Hello World"
EOF

Check Logs Watch the Chains controller logs to see the artifact being stored.
```
kubectl logs -f deploy/tekton-chains-controller -n tekton-chains
```
You should see messages indicating successful storage to DocDB.
Query MongoDB Connect to the MongoDB pod to verify the data exists.
```
kubectl exec -it mongo1 -- mongosh --eval "use tekton-chains; db.attestations.find()"
```
You should see the stored provenance documents.

Understanding Stored Artifacts

When Tekton Chains processes a TaskRun or PipelineRun, it generates provenance (attestation) and signs it. When using the docdb backend, these artifacts are stored as documents in your MongoDB collection.

What Artifacts are Stored?

TaskRun / PipelineRun Provenance:
- By enabling artifacts.taskrun.storage: docdb (or artifacts.pipelinerun.storage: docdb), Chains stores the signed provenance of the execution itself.
- The payload is typically in SLSA (Supply-chain Levels for Software Artifacts) format.
OCI Image Signatures:
- If you enable artifacts.oci.storage: docdb, Chains can also store signatures for built container images in the database, rather than attaching them to the container registry.

Document Schema

Each artifact is stored as a single document with the following fields:

Field	Type	Description
`_id`	String	The unique identifier for the document (e.g., `taskrun-uid`).
`name`	String	The name/key of the artifact.
`signed`	Binary	The raw payload that was signed (e.g., the SLSA provenance JSON).
`signature`	String	The base64-encoded signature of the payload.
`cert`	String	The certificate used for signing (if using X.509).
`chain`	String	The certificate chain (if applicable).
`object`	Object	The unmarshaled JSON object of the payload. This allows you to query fields directly (e.g., `db.attestations.find({"object.predicate.buildType": "..."})`).

Example Document (JSON representation):

{
  "_id": "taskrun-simple-task-abcde",
  "name": "taskrun-simple-task-abcde",
  "signed": <Binary Data>,
  "signature": "MEUCIQD...",
  "object": {
    "_type": "https://in-toto.io/Statement/v0.1",
    "subject": [...],
    "predicateType": "https://slsa.dev/provenance/v0.2",
    "predicate": { ... }
  }
}

Advanced: Hot Reloading Secrets

One advantage of using storage.docdb.mongo-server-url-path with Kubernetes Secrets is that Chains watches the file for changes. If you rotate your database credentials and update the Secret, Kubernetes updates the mounted file, and Chains automatically reloads the connection string without needing a restart.

To test this:

Spin up a second MongoDB instance (mongo2).
Update the mongo-secret with the new URL (mongodb://mongo2-svc.default:27017).
Chains will detect the change and switch to the new database automatically.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Using Google Cloud Storage (GCS) as the storage for Tekton Chains signed artifacts

This tutorial will guide you through the steps to enable the artifacts generated by Tekton Chains running on a Kind cluster to be uploaded to a GCS bucket.

Scope of the Tutorial

Before you begin, ensure that you have the following tools installed on your machine:

kind: To create and manage the local Kubernetes cluster.
gcloud: To interact with Google Cloud resources, such as GCS and IAM.
cosign: To sign and verify artifacts (if required).

Solution Overview

Configure Workload Identity Federation with the Kuberenetes cluster. For detailed instructions follow documentation here.

Kubernetes can project service account tokens into workloads using projected volumes. By enabling Workload Identity Federation, we allow these tokens to be used by workloads to authenticate with Google Cloud services. This eliminates the need to hardcode keys or store sensitive credentials for authentication. In our case, the tekton-chains-controller deployment running on the kind cluster must be configured to mount the projected service account token volume to use these tokens.

The following diagram illustrates the steps we will take.

Prerequisites

Google Cloud Project linked to a Billing Account. If you do not have one, you can create a new project following the documentation here.

gcloud auth login

gcloud projects create tekton-chains-project

Ensure that the project is linked to a valid Billing Account.This can be done from console or from CLI. Refer docs.

gcloud config set project tekton-chains-project

Google Cloud Storage (GCS) Bucket - You will need a bucket to which the artifacts will be uploaded. You can create a new GCS bucket using the following command

gcloud storage buckets create gs://tekton_artifacts --location=us-central1

Kubernetes cluster to deploy Tekton pipelines and chains

kind create cluster -n tekton

Install Tekton Pipelines and Tekton Chains

kubectl apply --filename https://infra.tekton.dev/tekton-releases/pipeline/latest/release.yaml
kubectl apply --filename https://infra.tekton.dev/tekton-releases/chains/latest/release.yaml

Ensure that all pods in the tekton-pipelines and tekton-chains namespaces are running, and that the pods are in the “Running” status. Below is an example of what the pods should look like:

tekton-chains                tekton-chains-controller-658dbffcd4-cqqxj           1/1     Running   0          69s
tekton-pipelines             tekton-events-controller-fcdb69895-m4xjn            1/1     Running   0          2m14s
tekton-pipelines             tekton-pipelines-controller-77fb6d9c6c-lkbkp        1/1     Running   0          2m14s
tekton-pipelines             tekton-pipelines-webhook-6f69779d47-j5lkx           1/1     Running   0          2m14s

This tutorial will be divided into 6 stages

Enable required api service and create a Workload Identity Pool and Provider in GCP
Create a Google Cloud IAM service Account and grant GCS Object admin role
Configure GCP Workload Identity Federation
Grant access to kubernetes workload
Update Chains configuration to enable GCS storage
Execute a task and verify artifact storage in GCS bucket

Stage-1: Create Workload Identity Pool and Provider

Ensure folloiwng services are enabled in the gcloud project

gcloud services enable iam.googleapis.com
gcloud services enable cloudresourcemanager.googleapis.com
gcloud services enable iamcredentials.googleapis.com
gcloud services enable sts.googleapis.com

Create the workload identity pool named kind-tekton-demo-pool

gcloud iam workload-identity-pools create kind-tekton-demo-pool --location="global" --description="BYO-workload identity demo" --display-name="Bare Metal Kind Cluster Pool"

Add the OIDC provider corresponding to the Kind cluster to the pool.

First, retrieve the JSON Web Key Set (JWKS) from the Kubernetes API server. JWKS is typically used in OpenID Connect (OIDC) scenarios to validate tokens.

kubectl get --raw /openid/v1/jwks > cluster-jwks.json

Get the issuer uri for the kind cluster using

kubectl get --raw /.well-known/openid-configuration | jq -r .issuer

For a kind cluster the uri would be:

https://kubernetes.default.svc.cluster.local

Then add the provider to the pool created. use the json and uri obtained in previous steps for the folloiwng command

gcloud iam workload-identity-pools providers create-oidc kind-cluster-provider \
  --location="global" \
  --workload-identity-pool="kind-tekton-demo-pool" \
  --issuer-uri="https://kubernetes.default.svc.cluster.local" \
  --attribute-mapping="google.subject=assertion.sub,attribute.namespace=assertion['kubernetes.io']['namespace'],attribute.service_account_name=assertion['kubernetes.io']['serviceaccount']['name'],attribute.pod=assertion['kubernetes.io']['pod']['name']" \
  --attribute-condition="assertion['kubernetes.io']['namespace'] in ['tekton-chains']" \
  --jwk-json-path="cluster-jwks.json"

Important: In the above command please note that the --attribute-condition is set to allow uploads from workloads/pods in tekton-chains namesapce only. If chains pod is running in someother namespace the condition should be updated.

Stage-2: Create a Google Cloud IAM service Account and grant GCS Object admin role

Create a Gcloud IAM service account. In this tutorial, we will use the name gcp-sa-for-kind, but feel free to choose a different name if you prefer. Just ensure to update the name in the subsequent commands accordingly.

gcloud iam service-accounts create gcp-sa-for-kind

Grant the IAM service account access to resources that we want the Kubernetes workload to access. In this tutorial the kind workload i.e the chains pod has to upload to a gcs bucket. Hence we will bind to role storage.objectAdmin, which provides full control over the objects (files) within a GCS bucket, which includes both read and write access. Note:Setting up the correct GCS permissions is up to the user. For the scope of tutorial we are restricting to objectAdmin.

export PROJECT_NUMBER=$(gcloud projects describe $(gcloud config get-value project) --format='value(projectNumber)')
export SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list --filter="name:gcp-sa-for-kind" --format='value(email)')
gcloud projects add-iam-policy-binding ${PROJECT_NUMBER} \
  --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
  --role="roles/storage.objectAdmin"

Stage-3: Configure GCP Workload Identity Federation

export PROJECT_NUMBER=$(gcloud projects describe $(gcloud config get-value project) --format='value(projectNumber)')
export SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list --filter="name:gcp-sa-for-kind" --format='value(email)')
export SUBJECT="system:serviceaccount:tekton-chains:tekton-chains-controller"
export POOL_ID=$(gcloud iam workload-identity-pools list --location=global --format="value(name.basename())") 
export PROVIDER_ID=$(gcloud iam workload-identity-pools providers list --workload-identity-pool=${POOL_ID} --location=global --format="value(name.basename())")

WARN: command for POOL_ID works only if there is one pool. If the project has multiple pools, then set the pool name directly from the pool creation step in Stage-1

Now we need to bind a policy to the IAM service account to allow it to use the workload Identity Federation and allow specify that the tekton-chains service account tekton-chains-controller can access the IAM service account.

gcloud iam service-accounts add-iam-policy-binding ${SERVICE_ACCOUNT_EMAIL} \
--role="roles/iam.workloadIdentityUser" \
--member="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${POOL_ID}/subject/${SUBJECT}"

Stage-4: Grant access to the Kuberenetes workload

Now that the configuration on GCP is complete, we need to configure the Kind cluster’s tekton-chains-controller service account with the credentials to access GCS resources.

To do this, we need to create a credential configuration file using the following command:

gcloud iam workload-identity-pools create-cred-config \
projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${POOL_ID}/providers/${PROVIDER_ID} \
--service-account=${SERVICE_ACCOUNT_EMAIL} \
--credential-source-file=/var/run/service-account/token \
--credential-source-type=text \
--output-file=credential-configuration.json

This will generate a credential configuration file, enabling the service account to access the GCS resources.

To ensure the tekton-chains-controller service account can access the necessary GCS resources, store the credential configuration in a ConfigMap. Then, we can volume-mount the ConfigMap onto the tekton-chains-controller deployment to make the configuration available to the workload.

kubectl create configmap kind-demo-wid-test --from-file credential-configuration.json --namespace tekton-chains

Note: We need to edit the chains deployment to include the configmap volume mount, corresponding volume and an environment variable. Below is the sample patch deployment for tekton-chains-controller with volume Mounts -workload-identity-credential-configuration,token; volumes - workload-identity-credential-configuration,token and an env variable GOOGLE_APPLICATION_CREDENTAILS.

kubectl patch deployment tekton-chains-controller -n tekton-chains --type='strategic' -p '
spec:
  template:
    spec:
      containers:
      - name: tekton-chains-controller
        env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /etc/workload-identity/credential-configuration.json
        volumeMounts:
        - mountPath: /var/run/service-account
          name: token
          readOnly: true
        - mountPath: /etc/workload-identity
          name: workload-identity-credential-configuration
          readOnly: true
      volumes:
      - name: token
        projected:
          defaultMode: 420
          sources:
          - serviceAccountToken:
              audience: https://iam.googleapis.com/projects/<PROJECT_NUMBER>/locations/global/workloadIdentityPools/<POOL_NAME>/providers/<PROVIDER_ID>
              expirationSeconds: 3600
              path: token
      - name: workload-identity-credential-configuration
        configMap:
          defaultMode: 420
          name: kind-demo-wid-test
'

!IMP Ensure to replace the PROJECT_NUMBER,POOL_NAME,PROVIDER_ID in theserviceAccountToken.audience corresponding to the token volume spec.

With the updated deployment ensure that the chains controller pod is running and the logs do not have any error

Stage-5. Update Chains configuration to enable GCS storage

kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.taskrun.format": "slsa/v1"}}'
kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.taskrun.storage": "gcs"}}'
kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.oci.storage": "gcs"}}'
kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"transparency.enabled": "true"}}'
kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"storage.gcs.bucket": "tekton_artifacts"}}'

The chains pod will automatically pick the changes done to the configmap. This can be verified in the tekton-chains-controller pod logs.

Stage-6. Execute a task and verify artifact storage in GCS bucket

Great, now that the setup is done we’re finally ready to build an image with kaniko!

First, apply the Kaniko Task to your cluster:

kubectl apply -f examples/kaniko/kaniko.yaml

and set the following environment variables:

export REGISTRY=[The registry you will be pushing to]

Assuming that kaniko task is executed from default namespace and the task by default uses the default service account to push the image to REGISTRY, set the following environment variables

export SERVICE_ACCOUNT_NAME=default
export NAMESPACE=default

docker login $REGISTRY #This helps to generate the dockerconfig locally at $HOME/.docker/config.json. Optional step in case we have the json handy

export DOCKERCONFIG_FILE_PATH=$HOME/.docker/config.json
kubectl create secret generic docker-registry --from-file=.dockerconfigjson=$DOCKERCONFIG_FILE_PATH --type=kubernetes.io/dockerconfigjson -n $NAMESPACE
kubectl patch serviceaccount $SERVICE_ACCOUNT_NAME -p "{\"secrets\": [{\"name\": \"docker-registry\"}]}" -n $NAMESPACE
export DOCKERCONFIG_SECRET_NAME=test-kaniko
kubectl create secret generic $DOCKERCONFIG_SECRET_NAME --from-file $DOCKERCONFIG_FILE_PATH

For signing purposes, you can generate a Cosign key pair and store the private key in a Kubernetes secret, while the public key (needed for verifying artifacts) will be stored in the local working directory.

To generate the key pair, use the following command:

cosign generate-key-pair k8s://tekton-chains/signing-secrets

Then, you can start the Kaniko Task with the Tekton CLI tool, tkn:

tkn task start --param IMAGE=$REGISTRY/kaniko-chains --use-param-defaults --workspace name=source,emptyDir="" --workspace name=dockerconfig,secret=$DOCKERCONFIG_SECRET_NAME kaniko-chains

You can watch the logs of this Task until they complete; if authentication is set up correctly than the final image should be pushed to $REGISTRY/kaniko-chains & the signed artifacts and attestation should be uploaded to the gcs bucket - tekton_artifacts.

Once the task run is succeeded the uploaded gcloud artifacts can be listed using the following command:

gcloud storage objects list gs://tekton_artifacts/** --format="json(name)"

Sample output for reference:

[
  {
    "name": "taskrun-default-kaniko-chains-run-zflcs/28998e6f75e5.cert"
  },
  {
    "name": "taskrun-default-kaniko-chains-run-zflcs/28998e6f75e5.chain"
  },
  {
    "name": "taskrun-default-kaniko-chains-run-zflcs/28998e6f75e5.payload"
  },
  {
    "name": "taskrun-default-kaniko-chains-run-zflcs/28998e6f75e5.signature"
  },
  {
    "name": "taskrun-default-kaniko-chains-run-zflcs/taskrun-65c59603-749f-4945-bcb0-821a2ae24e81.cert"
  },
  {
    "name": "taskrun-default-kaniko-chains-run-zflcs/taskrun-65c59603-749f-4945-bcb0-821a2ae24e81.chain"
  },
  {
    "name": "taskrun-default-kaniko-chains-run-zflcs/taskrun-65c59603-749f-4945-bcb0-821a2ae24e81.payload"
  },
  {
    "name": "taskrun-default-kaniko-chains-run-zflcs/taskrun-65c59603-749f-4945-bcb0-821a2ae24e81.signature"
  }
]

The .signature file can be downloaded to the current local directory and verified with cosign

gcloud storage cp "gs://tekton_artifacts/taskrun-default-kaniko-chains-run-zflcs/taskrun-65c59603-749f-4945-bcb0-821a2ae24e81.signature" .

The resulting file could be verified with the following cosign command

cosign verify-blob-attestation --insecure-ignore-tlog --key k8s://tekton-chains/signing-secrets --signature taskrun-65c59603-749f-4945-bcb0-821a2ae24e81.signature --type slsaprovenance --check-claims=false /dev/null

Sample output for reference:

Verified OK

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Deprecations

Introduction

This doc provides a list of features in Tekton Chains that are being deprecated.

Deprecations will follow this timeline:

Deprecation announcement is made during a release
Feature is removed two releases later

So, if a feature is deprecated at v0.1.0, then it would be removed in v0.3.0.

Deprecation Table

Feature Being Deprecated	Deprecation Announcement	API Compatibility Policy	Earliest Date or Release of Removal
`tekton-provenance` format is deprecated	v0.6.0	Alpha	v0.8.0

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Hashicorp Integration with Tekton Chains

In this tutorial, we will be running Chains Signed Provenance Tutorial using `kms` solution by integrating Tekton Chains with Hashicorp Vault

NOTE

Prerequisite
- Hashicorp vault should be installed
  - If not installed, you can also try on a minikube or a kind cluster. For more info see here
  - This provider also requires that the transit secret engine is enabled
  - If not done, you can login into the vault provider and run the following command
```
$ vault secrets enable transit
Success! Enabled the transit secrets engine at: transit/
```
- You can get more info about transit secrets here
- Make sure Tekton Pipelines and Tekton Chains is installed

Chains Signed Provenance Tutorial using `kms` solution by integrating with Hashicorp Vault

Step 1: Generate a Key Pair

This provider requires that the standard Vault environment variables ($VAULT_ADDR, $VAULT_TOKEN) are set correctly.

$ export VAULT_ADDR=http://localhost:8200
$ export VAULT_TOKEN=testtoken
$ vault secrets enable transit  ---> Ignore if you have already done

Now run the following command

$ cosign generate-key-pair --kms hashivault://$keyname

NOTE:- If you enabled transit secret engine at different path with the use of -path flag (i.e., $ vault secrets enable -path=“someotherpath” transit), you can use TRANSIT_SECRET_ENGINE_PATH environment variable to specify this path while generating a key pair like the following:

In that case the command will be

$ TRANSIT_SECRET_ENGINE_PATH="someotherpath" cosign generate-key-pair --kms hashivault://$keyname

Step 2: Set up Authenticaion

There are two forms of authentication that need to be set up:

The Chains controller will be pushing signatures to an OCI registry using the credentials linked to your TaskRun’s service account. See our authentication doc
The Kaniko Task that will build and push the image needs push permissions for your registry.

To set up auth for the Kaniko Task, you’ll need a Kubernetes secret of a docker config.json file which contains the required auth. You can create the secret by running:

kubectl create secret generic [DOCKERCONFIG_SECRET_NAME] --from-file [PATH TO CONFIG.JSON]

Step 3: Configuring Tekton Chains

You’ll need to make these changes to the Tekton Chains configMap i.e. chains-config configMap:

* artifacts.taskrun.format: slsa/v1
* artifacts.taskrun.storage: oci
* artifacts.taskrun.signer: kms
* artifacts.pipelinerun.signer: kms
* artifacts.oci.signer: kms
* transparency.enabled: "true"
* signers.kms.kmsref: hashivault://$keyname
* signers.kms.auth.address: <VAULT_ADDR>
* signers.kms.auth.token: <VAULT_TOKEN>

Step 4: Start the Kaniko Task

First apply the

kubectl apply -f examples/kaniko/kaniko.yaml

Substitute with the URI or file path to your Kaniko task.

Set the following enviornment variables:

export REGISTRY=<url_of_registry>
export DOCKERCONFIG_SECRET_NAME=<name_of_the_secret_in_docker_config_json>

Substitute with the URL of the registry where you want to push the image. Substitute with the name of the secret in the docker config.json file.

Start the Kaniko Task

tkn task start --param IMAGE=$REGISTRY/kaniko-chains --use-param-defaults --workspace name=source,emptyDir="" --workspace name=dockerconfig,secret=$DOCKERCONFIG_SECRET_NAME kaniko-chains

Wait for a minute to allow Tekton Chains to generate the provenance and sign it, and then check the availability of the chains.tekton.dev/signed=true annotation on the task run.

kubectl get tr <task_run_name> -o json | jq -r .metadata.annotations
{
	"chains.tekton.dev/signed": "true",
	...
}

Step 5: Verify the image and the attestation

cosign verify --key cosign.pub $REGISTRY/kaniko-chains
cosign verify-attestation --key cosign.pub --type slsaprovenance $REGISTRY/kaniko-chains

or you can use the hashivault://$keyname as key as well

cosign verify --key hashivault://testkey $REGISTRY/kaniko-chains
cosign verify-attestation --key hashivault://testkey --type slsaprovenance $REGISTRY/kaniko-chains

The output would be like this

Verification for index.docker.io/$REGISTRY/kaniko-chains:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The claims were present in the transparency log
  - The signatures were integrated into the transparency log when the certificate was valid
  - The signatures were verified against the specified public key

[{"critical":{"identity":{"docker-reference":"index.docker.io/$REGISTRY/kaniko-chains"},"image":{"docker-manifest-digest":"sha256:e14396b283abcbacddba403a923a7fdecf2c54537a1d6a1ee1076767bec742d1"},"type":"cosign container image signature"},"optional":null}]

Verification for docker.io/$REGISTRY/kaniko-chains --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The claims were present in the transparency log
  - The signatures were integrated into the transparency log when the certificate was valid
  - The signatures were verified against the specified public key
{"payloadType":"application/vnd.in-toto+json","payload":"eyJfdHlwZSI6Imh0dHBzOi8vaW4tdG90by5pby9TdGF0ZW1lbnQvdjAuMSIsInByZWRpY2F0ZVR5cGUiOiJodHRwczovL3Nsc2EuZGV2L3Byb3ZlbmFuY2UvdjAuMiIsInN1YmplY3QiOlt7Im5hbWUiOiJpbmRleC5kb2NrZXIuaW8vcHVuZWV0MjE0Ny9rYW5pa28tY2hhaW5zIiwiZGlnZXN0Ijp7InNoYTI1NiI6ImUxNDM5NmIyODNhYmNiYWNkZGJhNDAzYTkyM2E3ZmRlY2YyYzU0NTM3YTFkNmExZWUxMDc2NzY3YmVjNzQyZDEifX1dLCJwcmVkaWNhdGUiOnsiYnVpbGRlciI6eyJpZCI6Imh0dHBzOi8vdGVrdG9uLmRldi9jaGFpbnMvdjIifSwiYnVpbGRUeXBlIjoidGVrdG9uLmRldi92MWJldGExL1Rhc2tSdW4iLCJpbnZvY2F0aW9uIjp7ImNvbmZpZ1NvdXJjZSI6e30sInBhcmFtZXRlcnMiOnsiQlVJTERFUl9JTUFHRSI6Imdjci5pby9rYW5pa28tcHJvamVjdC9leGVjdXRvcjp2MS41LjFAc2hhMjU2OmM2MTY2NzE3ZjdmZTBiN2RhNDQ5MDhjOTg2MTM3ZWNmZWFiMjFmMzFlYzM5OTJmNmUxMjhmZmY4YTk0YmU4YTUiLCJDT05URVhUIjoiLi8iLCJET0NLRVJGSUxFIjoiLi9Eb2NrZXJmaWxlIiwiRVhUUkFfQVJHUyI6IiIsIklNQUdFIjoiZG9ja2VyLmlvL3B1bmVldDIxNDcva2FuaWtvLWNoYWlucyJ9LCJlbnZpcm9ubWVudCI6eyJhbm5vdGF0aW9ucyI6eyJwaXBlbGluZS50ZWt0b24uZGV2L3JlbGVhc2UiOiJjODAyMDY5In0sImxhYmVscyI6eyJhcHAua3ViZXJuZXRlcy5pby9tYW5hZ2VkLWJ5IjoidGVrdG9uLXBpcGVsaW5lcyIsInRla3Rvbi5kZXYvdGFzayI6Imthbmlrby1jaGFpbnMifX19LCJidWlsZENvbmZpZyI6eyJzdGVwcyI6W3siZW50cnlQb2ludCI6InNldCAtZVxuZWNobyBcIkZST00gYWxwaW5lQHNoYTI1Njo2OWU3MGE3OWYyZDQxYWI1ZDYzN2RlOThjMWUwYjA1NTIwNmJhNDBhODE0NWU3YmRkYjU1Y2NjMDRlMTNjZjhmXCIgfCB0ZWUgLi9Eb2NrZXJmaWxlXG4iLCJhcmd1bWVudHMiOm51bGwsImVudmlyb25tZW50Ijp7ImNvbnRhaW5lciI6ImFkZC1kb2NrZXJmaWxlIiwiaW1hZ2UiOiJkb2NrZXIuaW8vbGlicmFyeS9iYXNoQHNoYTI1NjoxZWEzMGQ5YjY1Nzk3ZmJhZTQ3ODdmNjE4ODc5NmU3MTg5MzcxMDE5MDMxOTU4YTE2NzQyM2QzNDdkMzJlYWRhIn0sImFubm90YXRpb25zIjpudWxsfSx7ImVudHJ5UG9pbnQiOiIiLCJhcmd1bWVudHMiOlsiIiwiLS1kb2NrZXJmaWxlPS4vRG9ja2VyZmlsZSIsIi0tY29udGV4dD0vd29ya3NwYWNlL3NvdXJjZS8uLyIsIi0tZGVzdGluYXRpb249ZG9ja2VyLmlvL3B1bmVldDIxNDcva2FuaWtvLWNoYWlucyIsIi0tZGlnZXN0LWZpbGU9L3Rla3Rvbi9yZXN1bHRzL0lNQUdFX0RJR0VTVCJdLCJlbnZpcm9ubWVudCI6eyJjb250YWluZXIiOiJidWlsZC1hbmQtcHVzaCIsImltYWdlIjoiZ2NyLmlvL2thbmlrby1wcm9qZWN0L2V4ZWN1dG9yQHNoYTI1NjpjNjE2NjcxN2Y3ZmUwYjdkYTQ0OTA4Yzk4NjEzN2VjZmVhYjIxZjMxZWMzOTkyZjZlMTI4ZmZmOGE5NGJlOGE1In0sImFubm90YXRpb25zIjpudWxsfSx7ImVudHJ5UG9pbnQiOiJzZXQgLWVcbmVjaG8gZG9ja2VyLmlvL3B1bmVldDIxNDcva2FuaWtvLWNoYWlucyB8IHRlZSAvdGVrdG9uL3Jlc3VsdHMvSU1BR0VfVVJMXG4iLCJhcmd1bWVudHMiOm51bGwsImVudmlyb25tZW50Ijp7ImNvbnRhaW5lciI6IndyaXRlLXVybCIsImltYWdlIjoiZG9ja2VyLmlvL2xpYnJhcnkvYmFzaEBzaGEyNTY6MWVhMzBkOWI2NTc5N2ZiYWU0Nzg3ZjYxODg3OTZlNzE4OTM3MTAxOTAzMTk1OGExNjc0MjNkMzQ3ZDMyZWFkYSJ9LCJhbm5vdGF0aW9ucyI6bnVsbH1dfSwibWV0YWRhdGEiOnsiYnVpbGRTdGFydGVkT24iOiIyMDIzLTA3LTE4VDA4OjM4OjU0WiIsImJ1aWxkRmluaXNoZWRPbiI6IjIwMjMtMDctMThUMDg6Mzk6MTNaIiwiY29tcGxldGVuZXNzIjp7InBhcmFtZXRlcnMiOmZhbHNlLCJlbnZpcm9ubWVudCI6ZmFsc2UsIm1hdGVyaWFscyI6ZmFsc2V9LCJyZXByb2R1Y2libGUiOmZhbHNlfSwibWF0ZXJpYWxzIjpbeyJ1cmkiOiJkb2NrZXIuaW8vbGlicmFyeS9iYXNoIiwiZGlnZXN0Ijp7InNoYTI1NiI6IjFlYTMwZDliNjU3OTdmYmFlNDc4N2Y2MTg4Nzk2ZTcxODkzNzEwMTkwMzE5NThhMTY3NDIzZDM0N2QzMmVhZGEifX0seyJ1cmkiOiJnY3IuaW8va2FuaWtvLXByb2plY3QvZXhlY3V0b3IiLCJkaWdlc3QiOnsic2hhMjU2IjoiYzYxNjY3MTdmN2ZlMGI3ZGE0NDkwOGM5ODYxMzdlY2ZlYWIyMWYzMWVjMzk5MmY2ZTEyOGZmZjhhOTRiZThhNSJ9fV19fQ==","signatures":[{"keyid":"SHA256:wvNLyVMa1zxAWD9ZjvKanoCuukphRbKYdLM24TEEAj0","sig":"MEYCIQDsMLBOWKZKDBiiVJOz4ZQbPTKfQwhdBgsbVupJlvlN+gIhAPxMbCfjKGSl1ity9RS9/UMXRcI5QtkCH+LX6t4V5/Ft"}]}

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Metrics

Tekton Chains exposes standard Knative Controller metrics. These metrics are served on /metrics on the Tekton Chains controller Pod. These are exposed on port :9090 by default.

Metric collectors like Prometheus and OpenTelemetry can be used to collect these metrics. See Knative - Collecting Metrics for more details.

Chains Controller Metrics

The following chains metrics are also available at tekton-chains-metrics service on port 9090.

Name	Type	Description
`watcher_pipelinerun_sign_created_total`	Counter	Total number of signed messages for pipelineruns
`watcher_pipelinerun_payload_uploaded_total`	Counter	Total number of uploaded payloads for pipelineruns
`watcher_pipelinerun_payload_stored_total`	Counter	Total number of stored payloads for pipelineruns
`watcher_pipelinerun_marked_signed_total`	Counter	Total number of objects marked as signed for pipelineruns
`watcher_pipelinerun_signing_failures_total`	Counter	Total number of PipelineRun signing failures
`watcher_taskrun_sign_created_total`	Counter	Total number of signed messages for taskruns
`watcher_taskrun_payload_uploaded_total`	Counter	Total number of uploaded payloads for taskruns
`watcher_taskrun_payload_stored_total`	Counter	Total number of stored payloads for taskruns
`watcher_taskrun_marked_signed_total`	Counter	Total number of objects marked as signed for taskruns
`watcher_taskrun_signing_failures_total`	Counter	Total number of TaskRun signing failures

To access the chains metrics, use the following commands:

kubectl port-forward -n tekton-chains service/tekton-chains-metrics 9090

And then check that changes have been applied to metrics coming from http://127.0.0.1:9090/metrics

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Performance

Tekton Chains exposes a few parameters that can be used to fine tune the controllers execution to improve its performance as needed.

The controller accepts the following parameters:

--threads-per-controller controls the number of concurrent threads the Chains controller processes. The default value is 2.

--kube-api-burst controle the maximum burst for throttle. The default value is 10.

--kube-api-qps controles the maximum QPS to the server from the client. The default value is 5.

Modify the Deployment to use those parameters, for example:

spec:
    template:
        spec:
            containers:
                - image: ghcr.io/tektoncd/chains/controller-92006fd957c0afd31de6a40b3e33b39f:v0.26.2
                  args:
                    - --threads-per-controller=32
                    - --kube-api-burst=100
                    - --kube-api-qps=50

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

v1 client changes proposal - storage refactoring

With v1 approaching, I want to take a moment to look at changes we want to make to the existing client libraries to better set us up for long term maintenance.

We already know that we want to reduce the external library surface of chains. But to do this, we need to define better interfaces between components that we expect external clients to use.

Today, I think a lot of the codebase’s complexity has come from a few places:

Storage libraries

Each storage type needs different pieces of data - i.e. Grafeas and OCI need to distinguish image signatures and attestation formats, and some clients need the original object to extract out information like GVKs, names, namespaces, etc. This has led to a organic growth of the chains libraries to pass the different types of data around, and a lot of typecasting and other generic object tricks.

Looking at the storage interfaces, I think this data roughly boils down to:
- The original Tekton object
- The formatted data object
- The signed payload + signature (with optional cert information)
Unlike when chains first started, we now have another useful tool available to us: generics. I think we can use this to create clearer interfaces.
Dependence on the config package

tkn depends on the chains server config, but it probably shouldn’t. We should aim to have better ways to initialize clients for others to use.

Good news, I don’t think we’re far off, but we should make some changes

Signables

At it’s core, Chains is basically an ETL pipeline. We Extract artifacts from run objects, Transform and sign them, then Load them into storage.

type Signable[T any] interface {
	Extract(ctx context.Context, obj objects.TektonObject) []T{}
}

Payloaders

I think payloaders are mostly in a good place, though we can introduce generics to start creating stricter type relationships between Signables and Payloaders.

type Payloader[Input any, Output BinaryMarshaler] interface {
	CreatePayload(ctx context.Context, in Input) (Output, error)
}

tl;dr: Some type comes in, some type comes out.

BinaryMarshaler comes from the encoding package, but basically all we’re aiming for here is to make sure we can get a []byte for signing. For existing payload types, this may mean we need to wrap external types for this functionality.

Signers

Signers are mostly in a good spot, though we should probably just embrace []byte instead of typecasting between string for cert details.

type Signer interface {
	signature.SignerVerifier
	Cert() []byte
	Chain() []byte
}

Storers

Now that we have all the other pieces defined, we can now have stricter typing for storing:

type Storer[Input any, Output any] interface {
	Store(ctx context.Context, req *StoreRequest) (*StoreResponse, error)
}

type StoreRequest[Input any, Output any] struct {
    Object objects.TektonObject
    Artifact Input
    Payload Output
    Bundle *signing.Bundle
}

type StoreResponse struct {
    // Some identifier for what we uploaded to reference later?
    ID string
}

type Bundle struct {
	Content   []byte
	Signature []byte
	Cert      []byte
	Chain     []byte
}

While the StoreRequest struct may not be necessary, it has a nice RPC-like quality in that it will make it easier to add/remove fields in the future.

Attestors

To put it all together, we can add a new type: Attestor. This is effectively just a wrapper type around all of the other interfaces that binds the generic types together. Because things are strictly typed, we’ll know at compile what clients are compatible with each other.

TBD if we expose this at all - it may remain an internal implementation detail of chains. This is what we will generate from the Chains server config.

type Attestor[Input, Output] struct {
    payloader Payloader[Input, Output]
    signer Signer
    storer Storer[Input, Output]
}

What this looks like in practice:

OCI Simple Signing:

attestor := &Attestor[name.Digest, simple.SimpleContainerImage]{
    payloader: NewSimpleSigningPayloader(),
    signer: x509Signer,
    storer: NewSimpleOCIStorage(),
}

SLSA:

attestor := &Attestor[TektonObject, *intoto.Statement]{
    payloader: NewSLSAPayloader(),
    signer: fulcio,
    storer: NewGCSStorage(),
}

Grafeas:

attestor := &Attestor[TektonObject, *Occurrence]{
    payloader: NewGrafeasPayloader(),
    signer: kmsSigner,
    storer: NewGrafeasClient(),
}

Final thoughts

If all goes well, this should have 0 impact on typical consumer usage of chains - these should all be internal refactors with no change in behavior. If our e2e start failing, we’ve done something wrong.

Tekton – Supply Chain Security

Docs:

Authentication for Chains

Authenticating to an OCI Registry

First we’ll cover creating the credentials.

Create a Secret based on existing credentials

Create a Secret by providing credentials on the command line

Setup credentials using the pod

Setup credentials using the service account

Authenticating to Fulcio for Keyless signing

Specifying a custom Fulcio endpoint

Specifying Spiffe as authentication provider

Docs:

Chains Configuration

Chains Configuration

TaskRun Configuration

PipelineRun Configuration

OCI Configuration

KMS Configuration

Storage Configuration

docstore

MongoDB

Grafeas

In-toto Configuration

Sigstore Features Configuration

Transparency Log

Keyless Signing with Fulcio

KMS OIDC and Spire Configuration

Visual Guide: ConfigMap Configuration Options

Namespaces Restrictions in Chains Controller

Usage

Example

Docs:

Signing Artifacts

Signing Secrets

x509

Cosign

Generate cosign Keypair

KMS

Authentication

Troubleshooting

Docs:

SLSA Provenance

Goal

Glossary

How does Tekton Chains work?

How to configure Tekton Chains

How to configure a Task or Pipeline

Type Hinting

Input Artifacts

Git Results

*ARTIFACT_INPUTS

Output Artifacts

*IMAGE_URL / *IMAGE_DIGEST

IMAGES

*ARTIFACT_URI / *ARTIFACT_DIGEST

*ARTIFACT_OUTPUTS

v2alpha4 formatter

Besides inputs/outputs

Docs:

Sigstore

Transparency Log Support

Enabling Transparency Log Support

Keyless Signing Mode

Enabling Keyless Signing Mode

Better Way Of Navigating in Transparency Log with rekor-search-ui

Docs:

Experimental Features

PubSub Storage Backend Support

Kafka

Docs:

Chains Getting Started Tutorial

x509

Configuring Tekton Chains

What you just created

Docs:

Chains Signed Provenance Tutorial

Prerequisites

Generate a Key Pair

Set up Authentication

`*ARTIFACT_INPUTS`

`IMAGE_URL` / `IMAGE_DIGEST`

`IMAGES`

`ARTIFACT_URI` / `ARTIFACT_DIGEST`

`*ARTIFACT_OUTPUTS`

`v2alpha4` formatter

In this tutorial, we will be running Chains Signed Provenance Tutorial using `kms` solution by integrating Tekton Chains with Hashicorp Vault

Chains Signed Provenance Tutorial using `kms` solution by integrating with Hashicorp Vault