Leveling Up Your CI/CD Pipeline with Security Best Practices

In my previous post, I walked through the foundational steps of a solid CI/CD pipeline including the essential tasks to build and deploy code. Getting your code to production quickly is good, but a great pipeline also makes sure that what you’re shipping is secure, verified, and compliant.

By weaving security into each stage of your pipeline, you can catch issues early, reduce risk, and build trust in every release. Here, I'll share some of what I learned from building secure pipelines for tens of thousands of repositories.

Security Scans

Security scanning is one of the easiest and most impactful improvements you can make. These scans will find issues and vulnerabilities in your code that you can then either fix or improve. There are many different types of scans, each with their own focus. Here are the most important types to consider.

Software Composition Analysis

Software composition analysis (SCA) focuses on the open source dependencies included in your project. These scanners can identify dependencies and their vulnerabilities, giving insight into areas that might need updates. In addition, these tools can create a software bill of materials (SBOM), which will be very useful for the Security Gates section.

In my experience, I've implemented this type of scan in two ways: directly scanning the application after installing dependencies and scanning the built container image. The container image will give a more complete scan, including vulnerabilities coming from any base image in use. A scan earlier in the pipeline will be more focused, giving a high confidence that any findings are a direct result of the application.

Trivy is a great open source scanning tool in this area.

Detecting Secrets

Sometimes developers accidentally add secrets to their GitHub repository. I use an open source tool called Detect Secrets to help discover when this has happened as part of the pipeline. It isn't ideal, because if this tool finds anything then the secret has already leaked, but it can identify the problem so that it can be resolved immediately.

Static Analysis

Static analysis is a scan of your source code that can recommend improvements as well as find vulnerabilities present. This should be run before the artifact is built, and can result in a variety of recommendations. The results of this scan can be output in the SARIF format, which is a machine readable output that can be persisted for analysis.

Dynamic Analysis

Dynamic analysis is another scanning option, but this scan runs against your deployed application. It does not have access to the source code of the application, and instead will attempt to find vulnerabilities by probing the actual application. Therefore, dynamic analysis should be run after deployment only in non-production environments.

Zed Attack Proxy is an open source option for running dynamic security testing.

Signing Build Artifacts

All build artifacts should be signed with a key that is only available to your CI system. This will prove that any build artifact with this signature was created by your secure build system, and included all of the security testing that is expected.

While all build artifacts should be signed, if you're creating container images cosign is a great open source tool for signing the image.

Signature Verification

While not part of the CI/CD pipeline itself, your deployment environments should be configured to validate the signature of everything deployed before it is run. Without this step, signing the images is a nice addition but not enforced for protection.

Audit Readiness

With all of these new security tasks added to your pipeline, you have a lot of data about the applications being built. This includes all of the dependencies, vulnerability data, versions, and information about the specific changes made. Saving this data can make audits run more smoothly, giving confidence that vulnerabilities are taken seriously and quickly resolved.

Your results from SCA will be among the most valuable. These should result in an SBOM, and using a standard format will make further analysis more straightforward. CycloneDX is a popular format for this requirement.

With this information saved, you can quickly show your developers which dependencies are responsible for vulnerabilities, even if they are not direct dependencies.

Security Gates

The final improvement to your pipelines is to apply policies to prevent risky changes from deploying. For example, if you would like to block all releases with a critical vulnerability, you can do so by running a policy task after the scans are completed. A great open source tool for applying policy checks is the Open Policy Agent.

These additions take your pipeline from simple CI/CD to an advanced pipeline ready to securely deploy changes.