Overview
If you are managing application development pipelines to deliver your software, one key security control needs to be in-place. You need to make sure that your secrets are protected. Secrets are, simply put, credentials that you need to protect because of the privileged capabilities of that credential. For example:
Username and password combinations
API Keys
JSON Web Tokens
Private Keys
Developers often accidentally save these credentials to the repository. Once these credentials or secrets are added to the repository, clean-up can be painful and require you to change the credentials. A better long-term approach is to use “pre-commit checks” to avoid storage of the secrets. That topic will be covered in another article. This blog focuses on how to scan existing repositories for stored secrets in a Gitlab pipeline.
Identify or create your repository
If you’d like to follow along, you can click this link for the public repository. https://gitlab.com/cmg_public/secrets_scanning_test
The first step is to identify your target repository. If you are getting started and don’t have one, simply create a new project in Gitlab. From the menu click the “Projects>Create New Project>Create Blank Project”.
Add the “Project Name”, “Project Slug”, “Description” and select your visibility level. It will be private by default. Then click “Create Project”.
Add Supporting Configuration Files
Click the “New File” button, this will take you to the “Web IDE”.
Then click the “New Directory” button, name it “.gitlab”.
Then create a new file in that directory, named “.gitlab/secret-detection-ruleset.toml”.
Once you have created the folder and the file you can “Commit” the changes to the “Main” branch.
This will open the “secret-detection-ruleset.toml” file for editing.
Copy and paste the following text into the file and commit the changes:
[secrets]
description = 'CMG DevSecOps Crash Course, secrets custom rules configuration'
[[secrets.passthrough]]
type = "raw"
target = "gitleaks.toml"
value = """\
title = "gitleaks config"
# Add regexes to the regex table
[[rules]]
description = "Test for Raw Custom Rulesets"
regex = '''Custom Raw Ruleset T[est]{3}'''
[[rules]]
description = "CMG Generic JWT"
regex = '''(?i)(jwt|jwt_token|secret|bearer)'''
tags = ["key", "JWT", "generic"]
[[rules]]
description = "CMG Custom Generic Password"
regex = '''(?i)(password|passw)'''
tags = ["key", "Custom Password", "generic"]
"""
You will notice that I have added a few custom rules. These will be used to identify some types of secrets that are not enabled by default. You can create as many rules as you like to find the patterns that typically appear in your developer’s code.
The custom changes are:
“CMG Generic JWT”: This regular expression (REGEX) looks for common terms stored inside of a file that equate to a JSON Web Token (“JWT”). Terms such as “JWT”, “jwt_token”, “secret”, “bearer”. The regex is case insensitive.
[[rules]]
description = "CMG Generic JWT"
regex = '''(?i)(jwt|jwt_token|secret|bearer)'''
tags = ["key", "JWT", "generic"]
“CMG Custom Generic Password”: This regular expression looks for common derivatives of the word “password” stored inside of a file. This regex is also case insensitive.
[[rules]]
description = "CMG Custom Generic Password"
regex = '''(?i)(password|passw)'''
tags = ["key", "Custom Password", "generic"]
Your repository should now contain one file and one folder. Next, we will need to create the pipeline to run against our test repository. On the repository screen of your project click the “Add file” button and select “New file”.
This will bring up the blank editor. From here you will want to select “.gitlab-ci.yml” from the “Select a template type” drop down list. Name your file “.gitlab-ci.yml”
In the body of the newly created pipeline file copy and paste the text below. Once you have completed the paste operation, click “Commit”.
# This is a testing pipeline for validating Secrets Scanning functionality
stages:
- test
include:
- template: Security/Secret-Detection.gitlab-ci.yml
We first set our variables to enable historical scanning and a depth of 100 commits. We then create a necessary stage called “test” and include the path to the Gitlab secrets detection template. This template will be used to scan for secrets based on the default settings and any custom regex strings, like the those that we added in the previous steps. At this point we are ready to run the pipeline; however, we don’t have anything to scan in the repository. In the next step we will create some fake secrets for testing the configuration.
Create fake secrets for testing
For this test we will create the following files in the repository:
Fake.json: We can use this to test for JWT. You can copy and paste the information below into the file you create in the repository.
{
"username": "SecurityBooBoo",
"password": "Oh-Y3aH_CmG_2022"
}
FakePrivKey.key: We can use this file to test for Private Keys
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
NhAAAAAwEAAQAAAYEAvO5F4B+lb8YYXB6gCgqxlmuNLGwTC58f96TbYQxUV/yzZkBS1hIp
H0WOba5ZKeULWh7B5KPrqpK3yfYitThw73wX+UQuBN8FCyFw41/dB5brApDLLdW+aYt9M2
KU3XcKDfij3Rmw4dWvihvxeOWjp1WTNZj480O1+CB+oTOWdElWp7fmhw912itreRQcJso7
rFnfgfvdCAJoRMwYtadKmm3f9I1UvUxaaBiy45vwo7ZbnMUBnEOMhoA9zqZDii7GVj80t+
1WfbrMy/WRzibspxs4cMpUa+K9FLXoms1I35QQ+VYtFMQehjsXkZEMprfewCF0ZzmNAavL
xoIW+ybzCZNeH8afWNQ0TmO4M9sPmhrorfenKU3mfo1yxW5P8WIyl4ee7aSYPGMQdqh0nV
qAuyc2XU6YF2Qz0HpSz9qq4gQVKNS/a10QbS7KiyGtGCFyRmCV3UGHqZGo4QvGVGA20qwV
DN+o+KnLjvrYcklPBogGtxUvrCcU34n7tFE4LQjHAAAFiOcpTkznKU5MAAAAB3NzaC1yc2
EAAAGBALzuReAfpW/GGFweoAoKsZZrjSxsEwufH/ek22EMVFf8s2ZAUtYSKR9Fjm2uWSnl
C1oeweSj66qSt8n2IrU4cO98F/lELgTfBQshcONf3QeW6wKQyy3VvmmLfTNilN13Cg34o9
0ZsOHVr4ob8Xjlo6dVkzWY+PNDtfggfqEzlnRJVqe35ocPddora3kUHCbKO6xZ34H73QgC
aETMGLWnSppt3/SNVL1MWmgYsuOb8KO2W5zFAZxDjIaAPc6mQ4ouxlY/NLftVn26zMv1kc
4m7KcbOHDKVGvivRS16JrNSN+UEPlWLRTEHoY7F5GRDKa33sAhdGc5jQGry8aCFvsm8wmT
Xh/Gn1jUNE5juDPbD5oa6K33pylN5n6NcsVuT/FiMpeHnu2kmDxjEHaodJ1agLsnNl1OmB
dkM9B6Us/aquIEFSjUv2tdEG0uyoshrRghckZgld1Bh6mRqOELxlRgNtKsFQzfqPipy476
2HJJTwaIBrcVL6wnFN+J+7RROC0IxwAAAAMBAAEAAAGADHxGm+Abg754n6Xad69rOwCSjM
v7mjoC18/KO6if7kyh4nD/yGvc0dc76V2rQMyFKoh70ctPaK9Xe/5LHuTC+eCeiPeLfwDq
CWlFV5FfPwAnOb0t4DKO6dSxCnNKWTRjsraqxZLMELCZcCwWkiHC5e0O1Gzujsz7upETLT
4GhFrQYjcSAfzwkeFqsc61aY7V0LcDwUhOvBfEoj9GpsKOeJQoR7YLUpM4Kkbvk4EbrwX/
GeYfDB+eqsjQzNZKAljC346B9J/7krlANZYd/2PtsHqzJrnws/08jSjkzit12RdgP5UPWK
L1Z0et8In4vXnKUspEAagB4pGrTLdgec7s14s/RMPqkUuUv016hGzW8k14YRG9Z1e91OsA
D/kJsyLButT262kemAC05p4LUrqaAjAzDqmASSMEXfR7BWo6geVfLkxv5yDExFTcOPATNQ
tf9R62h9k+W8wHUfrxfDo8fshgOLtrSsux+ROg1tNlOsvrNrmdpPH8tZD5RQPss28xAAAA
wAZk8IIU34nLTWUspzDyyiegyAXD5kO1r6dxZoAiudaCqGjNbRo82nzrg1ux8ZohnVbOQV
t6qQkDjDQB37airXVo4/j/s7H0HthSctlEPe4EYWaouhmPk3v45LV8z8aq3QZY5chyTGB5
N17I5QgoSBHEM42pf+gWTmNUX0N4bjwRL/a+HE7PnGC5ax+FNPY+fL9fHpBtz791SogNak
fopIxmfSJJbTbBctw01F6eo67uikGOVxzk5MClJaP2jyLYFAAAAMEA6/ShdpzMCd5Oy5+O
DtDV0v8lF9YCxKl6+Vaw1iQtdafvgZuy1Msr645o39IVbUWKEwNJsi0uokyxvojcQOgNp1
hWZCVvkXXlBlTxKcxRlSBVj+6eB/nIgDLEcLKH7DYALfat+IBaEdtPiL4/+FiOmXrx/JWR
E43yoYDkC59usOZKHVl8uM4YgOw7mhFx5I1W6qX2rXoT3toEDITRygjuMjreegzhf/oSUF
T2IaGxNBpOf0yCBo6nCyFhk5f3g185AAAAwQDM+vwMD6/ToLy5roZoJ75CcVUNHMh3Sj/l
f2AEe3ldM2VTtoV/nVOli6/N8jJOhF7tRm6RlR1CjYKDPALMuRgwg83dXqtr+jxuwrcV6t
deYr48HVdTgNooJ29SmYSB+7xWX7flPS+8gtpx95FXKnNqL+NFNEWxERjfe2oLQ+2/R7Jb
XChNNNV04eIAWuCFhyl0wGBDALdo7biwsRJ/mYB7vkptfOSKTTP3zNd4qFKQsN3VBfPGmZ
KQCdxFay/Lp/8AAAATYW5hbHlzdEBweXRob25kZXYwMQ==
-----END OPENSSH PRIVATE KEY-----
FakeAPIKey.txt: We can use this to find and test for API Keys
Apikey=33219b16f39fc40d3a1be5e9e6f56abe
Once you complete creating your files your repository should look like this:
Running the test and observing the results
Each time you added and committed a file it triggered a build, by default. If you click on the “CI/CD”>” Pipelines” object on the left-hand pane of the web interface, you can see the results of each job. Click the “Run Pipeline” button on each screen. You will be prompted for other options which can be ignored.
After a few seconds the pipeline job will begin. Once it turns blue you can click the “secret_detection” stage and watch the output.
If everything worked as planned, you should see similar text.
Download the results
After a few minutes the results will be available for download for the associated job. Click the “3 Button Ellipses” on the right side of the job and select the “secret_detection.secret_detection” file. It will download as JSON. You can then open it in your favorite text editor or online JSON formatting tool.
Review the results
If you have GitLab Ultimate
If you have Gitlab Ultimate, you don’t need to mess with the “pesky” JSON output. You can enable “Secrets Scanning” and everything will show up on the Vulnerabilities dashboard. This alone makes “Ultimate” version worth the money.
Click the “Security & Compliance” object on the left-hand menu and select “Vulnerability Dashboard”, then filter by “Tool”. You will see a very simple report that outlines all your vulnerabilities that can then be assigned to a developer as an issue. If you click through the report, you will see exactly where and how the “secret” was detected including the specific evidence.
If you click through the report, you will see exactly where and how the “secret” was detected including the specific evidence. Click the items found to explore.
This example found our “Fake.Json” file. On the details page click the “File: Fake.json:3” hyperlink.
The tool will direct you to the specific file and credential found:
Summary
In summary you can see that this is straightforward and valuable from a security perspective. Once your secrets are committed it is a pain in the butt to remediate. This is one approach but does not negate the need for “pre-commit scanning” or using “vaulting” for credential and secret storage. If you need help with your DevSecOps implementation, please feel free to contact us.