Update: GitLab 12.1, released on Jul 22, 2019, added a built-in Let’s Encrypt integration. For GitLab, you should of course use the built-in integration. You can use the method shown in this article to automate certificate generation and provisioning in other settings.
Let’s Encrypt is a CA (Certificate Authority) that provides free, automated domain-validated Certificates. Since it’s launch in April 2016 it has become a baseline for what CAs should provide and the go-to address for automated public certificates.
Let’s Encrypt is an extremely useful tool to securely provide continuously delivered services. For automation, Let’s Encrypt uses a component called Certbot, which is intended to be installed on the certificate’s target. Certbot also provides a manual mode for environments where it cannot be executed on, for instance in a shared hosting environment or with GitHub or GitLab Pages.
This article shows how to automate Certbot’s manual mode by using GitLab Pages as an example. The process itself is generic.
We need three steps to automate certificate creation and renewal:
We’re using OpenSSL to check for a valid certificate. The following snippet validates that a target domain has a certificate that does not expire in a set time. Let’s Encrypt certificates expire after 60 days. We’re updating the certificate 14 days before that to leave enough time to react to unforeseen issues before the certificate actually becomes invalid.
If the certificate is still valid, no further action is necessary.
if true | openssl s_client -servername "${DOMAIN}" -connect "${DOMAIN}:443" 2>/dev/null \
| openssl x509 -noout -checkend $((UPDATE_AHEAD_DAYS*24*3600))
then
echo "Certificate is still valid, nothing to do"
exit 0
fi
The certificate will be created or renewed by Certbot. In manual mode, certbot needs a number of arguments:
--force-renewal: tells Certbot to renew regardless whether the existing certificate is still valid. This is important to renew certificates ahead of time.--manual: enables manual mode--preferred-challenges http: Set the preferred challenge to http; To fulfil the challenge, a special file has to be served via HTTP.--agree-tos --no-eff-email --manual-public-ip-logging-ok: Shut up Certbot’s interactive prompts--manual-auth-hook "${SCRIPT_DIR}/auth-hook.sh" configures which script should perform the Let’s Encrypt challenge-d "${DOMAIN}" -m "${EMAIL}" provide domain and email for the certificate--config-dir "${CONFIG_DIR}" --work-dir "${WORK_DIR}" --logs-dir "${LOGS_DIR}" working directories for Certbot. Certbot by default uses directories under /etc, which are writable only by root. To run Certbot as a non-privileged user, we need to provide these directories.certbot certonly \
--force-renewal \
--manual \
--preferred-challenges http \
--agree-tos \
--no-eff-email \
--manual-public-ip-logging-ok \
--manual-auth-hook "${SCRIPT_DIR}/auth-hook.sh" \
-d "${DOMAIN}" \
-m "${EMAIL}" \
--config-dir "${CONFIG_DIR}" \
--work-dir "${WORK_DIR}" \
--logs-dir "${LOGS_DIR}"
The snippet above references an authentication hook. As we need to implement the http challenge, we need to put the challenge file in our GitLab page. We do this by adding the challenge file to our Git repository and pushing.
The CI build for the GitLab Pages site will pick up and publish the change. We block the authentication hook until that happens by polling the URL the challenge file will appear at.
# Path to the challenge file
CHALLENGE_PATH=".well-known/acme-challenge/${CERTBOT_TOKEN}"
# File system location of the challenge file
# This location works for Hugo (https://gohugo.io/)
CHALLENGE_FILE="./static/${CHALLENGE_PATH}"
# URL of the challenge file
CHALLENGE_URL="https://${CERTBOT_DOMAIN}/${CHALLENGE_PATH}"
cd "$LOCAL_REPO"
echo "Creating challenge file ${CHALLENGE_FILE}"
mkdir -p "$(dirname "${CHALLENGE_FILE}")"
echo "${CERTBOT_VALIDATION}" > "${CHALLENGE_FILE}"
echo "Adding challenge file ${CHALLENGE_FILE} to Git"
git add "${CHALLENGE_FILE}"
git commit -m "letsencrypt challenge added"
git push
echo "Waiting for challenge to become online at ${CHALLENGE_URL}"
while ! curl -Lsf "${CHALLENGE_URL}" > /dev/null
do
sleep 20
done
In case of an error, the CI build will eventually time out and get terminated. That’s good enough for our purpose.
As soon as the authentication hook is done, we have to install our new certificate. We can use the GitLab API to install the new certificate for our domain.
curl \
--request PUT \
--header "PRIVATE-TOKEN: ${GITLAB_API_TOKEN}" \
--form "certificate=@./config/live/${DOMAIN}/fullchain.pem" \
--form "key=@./config/live/${DOMAIN}/privkey.pem" \
https://gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/pages/domains/${DOMAIN}
We now have a script that automates Let’s Encrypt manual mode. All we need in addition to that is a CI solution that continuously runs our script and provides notifications.
In this example, I’m using GitLab CI, which only requires a small build YAML.
You can find the complete automation and CI script here: gitlab.com/az82/certbot-az82.de
Let’s Encrypt can be a bit hard to test. For testing against Let’s Encrypt itself, you should use it’s staging environment. The most common pitfall are the rate limits for Let’s Encrypt production and staging. If you run into rate limit issues consider running your own Let’s Encrypt compatible CA for testing. Let’s Encrypt provides a testing CA called Pebble and also it’s full-blown CA Boulder.
This article provides a comprehensive solution for automatically securing GitLab Pages with Let’s Encrypt. The general process can be used for any use case for which there’s a way to automate domain validation and an API that allows automated installation of certificates.
This makes Let’s Encrypt a valuable tool for securing continuously delivered services.