Overcoming Corporate Distrust

When your enterprise environment intercepts and inspects HTTPS traffic at a proxy, that proxy is operating as a man-in-the-middle (MITM). This is exactly the situation that HTTPS is designed to protect against. To get your applications to play nice with a proxy like ZScaler, you have to force them to trust its root certificate, which is not part of the globally accepted set of root certificates. System administrators or your proxy client (e.g. Zscaler Client Connector) generally ensure that your system’s trust bundle, used by applications such as your browser and the OS itself, includes these root certificates.

What is really happening. Image source

The problem

The presence of your company’s root certificates allows you to perform tasks like browsing the internet. Unfortunately, not all applications use your system’s trust bundle by default, resulting in errors like this:

curl: (60) SSL certificate problem: self-signed certificate in certificate chain

In fact, most scripts (Python, Node.js) do not integrate with the system trust bundle. Instead, they ship with their own bundles. I have spent more time than I would care to admit dealing with related issues.

This post lists some ways on how to create a trust bundle — preferably by extracting your system’s trust bundle into a PEM file — and how to get different applications to trust it by setting environment variables.

Step 1: Creating a trust bundle

The first step is to create a trust bundle that includes your corporate root certificates. I prefer to do this by extracting your system’s trust bundle into a PEM file. Depending on your environment, you may want to include public root certificates as well. It almost certainly does not hurt to include them.

Every operating system has its own way of storing and accessing the system trust bundle.

On MacOS

On MacOS, the system’s trust bundle is stored on your system keychain. You can extract it using the following convenient commands:

security find-certificate -a -p /System/Library/Keychains/SystemRootCertificates.keychain > ~/.bundle.pem
security find-certificate -a -p /System/Library/Keychains/System.keychain >> ~/.bundle.pem

It appears to me that the SystemRootCertificates.keychain includes the certificates that ship with macOS, and the System.keychain includes additional certificates added by administrators or enterprise tools.

On Linux

Linux distributions generally store your system’s trust bundle already in a PEM file. All you have to do is figure out its location. This location depends on your specific distribution.

• On Debian-based systems (Debian, Ubuntu, Mint, …), it is located at /etc/ssl/certs/ca-certificates.crt.
• On Red Hat-based systems (RHEL, CentOS, Fedora, Amazon Linux), it is located at /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt.

Your OS generally compiles this bundle from a collection of PEM files that are stored in a different location. On Debian/Ubuntu, for example, certificates are stored in /usr/local/share/ca-certificates/, and compiled into a bundle by the update-ca-certificates command. On Red Hat-based systems, certificates are stored in /etc/pki/ca-trust/source/anchors/ and compiled using update-ca-trust.

On Windows and other systems

Like MacOS, Windows stores your system’s trust bundle on your system keychain. Unlike MacOS, it does not provide convenient commands for extracting it into a PEM file.

Some combination of PowerShell commands and/or calls to certutil allow you to extract the system trust bundle. This is non-trivial, and corporate environments often restrict any type of shell access on Windows machines anyway. So on Windows, instead of extracting the system trust bundle, I tend to construct “my own” instead, starting from Python’s default certifi trust bundle and appending additional root certificates to it.

Unlike MacOS, Windows does not make it easy to extract your system trust bundle.

You can get the certifi trust bundle from its GitHub repository:

curl https://raw.githubusercontent.com/certifi/python-certifi/refs/heads/master/certifi/cacert.pem > ~/.bundle.pem

To get the additional certificates to trust, that is, the certificates used by the corporate proxy, you have several options:

• You can ask your admin for it;
• You can often find it hosted on Sharepoint or Confluence;
• You can also fetch it from the “man-in-the-middle” yourself.

I prefer the last option. If openssl is installed, you can use it to inspect served certificates:

hostname=google.com # any other hostname serving the untrusted certificate
openssl s_client -showcerts -connect "$hostname":443 </dev/null

You can craft a script to create a PEM file from this output automatically if you like.

If OpenSSL is not available, you can inspect the certificate served by the proxy from your browser’s developer tools or navigate to your PC’s trust store and export it from there. You can find some more detailed instructions on how to do this here.

Step 2: Get applications to use the trust bundle

Now that we have a trust bundle, we need to get applications to use it. There are many ways to do this, and they unfortunately depend on the application you are dealing with. The most common and universal way to configure this is by setting environment variables.

Depending on the application you are dealing with, you may need to set one or more environment variables. This is a list of the most common ones:

• SSL_CERT_FILE / SSL_CERT_DIR - OpenSSL and most tools
• CURL_CA_BUNDLE - curl
• REQUESTS_CA_BUNDLE - Python requests
• NODE_EXTRA_CA_CERTS - Node.js
• CARGO_HTTP_CAINFO - cargo
• GIT_SSL_CAINFO - git

These more or less apply cross-platform. On UNIX systems, you can set them by running the following commands:

export SSL_CERT_FILE=~/.bundle.pem
export SSL_CERT_DIR=~/.bundle.pem
export CURL_CA_BUNDLE=~/.bundle.pem
export REQUESTS_CA_BUNDLE=~/.bundle.pem
export NODE_EXTRA_CA_CERTS=~/.bundle.pem
export CARGO_HTTP_CAINFO=~/.bundle.pem
export GIT_SSL_CAINFO=~/.bundle.pem

I shamelessly copied this list from the excellent httpjail

You can set these “permanently” by exporting them from your .bashrc, .zshrc or any other shell configuration file.

conffile=~/.bashrc
echo "export SSL_CERT_FILE=~/.bundle.pem" >> "$conffile"
echo "export SSL_CERT_DIR=~/.bundle.pem" >> "$conffile"
echo "export CURL_CA_BUNDLE=~/.bundle.pem" >> "$conffile"
echo "export REQUESTS_CA_BUNDLE=~/.bundle.pem" >> "$conffile"
echo "export NODE_EXTRA_CA_CERTS=~/.bundle.pem" >> "$conffile"
echo "export CARGO_HTTP_CAINFO=~/.bundle.pem" >> "$conffile"
echo "export GIT_SSL_CAINFO=~/.bundle.pem" >> "$conffile"

On Windows, you can set environment variables from the System UI panel, or you can create a shortcut which sets them locally.

More approaches for different applications can be found here:

Adding Custom Certificate to Application-Specific Trust Store

https://help.zscaler.com/zia/adding-custom-certificate-application-specific-trust-store

What about Docker?

Applications running inside a container (e.g. Docker) will use the trust bundle that is part of the container image. Essentially all container images come with a trust bundle, even the minimal ones like distroless.

Yes, indeed, this means that your perfect self-contained containerized applications still have a limited shelf life if they connect to the outside internet. Because even root certificates expire, you will have to rebuild at some point, even if the containerized application itself does not change.

To ensure that your containerized application trusts your man-in-the-middle, you will have to either:

• bake your custom bundle into the image or
• mount your custom bundle into the container, preferably shadowing the system trust bundle that is part of the image.

Which one is most appropriate depends on your use case, but usually mounting is preferred. You may additionally have to set some of the environment variables listed above, if you are using applications that do not use the system trust bundle like we discussed above.

Distroless images contain a trust bundle at /etc/ssl/certs/ca-certificates.crt

For an Ubuntu- or distroless-based image, this can look like this:

docker run -it --rm \
  -v $HOME/.bundle.pem:/etc/ssl/certs/ca-certificates.crt:ro \
  -e SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt \
  -e CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt \
  -e REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt \
  -e NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt \
  -e CARGO_HTTP_CAINFO=/etc/ssl/certs/ca-certificates.crt \
  -e GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt \
  ubuntu bash

The process of building an image is itself containerized, and may also require you to set similar settings. Docker Desktop generally uses your system’s trust bundle out of the box, but when you use podman on an OS that is not Linux, you will have to update the trust bundle of the (Linux) VM running your containers. You can do this using podman machine ssh commands.