Security and self-signed certificates: Configuring a secure Mesos cluster using SSL

Enabling SSL in Mesos requires the libevent-openssl library and its dependencies. It is the library by which all messages flowing through the Mesos cluster will be encrypted.

More information at https://github.com/libevent/libevent.

We can retrieve the pre-packaged libevent library and its dependencies via the local linux package manager.

Package manager:

• yum:

sudo yum install libevent-devel -y
sudo yum install openssl-devel -y

• apt:

sudo apt-get update
sudo apt-get install libssl-dev -y
sudo apt-get install libevent-dev -y

The next step is to download the Mesos repository and install its dependencies. Do not build the Mesos binary yet.

Instructions for downloading Mesos and installing its dependencies can be found here: https://mesos.apache.org/gettingstarted.

 When you reach the "Building Mesos" section, run the configure script like such:

../configure --enable-libevent --enable-ssl

Continue to build Mesos following the instructions as they are.

If you are running Mesos on a remote machine and wish to access the Mesos web UI from the client, it is best to bind the Mesos master to a routable IP address.

For this example we will use IP 192.168.1.172. (If you are running the browser on the same machine where you are starting the Mesos master, you can bind the Mesos master to your localhost: 127.0.0.1).

Using the keys we have created from Part 1 we will start a secure Mesos master. At this point, we should have the following keys:

• Certificate Authority: ca.pem
• Digital Certificate: localhost.pem
• Private Key: localhost-key.pem

Let's put those in the following folder /etc/mesos/pki.

The startup command for the Mesos master will be:

export LIBPROCESS_SSL_CA_DIR=/etc/mesos/pki 
export LIBPROCESS_SSL_ENABLED=true 
export LIBPROCESS_SSL_SUPPORT_DOWNGRADE=false 
export LIBPROCESS_SSL_KEY_FILE=/etc/mesos/pki/localhost-key.pem 
export LIBPROCESS_SSL_CERT_FILE=/etc/mesos/pki/localhost.pem 
sudo -E ./bin/mesos-master.sh --ip=192.168.1.172 --work_dir=/tmp

LIBPROCESS_SSL_ENABLED is a convenient way of turning off SSL communication between the master and agents, between the submitted Mesos framework and the Mesos cluster or on any connection to a Mesos http endpoint. When it is false, Mesos will ignore all other LIBPROCESS_SSL_* related arguments. At this point, the Mesos master is up and running without any Mesos agent. However, the Mesos web UI should now appear from your browser at https://192.168.1.172:5050.

You will probably see a warning on your browser saying this connection is unstrusted. Remember from Part 1 we said a home made self-signed certificate is not certified by a known Certificate Authority. In other words, the Certificate Authority we created is itself not certified by a known Certificate Authority.

The reality is that when you login to your favorite e-mail service (or any official secure website) the Certificate Authority used is itself certified by a higher Certificate Authority to form a tree. At the root of the tree is what we call the Root Certificate.

Since the Certificate Authority we generated is itself not verified, the browser is issuing a warning stating that the identification cannot be trusted.

In fact, the browser is detecting a couple of issues. It warns that the issuer certificate (Certificate Authority ca.pem) is unknown. It also warns that the certificate (localhost.pem) is only valid for localhost and 127.0.0.1. In my case, I have binded the Mesos master to 192.168.1.172 and not 127.0.0.1. Beware that in Part 1, we have generated the certificate for a service running on the localhost.

"192.168.1.172:5050 uses an invalid security certificate.

The certificate is not trusted because the issuer certificate is unknown.
The server might not be sending the appropriate intermediate certificates.
An additional root certificate may need to be imported.
The certificate is only valid for the following names:
localhost, localhost, localhost, 127.0.0.1, 127.0.0.1

(Error code: sec_error_unknown_issuer)"

Let's look at the content of the generated certificate:

openssl x509 -in localhost.pem -text | less

If you look for "X509v3 Subject Alternative Name" in the output, you should see:

X509v3 Subject Alternative Name:                 DNS:localhost

To generate a certificate for the service running on the routable IP, change the command (from Part 1) from:

cfssl gencert -ca ca.pem -ca-key ca-key.pem -hostname localhost csr.json | cfssljson -bare localhost

to:

cfssl gencert -ca ca.pem -ca-key ca-key.pem -hostname 192.168.1.172 csr.json | cfssljson -bare localhost

Now the output of

 openssl x509 -in localhost.pem -text | less 

should contain:

X509v3 Subject Alternative Name:
                IP Address:192.168.1.172

If you generate the certificate with the routable IP, you should only get the warning that the issuer certificate is unknown.

"192.168.1.172:5050 uses an invalid security certificate.

The certificate is not trusted because the issuer certificate is unknown.
The server might not be sending the appropriate intermediate certificates.
An additional root certificate may need to be imported.

(Error code: sec_error_unknown_issuer)"

If you proceed and make an exception, you will see the Mesos web UI.

The Mesos agents and master all need to communicate via SSL. You can run the Mesos agent as follows:

export LIBPROCESS_SSL_CA_DIR=/etc/mesos/pki 
export LIBPROCESS_SSL_ENABLED=true 
export LIBPROCESS_SSL_SUPPORT_DOWNGRADE=false
export LIBPROCESS_SSL_KEY_FILE=/etc/mesos/pki/localhost-key.pem 
export LIBPROCESS_SSL_CERT_FILE=/etc/mesos/pki/localhost.pem 
sudo -E ./bin/mesos-agent.sh --master=192.168.1.172:5050 --work_dir=/tmp

If you have any questions or feedback, please do not hesitate to drop a note!