Security and self-signed certificates: Configuring a secure Mesos cluster using SSL

Enabling SSL in Mesos requires the libevent-openssl library and its dependencies. It is the library by which all messages flowing through the Mesos cluster will be encrypted.

More information at https://github.com/libevent/libevent.

We can retrieve the pre-packaged libevent library and its dependencies via the local linux package manager.

Package manager:

• yum:

sudo yum install libevent-devel -y
sudo yum install openssl-devel -y

• apt:

sudo apt-get update
sudo apt-get install libssl-dev -y
sudo apt-get install libevent-dev -y

The next step is to download the Mesos repository and install its dependencies. Do not build the Mesos binary yet.

Instructions for downloading Mesos and installing its dependencies can be found here: https://mesos.apache.org/gettingstarted.

 When you reach the "Building Mesos" section, run the configure script like such:

../configure --enable-libevent --enable-ssl

Continue to build Mesos following the instructions as they are.

If you are running Mesos on a remote machine and wish to access the Mesos web UI from the client, it is best to bind the Mesos master to a routable IP address.

For this example we will use IP 192.168.1.172. (If you are running the browser on the same machine where you are starting the Mesos master, you can bind the Mesos master to your localhost: 127.0.0.1).

Using the keys we have created from Part 1 we will start a secure Mesos master. At this point, we should have the following keys:

• Certificate Authority: ca.pem
• Digital Certificate: localhost.pem
• Private Key: localhost-key.pem

Let's put those in the following folder /etc/mesos/pki.

The startup command for the Mesos master will be:

export LIBPROCESS_SSL_CA_DIR=/etc/mesos/pki 
export LIBPROCESS_SSL_ENABLED=true 
export LIBPROCESS_SSL_SUPPORT_DOWNGRADE=false 
export LIBPROCESS_SSL_KEY_FILE=/etc/mesos/pki/localhost-key.pem 
export LIBPROCESS_SSL_CERT_FILE=/etc/mesos/pki/localhost.pem 
sudo -E ./bin/mesos-master.sh --ip=192.168.1.172 --work_dir=/tmp

LIBPROCESS_SSL_ENABLED is a convenient way of turning off SSL communication between the master and agents, between the submitted Mesos framework and the Mesos cluster or on any connection to a Mesos http endpoint. When it is false, Mesos will ignore all other LIBPROCESS_SSL_* related arguments. At this point, the Mesos master is up and running without any Mesos agent. However, the Mesos web UI should now appear from your browser at https://192.168.1.172:5050.

You will probably see a warning on your browser saying this connection is unstrusted. Remember from Part 1 we said a home made self-signed certificate is not certified by a known Certificate Authority. In other words, the Certificate Authority we created is itself not certified by a known Certificate Authority.

The reality is that when you login to your favorite e-mail service (or any official secure website) the Certificate Authority used is itself certified by a higher Certificate Authority to form a tree. At the root of the tree is what we call the Root Certificate.

Since the Certificate Authority we generated is itself not verified, the browser is issuing a warning stating that the identification cannot be trusted.

In fact, the browser is detecting a couple of issues. It warns that the issuer certificate (Certificate Authority ca.pem) is unknown. It also warns that the certificate (localhost.pem) is only valid for localhost and 127.0.0.1. In my case, I have binded the Mesos master to 192.168.1.172 and not 127.0.0.1. Beware that in Part 1, we have generated the certificate for a service running on the localhost.

"192.168.1.172:5050 uses an invalid security certificate.

The certificate is not trusted because the issuer certificate is unknown.
The server might not be sending the appropriate intermediate certificates.
An additional root certificate may need to be imported.
The certificate is only valid for the following names:
localhost, localhost, localhost, 127.0.0.1, 127.0.0.1

(Error code: sec_error_unknown_issuer)"

Let's look at the content of the generated certificate:

openssl x509 -in localhost.pem -text | less

If you look for "X509v3 Subject Alternative Name" in the output, you should see:

X509v3 Subject Alternative Name:                 DNS:localhost

To generate a certificate for the service running on the routable IP, change the command (from Part 1) from:

cfssl gencert -ca ca.pem -ca-key ca-key.pem -hostname localhost csr.json | cfssljson -bare localhost

to:

cfssl gencert -ca ca.pem -ca-key ca-key.pem -hostname 192.168.1.172 csr.json | cfssljson -bare localhost

Now the output of

 openssl x509 -in localhost.pem -text | less 

should contain:

X509v3 Subject Alternative Name:
                IP Address:192.168.1.172

If you generate the certificate with the routable IP, you should only get the warning that the issuer certificate is unknown.

"192.168.1.172:5050 uses an invalid security certificate.

The certificate is not trusted because the issuer certificate is unknown.
The server might not be sending the appropriate intermediate certificates.
An additional root certificate may need to be imported.

(Error code: sec_error_unknown_issuer)"

If you proceed and make an exception, you will see the Mesos web UI.

The Mesos agents and master all need to communicate via SSL. You can run the Mesos agent as follows:

export LIBPROCESS_SSL_CA_DIR=/etc/mesos/pki 
export LIBPROCESS_SSL_ENABLED=true 
export LIBPROCESS_SSL_SUPPORT_DOWNGRADE=false
export LIBPROCESS_SSL_KEY_FILE=/etc/mesos/pki/localhost-key.pem 
export LIBPROCESS_SSL_CERT_FILE=/etc/mesos/pki/localhost.pem 
sudo -E ./bin/mesos-agent.sh --master=192.168.1.172:5050 --work_dir=/tmp

If you have any questions or feedback, please do not hesitate to drop a note!  

Security and self-signed certificates: A primer on digital certificates and generating a self-signed certificate

What is a digital certificate?

The idea behind a certificate is to be assured the person (or service) we are talking to is in fact who the person (or service) claims to be.

Part of the 3 way handshake in establishing the security protocol is for the remote service to provide the client with its public key and certificate.

The certificate aims at assuring the client the public key does in fact belong to the remote party (person or service). The client then uses the public key to encrypt the data passed to the remote party. The remote party then decrypts the data using its private key (never shared).

Therefore, in establishing server authentication, there are 3 elements which come into play:

-          Certificate authority (Trusted 3^rd party assuring identity of remote service)

-          Server Certificate (Certificate embedding the remote service’s public key)

-          Private Key (Remote service’s private to decrypt incoming data)

The crux of the public-key cryptographic system is that only the entity with the private key can decrypt the data encrypted with the public key.

In our case, the remote party is the entity holding the private key.

Therefore, the public key is something that a service would send to the client in order to encrypt the data. However, the client needs to know that the public key in fact belongs to the remote service entity. What the service sends is its public key along with its identity together forming the Server Certificate which can be verified by a trustworthy 3^rd party which we refer to as the Certificate Authority. The client will take the Server Certificate and contact the Certificate Authority and make the verification.

How is the Server Certificate generated in theory?

If entity A wants to share its public key, it would want everyone who cares about the key to know it is really from A. Entity A makes a request to a known Certificate Authority by sending its public key, contact and identity attributes. The Certificate Authority contacts A to let it know it’s going to “certify” it and return the public key signed by the Certificate Authority with its private key. Now when entity A sends its certificate to a client (signed public key + identity attributes), the client will use the Certificate Authority’s public key to decrypt the entity A’s certificate. Since only the trustworthy Certificate Authority has the private key, we know the public key embedded in the Server Certificate is in fact entity A’s key.

Once the verification is established, the client can proceed in encrypting the data using the public key.

There are numerous reasons for not using self-signed certificates. In general, we resort to using self-signed certificates for testing secure applications in the development environment. A self-signed certificate is a certificate signed by an unknown (sometimes not trustworthy) Certificate Authority. In this case, we are the Certificate Authority and signing our own Server Certificate.

To generate the 3 elements necessary in establishing server authentication, we will use the 3^rd party tool cfssl ( https://github.com/cloudflare/cfssl ).

We will run two commands:

-          Generate the Certificate Authority

-          Generate the Server Certificate and Private Key

Both commands will take JSON as in input.

The JSON inputs are key requests containing identity attributes. For the first command, a test request in the JSON format is saved to file mycacsr.json:

{
    "CN": "myDevCa",
    "key" : {"algo": "rsa", "size": 2048},
    "names" : [{"C": "US", "L": "New Jersey", "OU": "WWW", "O": "Example", "ST": "New Jersey"}]
}

To generate the Certificate Authority (or root certificate in our case):

cfssl genkey -initca mycacsr.json | cfssljson -bare ca

At this point, you should have a Certificate Authority saved as ca.pem and a Certificate Authority key ca-key.pem.

The next command is generating the Server Certificate and Private Key which will take a JSON input saved to file myservercert.json:

{
    "key" : {"algo": "rsa", "size": 2048},
    "names" : [{"C": "US", "L": "New Jersey", "OU": "WWW", "O": "Example", "ST": "New Jersey"}]
}

Along with the JSON input, the second command will take the previously generated Certiticate Authority and Certificate Authority Key to generate the Server Certificate and Private Key:

cfssl gencert -ca ca.pem -ca-key ca-key.pem -hostname
localhost myservercert.json | cfssljson -bare localhost

At this point, you have a server certificate localhost.pem and private key localhost-key.pem

Why did we need the Certificate Authority and Certificate Authority’s key to generate our certificate?

As explained before, the Server Certificate will be signed using the Certificate Authority’s private key.

The next blog will be on how to start secured services such as Mesos or Kubernetes using our self-signed certificate.

If you have any questions or feedback, please do not hesitate to drop a note!