Mesosphere® DC/OS™ Enterprise offers a range of features that allow you to secure your cluster and prevent breaches and other attacks. This section provides an overview of the security features and recommendations for hardening your cluster.
The goals of DC/OS security are:
- Isolate the cluster perimeter with strong authentication and authorization across all interfaces.
- Secure and protect the internal cluster communication, containers, and sandboxes.
- Enhance cluster security with support for 3rd party security integrations.
DC/OS is based on a Linux® kernel and userspace. The same best practices for securing any Linux system apply to securing DC/OS, including setting correct file permissions, restricting root and normal user accounts, protecting network interfaces with
iptables or other firewalls, and regularly applying updates from the Linux distribution used with DC/OS to ensure that system libraries, utilities, and core services like
systemd and OpenSSH are secure.
At the highest level we can distinguish three security zones in a DC/OS deployment, namely the admin, private, and public security zones.
The admin zone is accessible through HTTP/HTTPS and SSH connections, and provides access to the master nodes. It also provides reverse proxy access to the other nodes in the cluster via URL routing. For security, the DC/OS cloud template allows configuring a whitelist so that only specific IP address ranges are permitted to access the admin zone.
Access to the admin zone is controlled by the Admin Router.
HTTP requests incoming to your DC/OS cluster are proxied through the Admin Router (using Nginx™ with OpenResty® at its core). The Admin Router denies access to most HTTP endpoints for unauthenticated requests. In order for a request to be authenticated, it needs to present a valid authentication token in its
Authorization header. A token can be obtained by going through the authentication flow.
The private zone is a non-routable network that is only accessible from the admin zone or through the edge router from the public zone. Deployed services are run in the private zone. This zone is where the majority of agent nodes are run.
The optional public zone is where publicly accessible applications are run. Usually, only a small number of agent nodes are run in this zone. The edge router forwards traffic to applications running in the private zone. The agent nodes in the public zone are labeled with a special role so that only specific tasks can be scheduled here. These agent nodes have both public and private IP addresses and only specific ports should be open in their
A typical deployment, including load balancers, is shown below:
Figure 1. Security zone typical deployment
You can control DC/OS Enterprise access by resource and operation (create, read, update, delete). The available security modes are permissive and strict. Strict mode provides the finest-grained controls. The DC/OS permissions are enforced based on your security mode. The security mode is set during DC/OS installation and can only be changed by performing an upgrade.
|Admin Router permissions (
|Apache® Mesos® permissions (
|Marathon™ and Metronome permissions (
|Secret store permissions (
See the permissions reference for a complete description.
This mode provides some of the security features, but does not include the Mesos permissions.
This mode provides the most robust security posture and requires a significant amount of configuration.
You can use either of the following methods to determine the security mode of an existing cluster.
GETrequest to the following endpoint:
http[s]: /mesosphere/dcos//<cluster-url>/dcos-metadata/bootstrap-config.json. Requirements: Your user account must have either the
dcos:adminrouter:ops:metadata fullpermission or the
strict, you must use HTTPS. Review Securing your TLS communications to discover how to obtain the root certificate of your DC/OS CA and provision it to your preferred client.
SSH into your master and view the contents of
All requests from outside of the DC/OS cluster require an authentication token. Depending on your security mode, in-cluster authentication tokens may be required. For more information, see the Service Accounts documentation.
DC/OS provisions masters with ZooKeeper credentials during the bootstrap sequence. This allows the masters to nominate themselves as potential Mesos leaders.
You can log in by using the DC/OS GUI, the DC/OS CLI, or a programmatic client.
- If you have configured an LDAP directory server, DC/OS will pass your credentials to the LDAP directory server for verification.
- If you have configured a SAML or an OpenID Connect identity provider (IdP), you will pass your credentials directly to the IdP.
The following diagram details the sequence.
Figure 2. User authentication sequence
When the authentication token expires, you can re-authenticate to receive another.
When you log in with the DC/OS GUI, the Identity and Access Manager plants a cookie that contains the authentication token. While it is protected with an
HttpOnly flag, you should Sign Out at the end of your browser session to clear this cookie.
Note that clearing the cookie does not invalidate the authentication token. If sniffed over an unencrypted connection or extracted from the cookie, someone could use the authentication token to log into DC/OS. To mitigate this risk, we recommend setting the secure flag on the cookie in
strict modes, as discussed in Hardening.
Credentials for cluster-local user accounts (those not using LDAP, SAML, or OpenID Connect®) consist of a user name and password that can be used to validate, but not reproduce, user passwords. Passwords are individually salted and cryptographically hashed using crypt(3) SHA-512. This results in one-way hashes that can be used to validate but not reproduce user passwords. To further impede brute force attacks and meet or exceed NIST FIPS security requirements, the hash function performs many iterations using a 128 bit salt length.
Once DC/OS IAM has validated your credentials, an authentication token is returned to you. The authentication token is then used for further request authentication during your session. This way the password does not need to be stored in the client and is only sent over the wire immediately after you enter it. Over the wire, the authentication request is encrypted using TLS. TLS is required and enforced in strict mode, but optional in permissive mode. For more information, see Security Modes.
Service accounts provide an identity for services to authenticate with DC/OS. Service accounts control communication between services and DC/OS components. DC/OS services may require service accounts depending on your security mode.
DC/OS automatically provisions DC/OS components (systemd services on the DC/OS nodes) with service accounts during the bootstrap sequence.
For example, the Mesos agents are provisioned with service accounts that they use to authenticate to the Mesos master. This ensures that only authorized agents can join the Mesos cluster, advertise resources, and get asked to launch tasks.
You can view the
systemd service accounts from the Organization -> Service Accounts tab of the DC/OS GUI. These service accounts are prefixed with
In addition to authenticating requests, DC/OS also checks the permissions associated with the account to determine whether the requestor is authorized to access the requested resource.
The following diagram illustrates the authorization sequence.
Figure 3. Authorization sequence
OPT sequence in the diagram illustrates how permission enforcement varies by security mode.
Admin Router and the Secret Store enforce their permissions in all security modes.
Metronome and Marathon enforce their permissions in
strictmodes. However, the enforcement in
permissivemode only occurs if the requestor presents an authentication token, which is optional in
permissivemode. If an in-cluster requestor does not present an authentication token, Metronome and Marathon will act as if the request was made by a user with the
The Mesos masters and agents enforce their permissions only in
The diagram does not show the Secret Store sequence. The Admin Router does not check the permissions on requests to the Secret Store. It routes these requests to the Secret Store, which enforces its own permissions on each request.
For more information about permissions, refer to Managing permissions.
The encryption of DC/OS communications varies according to your security mode.
|Security mode||External communications*||Internode communications|
|Permissive||HTTP and HTTPS are supported. HTTP requests to the root path (for example,
|Strict||Only HTTPS is supported. All HTTP connections are redirected to HTTPS. If you log in to DC/OS with a password, the password is transmitted securely (requires proper certificate verification, including hostname verification on the client side). If one or more HTTP proxies or load balancers are between the user agent and Admin Router, then the secure password transmission applies to the final communication between Admin Router and the previous proxy or load balancer.||Encryption enforced**|
* Communications with clients outside of the cluster. For example, browsers and the DC/OS CLI.
** Except internode communications between instances of ZooKeeper, which are not encrypted in any security mode. Each master node has a ZooKeeper instance. These ZooKeeper instances communicate periodically to keep their in-memory databases in sync. You can use IPsec to manually encrypt these communications.
Not all existing user services support encryption at this time. If the service supports encryption, you can enable it in
permissive mode. In
strict mode, encryption of user service communications is enforced. As a result, only user services that support encryption can be deployed in
Internode communications occur over TLS 1.2. To ensure browser support, external communications currently accept TLS 1.0, 1.1, and 1.2. These settings are configurable.
For more information, see Securing communication with TLS.
Spaces allow you to:
At a minimum, we recommend using spaces to restrict service access to secrets.
One aspect of spaces involves service and job groups. You can put services and jobs into groups in any security mode. This can help users find the jobs or services that pertain to them.
You can use permissions to restrict user’s access on a per service/job or service/job group basis.
The secret path controls which services can access it. If you do not specify a path when storing a secret, any service can access it.
Secret paths work in conjunction with service groups to control access. However, you do not need to have service groups to control access to secrets, you can also use the name of the service. The following table provides a few examples to show how it works.
|Secret||Service||Can service access secret?|
- If only a single service requires access to a secret, store the secret in a path that matches the name of the service (for example,
hdfs/secret). This prevents it from being accessed by other services.
- Service groups begin with
/, while secret paths do not.
To secure sensitive values like private keys, API tokens, and database passwords, DC/OS provides:
DC/OS stores Secret Store data in ZooKeeper encrypted under an unseal key using the Advanced Encryption Standard (AES) algorithm in Galois Counter Mode (GCM). The Secret Store uses the unseal key to encrypt secrets before sending them to ZooKeeper™ and to decrypt secrets after receiving them from ZooKeeper. This ensures that secrets are encrypted both at rest and in transit. TLS provides an additional layer of encryption on the secrets in transit from ZooKeeper to the Secret Store.
The unseal key is encrypted under a public GPG key. Requests to the Secrets API return only the encrypted unseal key. When the Secret Store becomes sealed, either manually or due to a failure, the private GPG key must be used to decrypt the unseal key and unseal the Secret Store. As a convenience, DC/OS automatically generates a new 4096-bit GPG keypair during the bootstrap sequence. It uses this keypair to initialize the Secret Store and stores the keypair in ZooKeeper.
The Secret Store is available in all security modes. By default, you cannot store a secret larger than one megabyte. If you need to exceed this limit, contact Mesosphere support. We do not support alternate or additional Secret Stores at this time. You should use only the
default Secret Store provided by Mesosphere.
DC/OS allows you to restrict:
User access to secrets: use permissions to control which users can access what secrets and what operations they can perform.
Application access to secrets: use spaces to control which applications can retrieve what secrets.
By default, all tasks will run inside of Docker® containers. Please see Deploying a Docker-based Service to Marathon for an example.
The following table identifies the default Linux user in each situation.
|Mesos (UCR)||Task runs under
||Task runs under
|Docker||Task runs under
||Task runs under
Docker tasks run under
root by default, but Docker user privileges are confined to the Docker container. Should you wish to change the default task user, modify the Docker container. Please reference the Docker documentation for more information, as well as the user service documentation.
Managing Users and Groups
Managing users and groups…Read More
Granting Access to the UI
Granting access to the DC/OS UI…Read More
Managing permissions…Read More
Understanding DC/OS access and permissions references…Read More
Setting up a directory-based authentication server via LDAP…Read More
Understanding the Secret Store…Read More
Identity provider-based authentication
Configuring identity provider-based authentication…Read More
Authenticating service accounts…Read More
Restricting Access to DC/OS Service Groups
Using the DC/OS UI to achieve multi-tenancy in permissive mode…Read More
Increasing the security of your cluster…Read More
Identity and Access Management API
Managing users and permissions with the IAM API…Read More
Securing Communication with TLS
Securing encrypted communications using TLS certificates…Read More