Ceph can be used as the CSI Provisioner in some environments. For environments where Ceph was installed before installing DKP, you can reuse your existing Ceph installation to satisfy the storage requirements of DKP Applications. Hello

This guide assumes you have a Ceph cluster that is not managed by DKP.

Refer to Rook Ceph Configuration for information on how to configure the Ceph instance installed by DKP for use by DKP platform applications.

Disable DKP Managed Ceph

Disable rook-ceph in your installer config since the default config of DKP has already installed a Ceph Cluster.

Disable rook-ceph in the installer config to prevent DKP from installing a Ceph cluster:

apiVersion: config.kommander.mesosphere.io/v1alpha1
kind: Installation
apps:
  rook-ceph:
    enabled: false
  rook-ceph-cluster:
    enabled: false
  ...
...
CODE

The DKP instances of velero and grafana-loki rely on the storage provided by Ceph. Before installing the Kommander component of DKP, be sure to configure appropriate Ceph resources for their usage as detailed in the next section.

Creating DKP Compatible Ceph Resources

This section walks you through the creation of CephObjectStore and then a set of ObjectBucketClaims, which can be consumed by either velero and grafana-loki.

Typically, Ceph is installed in the rook-ceph namespace, which is the default namespace if you have followed the Quickstart - Rook Ceph Documentation guide.

This guide assumes your Ceph instance is installed in the rook-ceph namespace. Configure the variable CEPH_NAMESPACE in subsequent steps as it is applicable to your environment.

Creating CephObjectStore

There are two ways to install Ceph:

Helm Chart Values

This section is relevant if you have installed Ceph using helm install or some other managed Helm resource mechanism.

If you have applied any configuration overrides to your Rook Ceph operator, ensure it was deployed with currentNamespaceOnly set to false (It is the default value, so unless you have applied any overrides, it will be false by default). This ensures that the Ceph Operator in the rook-ceph namespace is able to monitor and manage resources in other namespaces such as kommander.

Ensure the following configuration for rook-ceph helm chart:

# This is the default value, so need to overwrite if you are just using the defaults as-is
currentNamespaceOnly: false
CODE

You must enable the following configuration overrides for the rook-ceph-cluster helm chart:

cephObjectStores:
  - name: dkp-object-store
    # see https://github.com/rook/rook/blob/master/Documentation/CRDs/Object-Storage/ceph-object-store-crd.md#object-store-settings for available configuration
    spec:
      metadataPool:
        # The failure domain: osd/host/(region or zone if available) - technically also any type in the crush map
        failureDomain: osd
        # Must use replicated pool ONLY. Erasure coding is not supported.
        replicated:
          size: 3
      dataPool:
        # The failure domain: osd/host/(region or zone if available) - technically also any type in the crush map
        failureDomain: osd
        # Data pool can use either replication OR erasure coding. Consider the following example scenarios:
        # Erasure Coding is used here with 3 data chunks and 1 parity chunks which assumes 4 OSDs exist.
        # Configure this according to your CephCluster specification.
        erasureCoded:
          dataChunks: 3
          codingChunks: 1
      preservePoolsOnDelete: false
      gateway:
        port: 80
        instances: 2
        priorityClassName: system-cluster-critical
        resources:
          limits:
            cpu: "750m"
            memory: "1Gi"
          requests:
            cpu: "250m"
            memory: "500Mi"
      healthCheck:
        bucket:
          interval: 60s
    storageClass:
      enabled: true
      name: dkp-object-store
      reclaimPolicy: Delete
YAML

Managing resources directly

  1. Set a variable to refer to the namespace the AppDeployments are created in.
    NOTE: This is the kommander namespace on the management cluster or Workspace namespace on all other clusters.

    export CEPH_NAMESPACE=rook-ceph
export NAMESPACE=kommander
    CODE

  2. Create CephObjectStore in the same namespace as the CephCluster:

    cat <<EOF | kubectl apply -f -
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: dkp-object-store
  namespace: ${CEPH_NAMESPACE}
spec:
  metadataPool:
    # The failure domain: osd/host/(region or zone if available) - technically, any type in the crush map
    failureDomain: osd
    # Must use replicated pool ONLY. Erasure coding is not supported.
    replicated:
      size: 3
  dataPool:
    # The failure domain: osd/host/(region or zone if available) - technically, any type in the crush map
    failureDomain: osd
    # Data pool can use either replication OR erasure coding. Consider the following example scenarios:
    # Erasure Coding is used here with 3 data chunks and 1 parity chunks which assumes 4 OSDs exist.
    # Configure this according to your CephCluster specification.
    erasureCoded:
      dataChunks: 3
      codingChunks: 1
  preservePoolsOnDelete: false
  gateway:
    port: 80
    instances: 2
    priorityClassName: system-cluster-critical
    resources:
      limits:
        cpu: "750m"
        memory: "1Gi"
      requests:
        cpu: "250m"
        memory: "500Mi"
  healthCheck:
    bucket:
      interval: 60s
EOF
    CODE

  3. Wait for the CephObjectStore to be Connected:

    $ kubectl get cephobjectstore -A       
NAMESPACE   NAME               PHASE
rook-ceph   dkp-object-store   Progressing
...
...
rook-ceph   dkp-object-store   Connected
    NONE

  4. Create a StorageClass to consume the object storage:

    cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: dkp-object-store
parameters:
  objectStoreName: dkp-object-store
  objectStoreNamespace: ${CEPH_NAMESPACE}
provisioner: ${CEPH_NAMESPACE}.ceph.rook.io/bucket
reclaimPolicy: Delete
volumeBindingMode: Immediate
EOF
    YAML

Creating ObjectBucketClaims

  1. Once the Object Store is Connected, create the ObjectBucketClaim in the same namespace as velero and grafana-loki .
    This results in the creation of ObjectBucket , which creates Secrets that are consumed by velero and grafana-loki.

    1. For grafana-loki:

      cat <<EOF | kubectl apply -f -
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: dkp-loki
  namespace: ${NAMESPACE}
spec:
  additionalConfig:
    maxSize: 80G
  bucketName: dkp-loki
  storageClassName: dkp-object-store
EOF
      YAML

    2. For velero:

      cat <<EOF | kubectl apply -f -
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: dkp-velero
  namespace: ${NAMESPACE}
spec:
  additionalConfig:
    maxSize: 10G
  bucketName: dkp-velero
  storageClassName: dkp-object-store
EOF
      YAML

  2. Wait for the ObjectBuckets to be Bound by executing the following command: 

    kubectl get objectbucketclaim -n${NAMESPACE} -ocustom-columns='NAME:.metadata.name,PHASE:.status.phase'
    CODE

    which should display something similar to: 

    NAME         PHASE
dkp-loki     Bound
dkp-velero   Bound
    CODE

Overriding velero and grafana-loki Configuration

Once all the buckets are in the Bound state, DKP applications are now ready to be installed with the following configuration overrides populated in the installer config: 

cat <<EOF | kubectl apply -f -
apiVersion: config.kommander.mesosphere.io/v1alpha1
kind: Installation
apps:
  grafana-loki:
    enabled: true
    values: |
      loki:
        structuredConfig:
          storage_config:
            aws:
              s3: "http://rook-ceph-rgw-dkp-object-store.${CEPH_NAMESPACE}.svc:80/dkp-loki"
      ingester:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
      querier:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
      queryFrontend:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
      compactor:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
      ruler:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
      distributor:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
  velero:
    enabled: true
    values: |
      configuration:
        provider: "aws"
        backupStorageLocation:
          bucket: dkp-velero
          config:
            region: dkp-object-store
            s3Url: http://rook-ceph-rgw-dkp-object-store.${CEPH_NAMESPACE}.svc:80/
      credentials:
        # This secret is owned by the ObjectBucketClaim. A ConfigMap and a Secret with same name as bucket are created.
        extraSecretRef: dkp-velero
EOF
YAML

This installer config can be merged with your installer config with any other relevant configuration before installing DKP.

Overriding project-grafana-loki Configuration

When installing project level grafana loki, its configuration needs to be overridden in a similar manner to workspace level grafana loki, so that the project logs can be persisted in Ceph storage.

The following overrides need to be applied to project-grafana-loki : 

loki:
  structuredConfig:
    storage_config:
      aws:
        s3: "http://rook-ceph-rgw-dkp-object-store.${CEPH_NAMESPACE}.svc:80/dkp-loki"
CODE

These overrides can be applied from the UI directly while substituting the ${CEPH_NAMESPACE} appropriately.

If you are using using CLI, follow these steps:

  1. Set NAMESPACE to project namespace and CEPH_NAMESPACE to Ceph install namespace:

    export CEPH_NAMESPACE=rook-ceph
export NAMESPACE=my-project
    CODE

  2. Create a ConfigMap to apply the configuration overrides:

    cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  values.yaml: |
    loki:
      structuredConfig:
        storage_config:
          aws:
            s3: "http://rook-ceph-rgw-dkp-object-store.${CEPH_NAMESPACE}.svc:80/proj-loki-${NAMESPACE}"
kind: ConfigMap
metadata:
  name: project-grafana-loki-ceph
  namespace: ${NAMESPACE}
EOF
    CODE

  3. Create the AppDeployment with a reference to the above ConfigMap:
    NOTE: The clusterSelector can be adjusted according to your needs. 

    cat <<EOF | kubectl apply -f -
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
  name: project-grafana-loki
  namespace: ${NAMESPACE}
spec:
  appRef:
    kind: ClusterApp
    name: project-grafana-loki-0.48.6
  clusterSelector: {}
  configOverrides:
    name: project-grafana-loki-ceph
EOF
    CODE

The project level Grafana Loki creates an ObjectBucketClaim and assumes that the Ceph operator is montitoring the project namespace, so there is no need to create ObjectBucketClaim manually.