Kubernetes Helm charts

Basic Requirements¶

kubectl
Helm 3.4.1+
Kubernetes cluster 1.18+
helm.comet.com credentials
External MySQL 5.7+ database (Internal MySQL DB possible for testing)
External S3 compatible bucket (Internal Minio Bucket Possible for testing)

Installation steps¶

Configure the Comet helm repository with provided credentials:

helm repo add comet-ml https://helm.comet.com/ --username your-user --password your-password

Query the available settings for the helm chart for reference.

helm show values comet-ml/comet-ml > default-values.yaml
less default-values.yaml

Download the example override-values.yaml to begin customizing your settings.

helm pull comet-ml/comet-ml -d /tmp && tar -x comet-ml/override-values.yaml -f /tmp/comet-ml-*.tgz -O > override-values.yaml && rm -f /tmp/comet-ml-*.tgz
vim override-values.yaml

At the bare minimum, you will still need to define these two values:

comet:
  cometHost: "my-comet.my-company.com"
  licenseKey: "<YOUR LICENSE KEY>"

Run helm install

helm install --namespace cometml --create-namespace -f override-values.yaml <COMET-RELEASE-NAME> comet-ml/comet-ml

Run the following command after making any changes or updates to values.yaml:

helm upgrade --namespace cometml -f override-values.yaml <COMET-RELEASE-NAME> comet-ml/comet-ml

After a few minutes, your Comet installation can be accessed at https://<VALUE OF comet.cometHost>

In order to check if everything is running correctly, you can reach the following endpoints:

  https://<VALUE OF comet.cometHost>/api/isAlive/ping
  https://<VALUE OF comet.cometHost>/clientlib/isAlive/ping
  https://<VALUE OF comet.cometHost>/optimizer/isAlive/ping

Configuration¶

The Comet ML Helm Chart is configured with plenty of sane defaults which should allow for you to launch a basic installation of Comet on your Kubernetes Cluster very quickly and with minimal configuartion. The example override-values.yaml file you downloaded above provides some hints about what sort of settings are commonly configured as part of a standard deployment.

Our default values.yaml file explains in detail many different options for configuring how our Helm chart works, through extensive comments. However, a few important configuration options deserve special mention here.

Nginx Ingress¶

If using the ingress class nginx for the nginx ingress controller, specific ingress annotations are required to ensure proper functionality and optimal performance. These annotations are especially crucial for uploads from the SDK with large body payloads or have operations that may take a significant amount of time.

In your values.yaml, add the following:

frontend:
  ingress:
    annotations:
      nginx.ingress.kubernetes.io/proxy-body-size: 2048m
      nginx.ingress.kubernetes.io/client-body-buffer-size: 2048m
      nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"

Sizing Defaults¶

We support some standard size defaults, which are defined at the end of the values.yaml file under the deploySizes key. Which size preset is used is selected by setting the size key in your override-values.yaml. By default the standard preset is used, but others exist including mini which is best for deploying a test instance on your local machine. While these presets set default values for various things such as Persisten Volume Claims and Replica Counts, you can still set your own value for any of these settings which will take precedence.

## Select the defaults to use for various deployment sizes.
## Valid values:
##   standard
##     (minimum recommended production size)
##   mini
##     (suitable for simple functionality testing)
##   aggressiveMinimum
##     (aggresively reserves the minimum system requirments)
size: standard

Internal Redis¶

For most production deployments of Comet we recommend using an external managed cache that is Redis-compatible (like Elasticache on AWS). However, either for smaller testing instances of Comet, or due to procurement limitations there may be a desire to run Redis within the Kubernetes Cluster as part of the Comet Helm release. This is supported, and turned on by default. To use an external Redis, configure the following:

redis:
  enableInternalRedis: false
  redisHost: "<your redis host>"
  # redisPort: "6379"  # This is the default.
  redisToken: "<token>"
  redisSSL: false

Additional configuration for the internal redis instance is done under the redisInternal key. However the default configuration should be sufficient for most. See the default values.yaml for more details.

Internal MySQL¶

Currently this should not be used for anything aside from light testing as the durability of the data cannot be guaranteed. However, by default the chart will install an instance and configure Comet to use it. To use an external MySQL database, configure the following:

mysql:
  enableInternalMysql: false
  mysqlHost: "<Your MySQL Host>"
  # mysqlPort: 3306  # This is the default.
  mysqlUsername: "<Comet User>"
  mysqlPassword: "<Comet User Password>"
  mysqlRootPassword: "<Root Password>"

List of Useful `kubectl` Commands¶

Get all resources from the cometml namespace (excluding CRDs)

kubectl -n cometml get all

Get all resources related to comet-ml (excluding CRDs)

kubectl get -A all -l app.kubernetes.io/part-of=comet-ml

Get pods list

kubectl get -n cometml pods -l app.kubernetes.io/part-of=comet-ml

See logs from a specific container:

kubectl -n cometml logs -f <container-id>

The desired container ID can be retrieved by running the prior pods list command.

Restart all of cometml:

kubectl -n cometml rollout restart deployment -l app.kubernetes.io/part-of=comet-ml
kubectl -n cometml rollout restart statefulset -l app.kubernetes.io/part-of=comet-ml

Jul. 9, 2024