All in One deployment

Basic Requirements¶

To deploy Comet application self-hosted/on-premises, you will need a Linux server with specific hardware requirements, a supported Linux distribution, and http/https connectivity between the Comet SDK and the Comet server. Familiarity with Linux command line tools is also recommended. Additionally, while a DNS name is optional, it may be useful for SSL certificates and easier access to the application. In this section, we will provide detailed information on these requirements and any additional considerations that should be kept in mind when deploying the application.

A server with supported Linux distribution, adequate storage space and backup cabability.
http/https Connectivity between the Comet SDK (where you train experiments to the comet server): You may want to include specific network requirements, such as the required ports that need to be open for communication to work properly. It may also be helpful to provide instructions on how to configure any proxies or firewalls that may be in place.
License token: This token is a string of characters that will be requested during the installation process. The license token is unique to your organization and is required to activate the application.
DNS name (optional): If a DNS name is optional, you may want to provide guidance on when it would be useful to have one, such as when using SSL certificates or for easier access to the application.

Server Requirements¶

Hardware requirements¶

For a single server deployment, also called all-in-one, we recommend the following specifications. Please note that these specifications may vary depending on usage, simultaneous training, and the number of users.

CPU: At least 16 vCPUs, such as an Intel Xeon or equivalent processor
Memory: At least 32 GB of RAM, with 64 GB recommended for optimal performance
Storage: At least 1 TB of root disk space, preferably SSD. The disk space allocation may be adjusted downward if you plan on storing experiment data on another partition or offsite.
Network: At least 10 Gbps network interface

If you're using a public cloud provider, the following instance types are recommended for use:

Amazon Web Services (AWS): m6i.4xlarge or larger instance types
Microsoft Azure: D16 v3 or larger instance types
Google Cloud Platform (GCP): n1-standard-16 or larger instance types

The application supports both RedHat and Debian-based distributions, including:

Ubuntu 18.04, 20.04 and 22.04
Amazon Linux 2, 2023
RHEL ⅞/9
CentOS 8

If you have the option to choose any distribution from these options, we recommend selecting Ubuntu. This is because Ubuntu provides most of the software dependencies required by the application in its package manager, making the installation process smoother and quicker.

Installation Steps ¶

Installing Cometctl ¶

Fetching the cometctl utility:

curl -sL https://installer.comet.com/comet-installer.sh | sudo bash

This command pulls and executes a bash setup script that will provision cometctl Once this command completes you can run cometctl from the cli using the cometctl commmand.

Initialize the Comet repository:

cometctl init --license-token {provided token}

Include your provided license token as received from the sales department or via email. This command will initialize the comet-ml package repository, and takes care of modifying the appropriate system files to allow the comet packages to be installed and updated.

Install Comet Enterprise:

cometctl aio install

As part of the installation process, cometctl will prompt you to set your comet_base_domain, when it does, you can set it to the IP or domain from the basic requirements

After that, you should see a splash screen showing a successful install, with instructions on how to access the Comet GUI. This will be the same IP or domain from the basic requirements. If you did not set the domain, or would like to change it for any reason, run cometctl aio update-config and set or change the comet_base_domain variable to your new IP or domain.

If you made any changes, you must run cometctl aio update-config

Enabling SSL¶

Run cometctl aio enable-ssl

Setup either a cloud Load Balancer with an SSL Certificate attached or make use of LetsEncrypt.
If using AWS, you must use an Application Load Balancer or any other LB operating at layer 7. Elastic and Classic load balancers will not work as they operate on layer 4.

If using LetsEncrypt, run: cometctl aio enable-ssl --letsencrypt

Special cases¶

If comet is fronted by a load balancer, set the idle timeout to at least 150 seconds, in order to allow uploads of big files.

Authentication setup ¶

Comet currently supports the following authentication methods/protocols:

Basic authentication (Default)
GitHub OAuth based authentication
Google OAuth based authentication
LDAP
Okta
SAML

Prerequisites for all OAuth authentication methods ¶

Set the comet_base_domain to point to a FQDN
Enable SSL

Github OAuth Instructions ¶

Create a new OAuth app in GitHub

Set the callback url to https://your-comet.com/api/github/github_oauth_callback
Run cometctl aio update-config and set the following parameters:

environment_token: onprem-github-auth
github_oauth_enabled: true
github_client_id: $CLIENT_ID
github_client_secret: $CLIENT_SECRET

Enable GitHub authentication in the web application
Edit /opt/comet-ml/html/public/config.js and set the following parameters:

SHOW_GOOGLE_AUTH: false,
DISABLE_GITHUB_AUTH: false,
DISABLE_SIGN_UP: true,

Verify Installation
Open your browser from any machine within the network and visit http://comet-ml.yourcompany.com
Sign up to your Comet enterprise version by clicking "Log in" on the top right corner
You should see “Log in with GitHub” option enabled

Google OAuth Instructions ¶

Create a new client ID in Google Cloud Platform Console If this is your first time creating a client ID, you will need to configure your consent as well

Set the callback URL to: https://your-comet.com/api/google/google_oauth_callback
Take a note of client ID and client secret for configuration in the next step.

Run cometctl aio update-config file and set the following parameters:

# Google Oauth
google_oauth_enabled: true
google_client_id: $CLIENT_ID
google_client_secret: $CLIENT_SECRET
environment_token: onprem-google-auth

Update the system configuration cometctl aio update-config

Verify Installation
Open your browser from any machine within the network and visit http://comet-ml.yourcompany.com
Sign up to your Comet enterprise version by clicking "Log in" on the top right corner
You should see “Log in with Google” option enabled

Python SDK Configuration ¶

On every trainer machine where you install the Comet SDK (i.e pip install comet_ml) you will also need to define your internal Comet installation

import os
os.environ["COMET_URL_OVERRIDE"]="http://comet-ml.yourcompany.com/clientlib/"
from comet_ml import Experiment

Environment variable:

export COMET_URL_OVERRIDE=http://comet-ml.yourcompany.com/clientlib/

Config file:

[comet]
url_override=http://comet-ml.yourcompany.com/clientlib/
Create a test.py file with this code snippet to test the installation:
import os
os.environ["COMET_URL_OVERRIDE"] = "http://comet-ml.yourcompany.com/clientlib/"
from comet_ml import Experiment
from time import sleep
experiment = Experiment(project_name="test-comet")
experiment.set_name("testing comet")
experiment.add_tag("working!")
experiment.log_other("finished", False)
for y in range(5000):
   experiment.log_metric("a", y)
   experiment.log_metric("b", y)
   experiment.log_metric("c", y)
   experiment.log_metric("d", y)
   experiment.log_parameter("paramA", 0.31000005)
   experiment.log_parameter("paramB", 0.31000005)
   experiment.log_parameter("paramD", 0.31000005)
   experiment.log_parameter("paramE", 0.31000005)
   experiment.log_parameter("paramF", 0.31000005)
   sleep(0.1)
experiment.log_other("finished", True)

Run:

pip install comet-ml

Run:

python test.py

Upgrading Comet¶

The following command will upgrade Comet to the latest version and prompt for an application restart to apply:

cometctl aio upgrade

Updating License Token¶

The following command can be used to update Comet with a new license token:

cometctl license-update -t {provided-token}

Useful Administration Commands¶

Updating config

The following opens the Comet configuration file for editing and validates the changes on saving:

cometctl aio update-config

Restart all Comet services

cometctl aio restart-services

Perform a quick health check to verify the services are running

cometctl aio healthcheck

Enabling SMTP ¶

SMTP is enabled by default as of Comet installer version 2.0.50.
The Comet SMTP server is currently supported on a best-effort basis only.

If you want to provide your own SMTP server, invoke cometctl aio update-config and make sure to change the following values:

smtp_enabled: true
smtp_host: <host>
smtp_port: <port>
smtp_user: <user>
smtp_password: <password>
smtp_protocol: <smtp or smtps>
mail_from: <user@yourcompanymail.com>

Update system configuration:

cometctl aio update-config

For additional documentation, please see SMTP Configuration

Backups ¶

We recommend setting up a policy on your cloud provider to rotate snapshots from your instance volume(s). AWS guide GCP guide Azure guide Bare Metal: We recommend you periodically take a backup of the disk or volume containing your experiment data using a method of your choosing.

Server Usage Report and BI Events ¶

In order to manage your team’s Comet subscription, Comet needs to track and report usage statistics from your on-premesis or VPC deployment of the Comet system. This allows Comet to easily track and manage user statistics for true-ups and billing, without any additional overhead for your team’s management of the deployment.

Firewall whitelist information is as follows

Port: 443
Host: stats.comet.com
IP: 3.210.8.13
Proto: TCP

Hostname and IP are statically set and should not change.
A complete sample usage report can be found below:

{
    "usage_id": "2022-12",
    "report": [
      {
        "username": "benjamin",
        "email": "z@gmail.com",
        "created_at": 1662014273604,
        "uiUsageCount": 1,
        "uiUsageUpdateTs": 1662014275281,
        "sdkUsageCount": 0,
        "sdkUsageUpdateTs": 1662014275281,
        "suspended": false
      },
    ],
    "number_of_users": 1,
    "licenseKey": "licenseKey1234",
    "domain": "gmail.com",
    "backendVersion": "3.2.74",
    "frontendVersion": "NA"
  }

Below is an anonymized version of an actual BI event report:

{
  "anonymousId": null,
  "context": {
    "library": {
      "name": "analytics-python",
      "version": "2.2.1"
    }
  },
  "event": "new_experiment_be",
  "integrations": {},
  "messageId": "e400c2e6-d161-4225-a992-a3621fff95eb",
  "originalTimestamp": "2023-01-13T14:40:41.337309+00:00",
  "properties": {
    "backendVersion": "3.2.165",
    "client": "my-company",
    "domain": "my-company.com",
    "experimentKey": "c249967859b24153b3c8659aac9g3dab",
    "frontendVersion": "5.20.2",
    "licenseKey": "pffka9493j2bd963do98Tfabccf80997r",
    "on_prem": true,
    "optimizationId": null,
    "organizationId": "PJ9ULeGB7kqQ4G7J08DsrgpoS",
    "organizationName": "defaultOrganization",
    "projectId": "a6e22f54d84a548b3f1760bf6421602b",
    "projectName": "my-recommender-model",
    "pyVersion": "3.8.12",
    "sdkVersion": "3.31.22",
    "status": "true",
    "teamName": "my-team",
    "version": "3.31.22"
  },
  "receivedAt": "2023-01-13T14:40:41.956Z",
  "sentAt": "2023-01-13T14:40:41.714Z",
  "timestamp": "2023-01-13T14:40:41.579Z",
  "type": "track",
  "userId": "my-name",
  "writeKey": "zZVvzqFNeXDt2KSyeGofvrsKY6osd6Ht"
}

If you have any questions regarding Comet Usage and Events Reporting, please contact support@comet.com or your account manager.

Caveats and FAQ¶

Q: I'm seeing a prompt asking me to restart as part of the install A: On certain newer distributions of Linux, you may be prompted as part of cometctl init or cometctl install with an ncurses prompt asking whether you'd like to restart any services. If you do not know what to do here, you may hit enter and bypass this.

Jul. 9, 2024