Send data to MPM

Start logging model predictions, as well as new models, simply by sending Comet your model's input features and predictions. There is no additional configuration required.

There are two main integration paths when it comes to MPM:

• Sending data from the application making the predictions
• Using log forwarding

If the inference service is written in Python and predictions are not tracked already, we recommend sending events straight from the application using our MPM Python SDK.

If you are using a managed inference service like Sagemaker or Seldon, you can simply enable Data Capture or Request / Response logging and forward these events to MPM using our Rest API.

Integration with Experiment Management¶

Using MPM in conjunction with Experiment Management and Comet's Model Registry allows you to track models from development all the way to production ! In order to do this, you simply need to ensure that the model name used when sending MPM events matches up with the model name in the Model Registry.

MPM can also be used standalone, in which case models are automatically created in the Comet Model Registry whenever you send the first event for a new model.

MPM Python SDK¶

The MPM Python SDK has been developed with production inference services in mind and includes a number of optimizations to make sure the logging overhead is kept at a minimum.

The MPM SDK can be installed using:

pip install comet-mpm

Once installed, logging events to MPM takes just three lines of code:

from comet_mpm import CometMPM

MPM = CometMPM(model_name='credit-scoring', model_version='1.2.0')

MPM.log_event(
    prediction_id="1",
    input_features={'age': 29, 'country': 'UK'},
    output_features={'value': True, 'probability': 0.5}
)

You can also send labels using the SDK:

from comet_mpm import CometMPM

MPM = CometMPM(model_name='credit-scoring', model_version='1.2.0')

MPM.log_label(
    prediction_id="1",
    label=False
)

Support for different model types¶

Comet MPM supports the logging of many different model types, from regression models to multi-class classification models.

MPM Rest API¶

To track model performance, MPM needs to have access to a model's input features, output features and labels. In addition to the Python API, this data can be sent via a Rest API.

The MPM Rest API includes three methods that can be used to upload MPM events:

• https://www.comet.com/mpm/events: Used to upload single events that contain input and output features
• https://www.comet.com/mpm/events/batch: Used to upload batches of events that contain input and output features
• https://www.comet.com/mpm/labels: Used to upload labels

Sending input and output features to MPM¶

The JSON payload for the https://www.comet.com/mpm/events and https://www.comet.com/mpm/events/batch POST endpoints should contain the following attributes:

Here is an example payload:

{
    "workspaceName": "...",
    "modelName": "Credit Scoring",
    "modelVersion": "1.0.0",
    "timestamp": 1615922560000,
    "predictionId": "000001",
    "features": {
        "feature_1":0.34,
        "feature_2": "dog",
    },
    "prediction": {
        "value": "true",
        "probability": 0.86
    }
}

To test, you can use:

workspaceName=<workspace_name>
api_key=<api_key>

current_timestamp=$(date -v-1H +%s000)
payload='{"features": {"categorical_feature_0": "value_1", "numerical_feature_0": 0.5841119597210334}, "modelName": "test-model", "modelVersion": "1.0.0", "prediction": {}, "predictionId": "e347539b-a1df-432e-aa4a-3fe93805d3be", "timestamp": '$current_timestamp', "workspaceName": "'$workspaceName'"}'

curl -s -d "$payload" \
    -H "Content-Type: application/json"\
    -H "Authorization: $api_key"\
    -X POST https://www.comet.com/mpm/events

Sending labels to MPM¶

Ground truth labels can be sent to Comet and are used to compute accuracy related metrics (Accuracy, Precision, Recall, F1-score, etc). You can send the labels at any time after a prediction has been logged to MPM, we will take care of updating the relevant metrics.

Labels can be sent to Comet using the https://www.comet.com/mpm/labels POST endpoint with a JSON payload containing the following attributes:

Learn more¶

• About Comet's MPM
• Learn your way around the MPM UI