Monitors
For Tabular projects
Monitors let you easily set custom alerts to track your model's health and performance over time, ensuring your models are on track. Once configured, AryaXAI can notify users when it detects identified drift in the data, enabling proactive intervention to maintain model accuracy and reliability. Users can also choose a custom compute required for the alert.
You can set monitors to detect data and target drift or model performance degradation. To install the package, you can use the following functions:
To list all monitoring Triggers created:
# list monitoring triggers
project.monitoring_triggers()
You can also use the help function to create monitoring triggers for Data Drift, Target Drift, and Model Performance using payload
help(project.create_monitoring_trigger)
To create duplicate monitors:
project.duplicate_monitoring_triggers(trigger_name='', new_trigger_name='')
To retrieve alerts for a specific time interval by specifying the duration in hours
project.get_monitors_alerts(monitor_id="6834455e2c0411e6b95d997b", time=2)
For Image projects
For image projects, AryaXAI offers 5different types of Monitors:
- Image Dataset Drift: The overall distribution of images changes compared to the original training data.
(Example: Training images are mostly sunny, but new images are mostly rainy.)
- Label Drift: The distribution of output labels (classes) changes over time.
(Example: A disease detection model sees more COVID-19 cases now than before.)
- Model Performance: How well the model is doing (accuracy, precision, recall, etc.) on new data.
(Example: Accuracy drops when the input images change.)
- Image Property Drift: Changes in specific properties of images like brightness, sharpness, size, or color.
(Example: New images are darker than the training ones.)
- Property Label Correlation: How much image properties are linked to the labels.
(Example: If darker images are mostly classified as “night scenes.”)
You can set monitors to detect data and target drift or model performance degradation. To install the package, you can use the following functions:
Use the following function to retrieve Label Drift:
payload = {
"trigger_name": "labeldrift",
"trigger_type":"Label Drift",
"mail_list": ['apurv.harkhani@arya.ai'],
"frequency": "daily",
"stat_test_threshold": 1,
"model_type": "classification",
"base_line_tag": ['training'],
"current_tag": ["val"],
"instance_type": "xsmall",
"priority":1
}
project.create_monitoring_trigger(payload = payload)
To generate the Label Drift Dashboard:
project.get_label_drift_dashboard(
payload={
"base_line_tag":["cifar1_testdata"],
"current_tag":["cifar2_testdata"]
},
instance_type="small"
)
Use the following code to retrieve Image Dataset Drift
payload = {
"trigger_name": "imagedatasetdrift",
"trigger_type":"Image Dataset Drift",
"mail_list": ['apurv.harkhani@arya.ai'],
"frequency": "daily",
"stat_test_threshold": 1,
"model_type": "classification",
"base_line_tag": ['training'],
"current_tag": ["val"],
"instance_type": "xsmall",
"priority":10
}
project.create_monitoring_trigger(payload = payload)
To generate the Image Dataset Drift Dashboard:
payload = {
"trigger_name": "propertylabelcoorelation",
"trigger_type":"Property Label Correlation",
"mail_list": ['apurv.harkhani@arya.ai'],
"frequency": "daily",
"stat_test_threshold": 0.001,
"model_type": "classification",
"base_line_tag": ['training'],
"current_tag": ["val"],
"instance_type": "xsmall",
"priority":10,
"label_correlation_property": "Area"
}
project.create_monitoring_trigger(payload = payload)
Use the following code to retrieve Property Label Correlation
payload = {
"trigger_name": "propertylabelcoorelation",
"trigger_type":"Property Label Correlation",
"mail_list": ['apurv.harkhani@arya.ai'],
"frequency": "daily",
"stat_test_threshold": 0.001,
"model_type": "classification",
"base_line_tag": ['training'],
"current_tag": ["val"],
"instance_type": "xsmall",
"priority":10,
"label_correlation_property": "Area"
}
project.create_monitoring_trigger(payload = payload)
To generate Property Label Correlation Dashboard:
project.get_property_label_correlation_dashboard(
payload={
"base_line_tag":["cifar1_testdata"],
"current_tag":["cifar2_testdata"]
},
instance_type="small"
)
Use the following code to retrieve Image Property Drift
payload = {
"trigger_name": "propertydrift1",
"trigger_type":"Image Property Drift",
"mail_list": ['apurv.harkhani@arya.ai'],
"frequency": "daily",
"stat_test_threshold": 1,
"model_type": "classification",
"base_line_tag": ['training'],
"current_tag": ["val"],
"instance_type": "xsmall",
"priority":10,
"image_drift_property": "Area"}
project.create_monitoring_trigger(payload = payload)
Use project.get_score() to retrieve the drift score for a specific feature from a drift monitoring dashboard.
It accepts the following parameters:
• dashboard_id – The unique identifier of the drift monitoring dashboard.
• feature_name – The name of the feature for which the drift score is to be fetched.
project.get_score(dashboard_id="682dccd887ec65d1cbe2f355",feature_name="funded_amnt_inv")
Use the following function to retrieve Model Performance:
payload = {
"trigger_name": "model performance6",
"trigger_type":"Model Performance",
"mail_list": ['apurv.harkhani@arya.ai'],
"frequency": "daily",
"model_type": "classification",
"current_tag": ["val"],
"instance_type": "small",
"priority":10,
"model_performance_threshold":1,
"model_name": "cifar_1",
"model_performance_metric": "accuracy"
}
project.create_monitoring_trigger(payload = payload)
Alerts
Monitors that have been previously set up will appear as alerts once triggered. AryaXAI alerts enable users to get detailed root cause analysis of triggered alerts and pinpoint factors contributing to model degradation. Users can set up alerts to detect data and target drift, performance degradation, anomalies, etc.
To retrieve alerts for a specific time interval, provide the following inputs:
• Monitor ID – The unique identifier of the monitor.
• Hours – The number of past hours for which alerts should be fetched.
project.get_monitors_alerts(monitor_id="682341c8d1ff7f25ceffcabd", time=14)
Delete the monitoring trigger:
# delete monitoring trigger
project.delete_monitoring_trigger('test trigger 5')
Fetch details of executed triggers:
# Fetch details of executed Triggers
project.alerts()
View email notifications
When setting up monitors, users can specify the email addresses where they want to receive notifications for triggered alerts. For example:
{
"trigger_type": "" #["Data Drift", "Target Drift", "Model Performance"]
"trigger_name": "",
"mail_list": [],
"frequency": "", #['daily','weekly','monthly','quarterly','yearly']
"stat_test_name": "",
"stat_test_threshold": 0,
"datadrift_features_per": 7,
"features_to_use": [],
"date_feature": "",
"baseline_date": { "start_date": "", "end_date": ""},
"current_date": { "start_date": "", "end_date": ""},
"base_line_tag": "",
"current_tag": ""
}
Here, 'mail_list' is where you specify the email addresses of the users who want to be notified.
