This is your AI metrics cheat sheet:

Classification Metrics: When sorting data into categories.

• Precision: Out of all the items the model labeled as positive, how many truly are?
  - Think of it as the model’s trustworthiness when it claims something is positive.

• Recall (Sensitivity): Out of all actual positive items, how many did the model correctly identify?
  - It’s about not missing out on true positives.

• F1 Score: The balance between Precision and Recall.
  - When you want a single metric that considers both false positives and false negatives.

• AUC-ROC: The model’s ability to distinguish between classes.
  - A score of 1.0 is perfect. Above 0.9 is excellent. Below 0.5 means the model is worse than random guessing.

Regression Metrics: When predicting continuous values.

• Mean Absolute Error (MAE): The average difference between predicted and actual values.
  - Lower is better. If MAE is zero, be cautious: it could indicate overfitting or other issues.

• Root Mean Squared Error (RMSE): Like MAE, but punishes large errors more.
  - Critical when significant mispredictions can have major consequences.

• R-squared: How well the model’s predictions match the real data.
  - Ranges from 0 to 1. Closer to 1 means the model explains more of the variability.

Clustering Metrics: When you’re grouping data.

• Silhouette Score: Measures how similar items are within a cluster compared to other clusters.
  - Ranges from -1 to 1. Higher values indicate better-defined clusters.

• Davies-Bouldin Index: Lower values indicate better partitioning of clusters.
  - Zero is the theoretical ideal score.

Data Quality Metrics:

• Completeness: Percentage of non-missing data points.
  - Aim for 100%, but be wary of artificially complete data.

• Consistency: Ensuring data doesn’t contradict itself.
  - Inconsistent data can lead to misleading model results.

• Outlier Detection: Identify data points that deviate significantly.
  - Outliers can skew model training.

Model Robustness:

• Cross-Validation Score: A measure of a model’s performance on different subsets of data.
  - Ensures the model isn’t just memorizing the training data.

• Bias-Variance Tradeoff: Balance between the model’s adaptability and its ability to generalize to new data.
  - Ideally a model that captures patterns without overfitting or being overly simplistic.

Model Interpretability:

• Feature Importance: Identifies the input variables that have the most influence on the model’s predictions.
  - Understanding what the model deems important.

• SHAP Values: Breaks down the contribution of each feature to specific model predictions.
  - Clarifies the reasoning behind individual predictions.

Overfitting & Underfitting: Ensuring the model performs well on both training and unseen data.

• Training vs. Validation Error: A low training error coupled with a high validation error suggests potential overfitting.

• Overfitting: Model performs well on training data but poorly on new data.

• Underfitting: Model performs poorly on both training and new data.

This guide serves as a simple starting point for how to measure the effectiveness of different models!