Usage

Everything you have to know to master guap is here. Buckle up! 🚀

The first job of guap is to evaluate how much value an AI/ML model will create (profit or savings) based on the cost matrix.

The setup

To generate guap scores, you'll need:

• A trained and tested model

• Expected values (profit or savings) for each scenario

First, make sure guap is installed and up to date.

pip install guap

Now you will need to import it, using this following line, with all his pandas, sklearn, numpy friends. Maybe? 🤓

from guap.metrics import guap_metric, binary_guap_metric

You will use :

• guap_metric for Multiclass classification

• binary_guap_metric for binary classification

Then, after you've done your data work, including the split, you'll need to train and test the method of your choice. Let's take a logistic regression, from our guap demo.

lr = LogisticRegression()
lr.fit(X_train, y_train)
lr_yhat = lr.predict(X_test)

When it's done, it's guap time!

Set the cost matrix

This is where all the ✨magic✨ happens.

In an overwhelming number of circumstances, both in textbooks and in practice, the performance of a model is evaluated where the cost of TN, FP, FN, and TP are taken to be equal.

But in most cases, nothing could be farther from the truth! There are drastically different costs and benefits from each classification.

It's also a great practice to think about profit/cost at the setup stage of an ML project, to make sure everyone is aligned on the expected opportunities and risks you may encounter later on with your usecase.

Set your cost matrix, replace each of your scenarios in this order : [[TN, FP], [FN, TP]]

cost_matrix = [[200, -1500], [0, 1000]]

Generate the metrics

Now you can invoke guap metrics, using your confusion matrix and the cost matrix, you'll have in output our profit scores!

output = guap_metric(y_true, y_pred, cost_matrix)
output

guap scores

For now, we have two metrics.

• Get the total profit. Based on the test set, guap will give you the total expected profit based on the cost matrix. A great way to have an overview of the model profitability.

• Average profit per prediction. Along with the total profit score, guap will give you the average profit/cost per prediction. Perfect if you have costs per prediction, or if you need to estimate the profitability while scaling (for multi-class only).

Check our public roadmap on GitHub to learn what's coming next!