By now, most of us probably realize that there is incredible power in predictive models and machine learning models. These models can help predict our customers’ future behavior before it happens, create individualized customer experiences, and help us connect to our customers on a deeper level.

However, did you know that every predictive model comes with its own set of tradeoffs? That’s right, many times, by maximizing predictive accuracy in one area, you can actually increase your error in another area.

I could rant about this all day long, but this is one reason why I don’t tend to like the “out-of-the-box” predictive models that are embedded in many modern software packages. These canned models don’t take into account a particular business’ unique needs and goals for the model. They simply apply defaults to many of the decisions that are realistically not set in stone.

They’re all science and no art.

At Sprocket, we custom build all of our predictive models alongside our clients to make sure the judgement calls we’re making (of which there are many in a predictive modeling project) skew in the direction of a predictive model that works best for our client’s unique goals and needs.

By tailoring the way your predictive model skews and building it to be accurate in the direction that will help your specific business need, you can actually get even more business value and predictive capability out of a predictive model.

The answers lie in the confusion matrix. By using this obscurely and horribly-named bit of model output in a smart way, you can make the right set of judgement calls to ensure that your predictive model will meet your unique needs.

Wanna know more? Read on, my friend!

WTH is a confusion matrix?

A confusion matrix is simply a table that describes the performance of a predictive model. In this case, it describes the performance of a predictive model that is used to classify items. Business problems that involve classification are ones such as:

  • Predicting who will attrite in the first 30 days after joining a gym
  • Predicting whether a customer will renew their subscription
  • Predicting if a customer is at risk of defaulting on a payment

In all of these cases, we’re attempting to use a predictive model to put customers into a particular category (i.e., those likely to attrite, those likely to renew, those at risk of default).

When we write a model like this, we use customers’ past behavior to fit a model to predict future customers’ behavior. So, a confusion matrix simply crosses the classification that the model predicted with the actual classification in the dataset that was used to create the model.

Let’s look at an example

Here’s an example of a blank confusion matrix for a classification model that is being created to predict who is at risk of defaulting on a payment.

Confusion Matrix

In this case, the business goal is to identify customers who are likely to default on a payment. In other words, this is a risk model. This particular business is more interested in knowing who is going to default than they are knowing who is going to pay on time. Keep that in mind as you explore a bit deeper.

As you examine this confusion matrix, you’ll notice that there are 2 ways for the model to be accurate (true positives and true negatives), and 2 ways for the model to be inaccurate (false positives and false negatives).

Confusion Matrix Filled In

This total model’s accuracy level would be determined by looking at how often it correctly classifies a true positive and a true negative, versus how often it gets it wrong.

Let’s go deeper into the [confusion] matrix

Here’s the nuance that many businesses miss.

If you stop at simply maximizing total model accuracy as described above (and as many out-of-the-box models do), you will end up with an under-performing model.

In this case, this business is interested in knowing who is likely to default. They’re extremely interested in the true positives. These customers pose financial risk to the business, and there is a significant financial incentive to the business to make sure they see these risky customers coming and manage them before they default.

In contrast, this business is less interested in correctly identifying true negatives. Customers who pay on time pose no financial risk and are not the focus of this model.

April, who cares?

Here’s why it matters: the boxes inside a confusion matrix are dependent upon one another.

There are ways to build a model that actually allow you to put your thumb on the scale to tip it in the direction that better meets your business needs. You can fit a model that maximizes your ability to detect true positives by allowing a few false positives to sneak through the gate.

For example, this business is likely willing to accidentally capture a reasonable number of false positives (people the model predicted to default but who ultimately pay on time) in order to ensure that they’re maximizing the number of true positives they identify.

In contrast, the bottom two boxes of this matrix are of significantly less importance to this fictitious business, and so we wouldn’t attempt to fit a model that skews in the direction of correctly identifying false negatives.

Art & science

One of the most magical things that we Sprocketeers do is force the left and right brain to work in tandem. We force design and economics to play nicely in the sandbox. And, we blend art with science.

This is one beautiful example of that.

Sure, data science has it’s hard and fast rules. If you’re interested in a classification model such as the ones I described, only certain statistical techniques will get you there. A traditional linear regression model isn’t appropriate, for example.

However, once those hard and fast rules are satisfied….the real fun begins.

Your business knowledge, your knowledge of your customers, and your working hypotheses about why certain behaviors might have happened in the past are all inputs we take into account as we create your predictive model.

That is pure art.

By taking into account your unique goal of identifying default payers, and knowing that the risk in mid-identifying someone (false positive) is simply that they’ll receive a few more tailored reminder emails and a phone call, can lead us to make a judgement call to tip the scales in favor of identifying more true positives, at the expense of the rest of the confusion matrix.

Want to learn more?

We’d love to jam with you about your unique business problem! We excel at writing affordable, custom predictive models that are tailored to meet your unique business needs and goals. And darn it, we’re fun to work with, too!

Drop us a line, and let’s chat about how we can begin blending art and science to help you achieve your goals!