I can’t figure out if I’ve setup my binary classification problem correctly. I labeled the positive class 1 and the negative 0. However It is my understanding that by default scikit-learn uses class 0 as the positive class in its confusion matrix (so the inverse of how I set it up). This is confusing to me. Is the top row, in scikit-learn’s default setting, the positive or negative class?
Lets assume the confusion matrix output:

confusion_matrix(y_test, preds)
 [ [30  5]
    [2 42] ]

How would it look like in a confusion matrix? Are the actual instances the rows or the columns in scikit-learn?

          prediction                        prediction
           0       1                          1       0
         -----   -----                      -----   -----
      0 | TN   |  FP        (OR)         1 |  TP  |  FP
actual   -----   -----             actual   -----   -----
      1 | FN   |  TP                     0 |  FN  |  TN

scikit learn sorts labels in ascending order, thus 0’s are first column/row and 1’s are the second one

>>> from sklearn.metrics import confusion_matrix as cm
>>> y_test = [1, 0, 0]
>>> y_pred = [1, 0, 0]
>>> cm(y_test, y_pred)
array([[2, 0],
       [0, 1]])
>>> y_pred = [4, 0, 0]
>>> y_test = [4, 0, 0]
>>> cm(y_test, y_pred)
array([[2, 0],
       [0, 1]])
>>> y_test = [-2, 0, 0]
>>> y_pred = [-2, 0, 0]
>>> cm(y_test, y_pred)
array([[1, 0],
       [0, 2]])
>>> 

This is written in the docs:

labels : array, shape = [n_classes], optional
List of labels to index the matrix. This may be used to reorder or select a subset of labels. If none is given, those that appear at least once in y_true or y_pred are used in sorted order.

Thus you can alter this behavior by providing labels to confusion_matrix call

>>> y_test = [1, 0, 0]
>>> y_pred = [1, 0, 0]
>>> cm(y_test, y_pred)
array([[2, 0],
       [0, 1]])
>>> cm(y_test, y_pred, labels=[1, 0])
array([[1, 0],
       [0, 2]])

And actual/predicted are oredered just like in your images – predictions are in columns and actual values in rows

>>> y_test = [5, 5, 5, 0, 0, 0]
>>> y_pred = [5, 0, 0, 0, 0, 0]
>>> cm(y_test, y_pred)
array([[3, 0],
       [2, 1]])
  • true: 0, predicted: 0 (value: 3, position [0, 0])
  • true: 5, predicted: 0 (value: 2, position [1, 0])
  • true: 0, predicted: 5 (value: 0, position [0, 1])
  • true: 5, predicted: 5 (value: 1, position [1, 1])

Supporting Answer:

When drawing the confusion matrix values using sklearn.metrics, be aware that the order of the values are

[ True Negative False positive]
[ False Negative True Positive ]

If you interpret the values wrong, say TP for TN, your accuracies and AUC_ROC will more or less match, but your precision, recall, sensitivity, and f1-score will take a hit and you will end up with completely different metrics. This will result in you making a false judgement of your model’s performance.

Do make sure to clearly identify what the 1 and 0 in your model represent. This heavily dictates the results of the confusion matrix.

Experience:

I was working on predicting fraud (binary supervised classification), where fraud was denoted by 1 and non-fraud by 0. My model was trained on a scaled up, perfectly balanced data set, hence during in-time testing, values of confusion matrix did not seem suspicious when my results were of the order
[TP FP]
[FN TN]

Later, when I had to perform an out-of-time test on a new imbalanced test set, I realized that the above order of confusion matrix was wrong and different from the one mentioned on sklearn’s documentation page which refers to the order as tn,fp,fn,tp. Plugging in the new order made me realize the blunder and what a difference it had caused in my judgement of the model’s performance.

Following the example of wikipedia. If a classification system has been trained to distinguish between cats and non cats, a confusion matrix will summarize the results of testing the algorithm for further inspection. Assuming a sample of 27 animals — 8 cats, and 19 non cats, the resulting confusion matrix could look like the table below:

enter image description here

With sklearn

If you want to maintain the structure of the wikipedia confusion matrix, first go the predicted values and then the actual class.

from sklearn.metrics import confusion_matrix
y_true = [0,0,0,1,0,0,1,0,0,1,0,1,0,0,0,0,1,0,0,1,1,0,1,0,0,0,0]
y_pred = [0,0,0,1,0,0,1,0,0,1,0,1,0,0,0,0,1,0,0,0,0,1,0,1,0,0,0]
confusion_matrix(y_pred, y_true, labels=[1,0])

Out[1]: 
array([[ 5,  2],
       [ 3, 17]], dtype=int64)

Another way with crosstab pandas

true = pd.Categorical(list(np.where(np.array(y_true) == 1, 'cat','non-cat')), categories = ['cat','non-cat'])
pred = pd.Categorical(list(np.where(np.array(y_pred) == 1, 'cat','non-cat')), categories = ['cat','non-cat'])

pd.crosstab(pred, true, 
            rownames=['pred'], 
            colnames=['Actual'], margins=False, margins_name="Total")

Out[2]: 
Actual   cat  non-cat
pred                 
cat        5        2
non-cat    3       17

Short answer
In binary classification, when using the argument labels ,

confusion_matrix([0, 1, 0, 1], [1, 1, 1, 0], labels=[0,1]).ravel()

the class labels, 0, and 1, are considered to be Negative and Positive, respectively. This is due to the order implied by the list, and not the alpha-numerical order.


Verification:
Consider imbalanced class labels like this: (using imbalance class to make the distinction easier)

>>> y_true = [0,0,0,1,0,0,0,0,0,1,0,0,1,0,0,0]
>>> y_pred = [0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0]
>>> table = confusion_matrix(y_true, y_pred, labels=[0,1]).ravel()

this would give you a confusion table as follows:

>>> table
array([12,  1,  2,  1])

which corresponds to:

              Actual
        |   1   |   0  |
     ___________________
pred  1 |  TP=1 | FP=1 |
      0 |  FN=2 | TN=12|

where FN=2 means that there were 2 cases where the model predicted the sample to be negative (i.e., 0) but the actual label was positive (i.e., 1), hence False Negative equals 2.

Similarly for TN=12, in 12 cases the model correctly predicted the negative class (0), hence True Negative equals 12.

This way everything adds up assuming that sklearn considers the first label (in labels=[0,1] as the negative class. Therefore, here, 0, the first label, represents the negative class.

I think what we decide on our end to call “positive” or “negative” is a matter of choice and sklearn is NOT aware of that. You can label your data points any way you want (not just 0 and 1) so your statement that “sklearn uses 0 as ‘positive'” – or vice versa – simply does not hold.
If you do not specifically use ‘labels’ parameters to specify order of row and column labels then sklearn will sort them alphanumerically. So you can output your confusion matrix any way you want and you can decide what you call (in the simplest binary case) “positive/negative”. sklearn does not make those decisions for you.