Lisp library for creating a confusion matrix, incrementally adding information, and retrieving statistical information.
Overview
A confusion matrix represents "the relative frequencies with which each of a number of stimuli is mistaken for each of the others by a person in a task requiring recognition or identification of stimuli" (R. Colman, A Dictionary of Psychology, 2008). Each row represents the predicted label of an instance, and each column represents the observed label of that instance. Numbers at each (row, column) reflect the total number of instances of predicted label "row" which were observed as having label "column".
A two-class example is:
Observed Observed |
Positive Negative | Predicted
------------------------------+------------
a b | Positive
c d | Negative
Here the value:
- a
-
is the number of true positives (those predicted positive and observed positive)
- b
-
is the number of false negatives (those predicted positive but observed negative)
- c
-
is the number of false positives (those predicted negative but observed positive)
- d
-
is the number of true negatives (those predicted negative and observed negative)
From this table we can calculate statistics like:
- true_positive_rate
-
a/(a+b)
- positive recall
-
a/(a+c)
As statistics can also be calculated for the negative label, e.g. the true
negative rate is d/(c+d), the functions below have an optional :label-for
key, to specify which label they are calculated for: the default is to report
for the first label named when the matrix is created
The implementation supports confusion matrices with more than two labels. When more than two labels are in use, the statistics are calculated as if the first, or named, label were positive and all the other labels are grouped as if negative.
For example, in a three-class example:
Observed Observed Observed |
Red Blue Green | Predicted
--------------------------------------------+------------
a b c | Red
d e f | Blue
g h i | Green
We can calculate:
- true_red_rate
-
a/(a+b+c)
- red recall
-
a/(a+d+g)
Install and Test
To install, download the latest release and make the code available to the asdf system.
The API documentation is best built using baruch.
To test:
* (asdf:test-system :confusion-matrix)
Example
The following example creates a simple two-label confusion matrix, prints a few statistics and displays the table.
|
The full package name is "confusion-matrix", with the nickname "cm". |
(require 'asdf)
(require 'confusion-matrix)
(defvar cm (cm:make-confusion-matrix :labels '(:pos :neg)))
(cm:confusion-matrix-add cm :pos :pos 10)
(cm:confusion-matrix-add cm :pos :neg 3)
(cm:confusion-matrix-add cm :neg :neg 20)
(cm:confusion-matrix-add cm :neg :pos 5)
(format t "Confusion Matrix~%~%~a~%" cm)
(format t "Precision: ~f~&" (cm:precision cm))
(format t "Recall: ~f~&" (cm:recall cm))
(format t "MCC: ~f~&" (cm:matthews-correlation cm))which outputs:
> sbcl --script example-confusion-matrix.lisp Confusion Matrix Observed | POS NEG | Predicted ---------+---------- 10 3 | POS 5 20 | NEG Precision: 0.6666667 Recall: 0.7692308 MCC: 0.55248505
Structure
(make-confusion-matrix &optional labels)
-
labels- the labels used for the predicted values. By default, these are (:positive :negative)
(confusion-matrix-p cm)
Returns T if cm is a confusion matrix, or NIL if it is not.
A confusion matrix will display in a tabulated format when used in a print or
format or similar statement, as shown in the example above.
Functions
confusion-matrix-add (cm predicted observed &optional total)
Adds 1, or given total, to the value in the (predicted, observed) cell.
Example confusion matrix:
> sbcl * (require 'asdf) * (require 'confusion-matrix) * (defvar cm (cm:make-confusion-matrix)) * (cm:confusion-matrix-add cm :positive :positive 2) * (cm:confusion-matrix-add cm :positive :negative 5) * (cm:confusion-matrix-add cm :negative :positive 1) * (cm:confusion-matrix-add cm :negative :negative 3) * cm Observed | POSITIVE NEGATIVE | Predicted ------------------+---------- 2 5 | POSITIVE 1 3 | NEGATIVE
confusion-matrix-count (cm predicted observed)
Returns the current count for (predicted, observed) pair.
Using the confusion matrix from confusion-matrix-add:
* (cm:confusion-matrix-count cm :positive :positive) 2
Returns the total number of instances referenced in the confusion matrix.
Using the confusion matrix from confusion-matrix-add:
* (cm:confusion-matrix-total cm) 11
cohen-kappa (cm &key for-label)
Returns Cohen’s Kappa Statistic, which is a measure of the quality of binary classification.
Using the confusion matrix from confusion-matrix-add:
* (cm:cohen-kappa cm) 1/34
false-negative (cm &key for-label)
Returns the number of instances of the given label which are incorrectly observed.
Using the confusion matrix from confusion-matrix-add:
* (cm:false-negative cm) ; assumes :positive 5 * (cm:false-negative cm :for-label :positive) 5 * (cm:false-negative cm :for-label :negative) 1
false-positive cm &key for-label)
Returns the number of incorrectly observed instances with the given label.
Using the confusion matrix from confusion-matrix-add:
* (cm:false-positive cm) 1 * (cm:false-positive cm :for-label :negative) 5
false-rate cm &key for-label)
Returns the proportion of instances with the given label which are incorrectly observed, out of all those instances of that label.
Using the confusion matrix from confusion-matrix-add:
* (cm:false-rate cm) 1/4 * (cm:false-rate cm :for-label :negative) 5/7
f-measure (cm &key for-label)
Returns the f-measure for a given label - the harmonic mean of the precision and recall for that label.
Using the confusion matrix from confusion-matrix-add:
* (cm:f-measure cm) 2/5 * (cm:f-measure cm :for-label :negative) 1/2
geometric-mean (cm)
Returns the geometric mean of the true rates for each label: the geometric mean is the nth root of the product of the true rates, where n is the number of labels.
Using the confusion matrix from confusion-matrix-add:
* (cm:geometric-mean cm) 0.46291006
matthews-correlation (cm &key for-label)
Returns Matthew’s Correlation Coefficient, which is a measure of the quality of binary classification.
Using the confusion matrix from confusion-matrix-add:
* (cm:matthews-correlation cm) 0.03857584
overall-accuracy (cm)
Returns the proportion of instances which are correctly observed.
Using the confusion matrix from confusion-matrix-add:
* (cm:overall-accuracy cm) 5/11
precision (cm &key for-label)
Returns the proportion of instances observed as the given label which are correct.
Using the confusion matrix from confusion-matrix-add:
* (cm:precision cm) 2/3 * (cm:precision cm :for-label :negative) 3/8
prevalence (cm &key for-label)
Returns the proportion of instances of the given label, out of the total.
Using the confusion matrix from confusion-matrix-add:
* (cm:prevalence cm) 7/11 * (cm:prevalence cm :for-label :negative) 4/11
recall (cm &key for-label)
Recall is another name for the true rate for that label (see true-rate).
sensitivity (cm &key for-label)
Sensitivity is another name for the true rate (recall) for that label
(see true-rate).
specificity (cm &key for-label)
Returns 1-(false-rate for that label).
true-negative (cm &key for-label)
Returns the number of instances not of the given predicted label which are correctly observed.
Using the confusion matrix from confusion-matrix-add:
* (cm:true-negative cm) 3 * (cm:true-negative cm :for-label :negative) 2
true-positive (cm &key for-label)
Returns the number of instances of the given label which are correctly observed.
Using the confusion matrix from confusion-matrix-add:
* (cm:true-positive cm) 2 * (cm:true-positive cm :for-label :negative) 3
true-rate (cm &key for-label)
Returns the proportion of instances of the given label which are correctly observed.
Using the confusion matrix from confusion-matrix-add:
* (cm:true-rate cm) 2/7 * (cm:true-rate cm :for-label :negative) 3/4