Lisp library for creating a confusion matrix, incrementally adding information, and retrieving statistical information.
Overview
A confusion matrix represents "the relative frequencies with which each of a number of stimuli is mistaken for each of the others by a person in a task requiring recognition or identification of stimuli" (R. Colman, A Dictionary of Psychology, 2008). Each row represents the predicted label of an instance, and each column represents the observed label of that instance. Numbers at each (row, column) reflect the total number of instances of predicted label "row" which were observed as having label "column".
A twoclass example is:
Observed Observed  Positive Negative  Predicted + a b  Positive c d  Negative
Here the value:
 a

is the number of true positives (those predicted positive and observed positive)
 b

is the number of false negatives (those predicted positive but observed negative)
 c

is the number of false positives (those predicted negative but observed positive)
 d

is the number of true negatives (those predicted negative and observed negative)
From this table we can calculate statistics like:
 true_positive_rate

a/(a+b)
 positive recall

a/(a+c)
As statistics can also be calculated for the negative label, e.g. the true
negative rate is d/(c+d), the functions below have an optional :labelfor
key, to specify which label they are calculated for: the default is to report
for the first label named when the matrix is created
The implementation supports confusion matrices with more than two labels. When more than two labels are in use, the statistics are calculated as if the first, or named, label were positive and all the other labels are grouped as if negative.
For example, in a threeclass example:
Observed Observed Observed  Red Blue Green  Predicted + a b c  Red d e f  Blue g h i  Green
We can calculate:
 true_red_rate

a/(a+b+c)
 red recall

a/(a+d+g)
Install and Test
To install, download the latest release and make the code available to the asdf system.
The API documentation is best built using baruch.
To test:
* (asdf:testsystem :confusionmatrix)
Example
The following example creates a simple twolabel confusion matrix, prints a few statistics and displays the table.
The full package name is "confusionmatrix", with the nickname "cm". 
(require 'asdf)
(require 'confusionmatrix)
(defvar cm (cm:makeconfusionmatrix :labels '(:pos :neg)))
(cm:confusionmatrixadd cm :pos :pos 10)
(cm:confusionmatrixadd cm :pos :neg 3)
(cm:confusionmatrixadd cm :neg :neg 20)
(cm:confusionmatrixadd cm :neg :pos 5)
(format t "Confusion Matrix~%~%~a~%" cm)
(format t "Precision: ~f~&" (cm:precision cm))
(format t "Recall: ~f~&" (cm:recall cm))
(format t "MCC: ~f~&" (cm:matthewscorrelation cm))
which outputs:
> sbcl script exampleconfusionmatrix.lisp Confusion Matrix Observed  POS NEG  Predicted + 10 3  POS 5 20  NEG Precision: 0.6666667 Recall: 0.7692308 MCC: 0.55248505
Structure
(makeconfusionmatrix &optional labels)

labels
 the labels used for the predicted values. By default, these are (:positive :negative)
(confusionmatrixp cm)
Returns T
if cm
is a confusion matrix, or NIL
if it is not.
A confusion matrix will display in a tabulated format when used in a print
or
format
or similar statement, as shown in the example above.
Functions
confusionmatrixadd (cm predicted observed &optional total)
Adds 1, or given total, to the value in the (predicted, observed) cell.
Example confusion matrix:
> sbcl * (require 'asdf) * (require 'confusionmatrix) * (defvar cm (cm:makeconfusionmatrix)) * (cm:confusionmatrixadd cm :positive :positive 2) * (cm:confusionmatrixadd cm :positive :negative 5) * (cm:confusionmatrixadd cm :negative :positive 1) * (cm:confusionmatrixadd cm :negative :negative 3) * cm Observed  POSITIVE NEGATIVE  Predicted + 2 5  POSITIVE 1 3  NEGATIVE
confusionmatrixcount (cm predicted observed)
Returns the current count for (predicted, observed) pair.
Using the confusion matrix from confusionmatrixadd
:
* (cm:confusionmatrixcount cm :positive :positive) 2
Returns the total number of instances referenced in the confusion matrix.
Using the confusion matrix from confusionmatrixadd
:
* (cm:confusionmatrixtotal cm) 11
cohenkappa (cm &key forlabel)
Returns Cohen’s Kappa Statistic, which is a measure of the quality of binary classification.
Using the confusion matrix from confusionmatrixadd
:
* (cm:cohenkappa cm) 1/34
falsenegative (cm &key forlabel)
Returns the number of instances of the given label which are incorrectly observed.
Using the confusion matrix from confusionmatrixadd
:
* (cm:falsenegative cm) ; assumes :positive 5 * (cm:falsenegative cm :forlabel :positive) 5 * (cm:falsenegative cm :forlabel :negative) 1
falsepositive cm &key forlabel)
Returns the number of incorrectly observed instances with the given label.
Using the confusion matrix from confusionmatrixadd
:
* (cm:falsepositive cm) 1 * (cm:falsepositive cm :forlabel :negative) 5
falserate cm &key forlabel)
Returns the proportion of instances with the given label which are incorrectly observed, out of all those instances of that label.
Using the confusion matrix from confusionmatrixadd
:
* (cm:falserate cm) 1/4 * (cm:falserate cm :forlabel :negative) 5/7
fmeasure (cm &key forlabel)
Returns the fmeasure for a given label  the harmonic mean of the precision and recall for that label.
Using the confusion matrix from confusionmatrixadd
:
* (cm:fmeasure cm) 2/5 * (cm:fmeasure cm :forlabel :negative) 1/2
geometricmean (cm)
Returns the geometric mean of the true rates for each label: the geometric mean is the n^{th} root of the product of the true rates, where n is the number of labels.
Using the confusion matrix from confusionmatrixadd
:
* (cm:geometricmean cm) 0.46291006
matthewscorrelation (cm &key forlabel)
Returns Matthew’s Correlation Coefficient, which is a measure of the quality of binary classification.
Using the confusion matrix from confusionmatrixadd
:
* (cm:matthewscorrelation cm) 0.03857584
overallaccuracy (cm)
Returns the proportion of instances which are correctly observed.
Using the confusion matrix from confusionmatrixadd
:
* (cm:overallaccuracy cm) 5/11
precision (cm &key forlabel)
Returns the proportion of instances observed as the given label which are correct.
Using the confusion matrix from confusionmatrixadd
:
* (cm:precision cm) 2/3 * (cm:precision cm :forlabel :negative) 3/8
prevalence (cm &key forlabel)
Returns the proportion of instances of the given label, out of the total.
Using the confusion matrix from confusionmatrixadd
:
* (cm:prevalence cm) 7/11 * (cm:prevalence cm :forlabel :negative) 4/11
recall (cm &key forlabel)
Recall is another name for the true rate for that label (see truerate
).
sensitivity (cm &key forlabel)
Sensitivity is another name for the true rate (recall) for that label
(see truerate
).
specificity (cm &key forlabel)
Returns 1(falserate for that label).
truenegative (cm &key forlabel)
Returns the number of instances not of the given predicted label which are correctly observed.
Using the confusion matrix from confusionmatrixadd
:
* (cm:truenegative cm) 3 * (cm:truenegative cm :forlabel :negative) 2
truepositive (cm &key forlabel)
Returns the number of instances of the given label which are correctly observed.
Using the confusion matrix from confusionmatrixadd
:
* (cm:truepositive cm) 2 * (cm:truepositive cm :forlabel :negative) 3
truerate (cm &key forlabel)
Returns the proportion of instances of the given label which are correctly observed.
Using the confusion matrix from confusionmatrixadd
:
* (cm:truerate cm) 2/7 * (cm:truerate cm :forlabel :negative) 3/4