{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "和回归模型一样，这一节介绍一些常用的方法来评估分类模型的好坏。\n",
    "\n",
    "## 错误率和精度\n",
    "\n",
    "错误率指分类错误的样本数占样本总数的比例：\n",
    "\n",
    "$$\n",
    "E(f;D) = \\frac{1}{m} \\sum_{i=1}^m I(f(x_i) \\neq y_i)\n",
    "$$\n",
    "\n",
    "精度和错误率相反，指的是分类正确的样本数占样本总数的比例：\n",
    "\n",
    "$$\n",
    "acc(f;D) = \\frac{1}{m} \\sum_{i=1}^m I(f(x_i) = y_i) = 1 - E(f;D)\n",
    "$$\n",
    "\n",
    "## 查准率和查全率\n",
    "\n",
    "对于二分类问题，可以将样本的真实类别和模型的识别结果组成四种情形：\n",
    "\n",
    "* True Positive\n",
    "  * 真正类(TP)，样本的真实类别是正类，并且模型识别的结果也是正类。\n",
    "* False Negative\n",
    "  * 假负类(FN)，样本的真实类别是正类，但是模型将其识别成为负类。\n",
    "* False Positive\n",
    "  * 假正类(FP)，样本的真实类别是负类，但是模型将其识别成为正类。\n",
    "* True Negative\n",
    "  * 真负类(TN)，样本的真实类别是负类，并且模型将其识别成为负类。\n",
    "  \n",
    "将这四种情形写成矩阵形式如下：\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <th rowspan=\"2\">真实情况</th>\n",
    "        <th colspan=\"2\">预测结果</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <th>正例</th> \n",
    "        <th>反例</th> \n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>正例</td> \n",
    "        <td>TP（真正例）</td> \n",
    "        <td>FN（假反例）</td> \n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>反例</td> \n",
    "        <td>FP（假正例）</td> \n",
    "        <td>TN（真反例）</td> \n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "这个矩阵被称为 **混淆矩阵**（confusion matrix）。\n",
    "\n",
    "**查准率**（precision）也叫做 **准确率**，定义如下：\n",
    "\n",
    "$$\n",
    "P = \\frac{TP}{TP + FP}\n",
    "$$\n",
    "\n",
    "**查全率**（recall）也叫做 **召回率**，定义如下：\n",
    "\n",
    "$$\n",
    "R = \\frac{TP}{TP + FN}\n",
    "$$\n",
    "\n",
    "查准率和查全率是一对矛盾的度量，一般情况下，查准率高时，查全率会偏低，查全率高时，查准率会偏低。以查准率为纵轴，查全率为横轴，可以画出查准率-查全率曲线，简称为 **P-R曲线**：\n",
    "\n",
    "![](../images/pr-curve.webp)\n",
    "\n",
    "如果一个学习器的 P-R 曲线把另一个学习器的 P-R 曲线包住，则可以断言第一个学习器要好，比如上图中 A 和 B 的性能要大于 C。\n",
    "\n",
    "如果两个学习器的 P-R 曲线发生了交叉，如上图中 A 和 B，这时两个学习器的性能就不好比较了，通常有几种不同的方法：\n",
    "\n",
    "* 比较 P-R 曲线下面积的大小，但这个面积不太容易估算\n",
    "* 比较 BEP 大小，BEP 是 **平衡点**（Break-Even Point）的简称，它是 查准率 = 查全率 时的取值，比如上图中 A 的 BEP = 0.8，B 的 BEP = 0.72，所以 A 优于 B\n",
    "* 比较 $F_1$ 大小，根据 $F_1$ 还可以推广到一般形式 $F_\\beta$，可以根据实际情况调整查准率和查全率的相对重要性\n",
    "\n",
    "$F_1$ 定义如下：\n",
    "\n",
    "$$\n",
    "F_1 = \\frac{2 \\times P \\times R}{P + R}\n",
    "$$\n",
    "\n",
    "$F_1$ 是基于查准率和查全率的 **调和平均**（harmonic mean）定义出来的：\n",
    "\n",
    "$$\n",
    "\\frac{1}{F_1} = \\frac{1}{2} (\\frac{1}{P} + \\frac{1}{R})\n",
    "$$\n",
    "\n",
    "$F_\\beta$ 定义如下：\n",
    "\n",
    "$$\n",
    "F_\\beta = \\frac{(1 + \\beta^2) \\times P \\times R}{(\\beta^2 + P) + R}\n",
    "$$\n",
    "\n",
    "$F_\\beta$ 是基于查准率和查全率的 **加权调和平均** 定义出来的：\n",
    "\n",
    "$$\n",
    "\\frac{1}{F_\\beta} = \\frac{1}{1 + \\beta^2} (\\frac{1}{P} + \\frac{\\beta^2}{R})\n",
    "$$\n",
    "\n",
    "> 和算术平均（$\\frac{P+R}{2}$）和几何平均（$\\sqrt{P \\times R}$）相比，调和平均更重视较小值。\n",
    "\n",
    "## ROC 与 AUC\n",
    "\n",
    "从混淆矩阵中还可以定义出 **真正例率**（True Positive Rate，简称 TPR）和 **假正例率**（False Positive Rate，简称 FPR）：\n",
    "\n",
    "$$\n",
    "TPR = \\frac{TP}{TP + FN}\n",
    "$$\n",
    "\n",
    "$$\n",
    "FPR = \\frac{FP}{TN + FP}\n",
    "$$\n",
    "\n",
    "将 TPR 作为纵轴，FPR 作为横轴，可以画出 **ROC 曲线**（ROC 全称为 Receiver Operating Characteristic，受试者工作特征）：\n",
    "\n",
    "![](../images/roc-auc.webp)\n",
    "\n",
    "通过 ROC 也可以判断两个学习器的性能优劣，如果一个学习器的 ROC 包住另一个学习器的 ROC，则认为第一个学习器要好。如果两个学习器的 ROC 发生交叉，则可以通过比较 ROC 曲线下的面积，这个面积有一个专门的名称叫 **AUC**（Area Under ROC Curve）。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}