{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fit Model\n",
    "This notebook fits a 3-state classification model on a training set and calculates metrics on a test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "**Rule 9: Design Your Notebooks to Be Read, Run, and Explored.** We use ipywidgets to present the user with a pull-down menu to select a machine learning model.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import mlutils\n",
    "from sklearn import svm, metrics\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.externals import joblib\n",
    "from ipywidgets import widgets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# column names\n",
    "feature_col = \"features\" # feature vector\n",
    "value_col = \"foldClass\" # fold class to be predicted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read data set with fold type classifications and feature vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_json(\"./intermediate_data/features.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of data: 5370 \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Exptl.</th>\n",
       "      <th>FreeRvalue</th>\n",
       "      <th>R-factor</th>\n",
       "      <th>alpha</th>\n",
       "      <th>beta</th>\n",
       "      <th>coil</th>\n",
       "      <th>features</th>\n",
       "      <th>foldClass</th>\n",
       "      <th>length</th>\n",
       "      <th>ngram</th>\n",
       "      <th>pdbChainId</th>\n",
       "      <th>resolution</th>\n",
       "      <th>secondary_structure</th>\n",
       "      <th>sequence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>XRAY</td>\n",
       "      <td>0.26</td>\n",
       "      <td>0.19</td>\n",
       "      <td>0.469945</td>\n",
       "      <td>0.046448</td>\n",
       "      <td>0.483607</td>\n",
       "      <td>[-2.6183412084, -0.37215537190000003, 0.140630...</td>\n",
       "      <td>alpha</td>\n",
       "      <td>366</td>\n",
       "      <td>[SRM, RMP, MPS, PSP, SPP, PPM, PMP, MPV, PVP, ...</td>\n",
       "      <td>16VP.A</td>\n",
       "      <td>2.1</td>\n",
       "      <td>CCSCCCCCCCCHHHHHHHHHHHHTCTTHHHHHHHHHHCCCCCSTTS...</td>\n",
       "      <td>SRMPSPPMPVPPAALFNRLLDDLGFSAGPALCTMLDTWNEDLFSAL...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000</th>\n",
       "      <td>XRAY</td>\n",
       "      <td>0.23</td>\n",
       "      <td>0.18</td>\n",
       "      <td>0.504630</td>\n",
       "      <td>0.004630</td>\n",
       "      <td>0.490741</td>\n",
       "      <td>[-2.4130836608, -0.5122827316, 0.1969318015, -...</td>\n",
       "      <td>alpha</td>\n",
       "      <td>216</td>\n",
       "      <td>[MEA, EAD, ADV, DVE, VEQ, EQQ, QQA, QAL, ALT, ...</td>\n",
       "      <td>1PBW.B</td>\n",
       "      <td>2.0</td>\n",
       "      <td>CCCCCCCCCCCCCCHHHHCCTTSCSCHHHHHHHHHHHHHHTTCTTT...</td>\n",
       "      <td>MEADVEQQALTLPDLAEQFAPPDIAPPLLIKLVEAIEKKGLECSTL...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Exptl.  FreeRvalue  R-factor     alpha      beta      coil  \\\n",
       "1      XRAY        0.26      0.19  0.469945  0.046448  0.483607   \n",
       "1000   XRAY        0.23      0.18  0.504630  0.004630  0.490741   \n",
       "\n",
       "                                               features foldClass  length  \\\n",
       "1     [-2.6183412084, -0.37215537190000003, 0.140630...     alpha     366   \n",
       "1000  [-2.4130836608, -0.5122827316, 0.1969318015, -...     alpha     216   \n",
       "\n",
       "                                                  ngram pdbChainId  \\\n",
       "1     [SRM, RMP, MPS, PSP, SPP, PPM, PMP, MPV, PVP, ...     16VP.A   \n",
       "1000  [MEA, EAD, ADV, DVE, VEQ, EQQ, QQA, QAL, ALT, ...     1PBW.B   \n",
       "\n",
       "      resolution                                secondary_structure  \\\n",
       "1            2.1  CCSCCCCCCCCHHHHHHHHHHHHTCTTHHHHHHHHHHCCCCCSTTS...   \n",
       "1000         2.0  CCCCCCCCCCCCCCHHHHCCTTSCSCHHHHHHHHHHHHHHTTCTTT...   \n",
       "\n",
       "                                               sequence  \n",
       "1     SRMPSPPMPVPPAALFNRLLDDLGFSAGPALCTMLDTWNEDLFSAL...  \n",
       "1000  MEADVEQQALTLPDLAEQFAPPDIAPPLLIKLVEAIEKKGLECSTL...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Total number of data:\", df.shape[0], \"\\n\")\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split dataset into a training set and a test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set size: 4027 \n",
      "\n",
      "alpha         2107\n",
      "alpha+beta    1266\n",
      "beta           654\n",
      "Name: foldClass, dtype: int64\n",
      "\n",
      "Test set size: 1343 \n",
      "\n",
      "alpha         703\n",
      "alpha+beta    422\n",
      "beta          218\n",
      "Name: foldClass, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "train, test = train_test_split(df, test_size=0.25, random_state=13, stratify=df[value_col])\n",
    "print(\"Training set size:\", train.shape[0], \"\\n\")\n",
    "print(train[value_col].value_counts())\n",
    "print()\n",
    "print(\"Test set size:\", test.shape[0], \"\\n\")\n",
    "print(test[value_col].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select a Classification Method (default SVM)\n",
    "Using the pull-down menu, you can select one of three machine learning models and compare the performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "method = widgets.Dropdown(options=['SVM', 'LogisticRegression', \"NeuralNetwork\"],description='Method')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c304a2d147484533bca674876ec1235f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Method', options=('SVM', 'LogisticRegression', 'NeuralNetwork'), value='SVM')"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(method)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight='balanced', coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',\n",
       "  max_iter=-1, probability=True, random_state=13, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "if (method.value == 'SVM'):\n",
    "    classifier = svm.SVC(gamma='auto', class_weight='balanced', random_state=13, probability=True)\n",
    "elif (method.value == 'LogisticRegression'):\n",
    "    classifier = LogisticRegression(class_weight='balanced', random_state=13, solver='lbfgs', multi_class='auto', max_iter=500)\n",
    "elif (method.value == 'NeuralNetwork'):\n",
    "    # Neural network with one hidden layer of 20 nodes\n",
    "    classifier = MLPClassifier(hidden_layer_sizes = (20), random_state=13, early_stopping=True)\n",
    "\n",
    "\n",
    "classifier.fit(train[feature_col].tolist(), train[value_col])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make prediction for the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted = classifier.predict(test[feature_col].tolist())\n",
    "expected = test[value_col]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate metrics for the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification metrics:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "       alpha       0.88      0.78      0.83       703\n",
      "  alpha+beta       0.65      0.73      0.69       422\n",
      "        beta       0.71      0.80      0.75       218\n",
      "\n",
      "   micro avg       0.77      0.77      0.77      1343\n",
      "   macro avg       0.75      0.77      0.76      1343\n",
      "weighted avg       0.78      0.77      0.77      1343\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUUAAAEYCAYAAADLZOR0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XeYFFXWx/HvbxiCCiKiAjNkkKwIiAoGEAEREBMo5pyzq76uaVkVdXXNuq7ouuaIETCxKgYkJwUxgAxZJQhmwnDeP6pm6BmG7mZCV8/M+fj0Y3fV7apTPfTpe29V3SszwznnXCAj6gCccy6deFJ0zrkYnhSdcy6GJ0XnnIvhSdE552J4UnTOuRieFF0+ScMkPRM+byzpV0lVSnkfOZJ6l+Y2t2Hft0haKen7EmyjTD6XVJN0raTHoo4jHXlSTKEwIfwoaYeYZWdJGhdhWEUys0VmVtPMclO5X0n7SHpL0hpJqyVNlnR6KWy3MfAXoJ2Z1S/udsryc5Fk4b+PzJhlVcNlSV1QLKmnpCWJypnZrWZ2Vknirag8KaZeFeDSkm5EgQr195PUDfgA+AhoCdQFzgcOK4XNNwZWmdmPpbCtsvQTBY/3sHBZqYlNum5LFepLVU7cCVwpaaeiVkrqLmmKpLXh/7vHrBsnabik8cDvQPNw2S2SPgubdaMk1ZX0rKSfw200jdnGfZIWh+umSTpwK3E0DWsumZK6hdvOe/wpKScslyHpGknzJa2S9JKknWO2c7KkheG665L4bJ40s3+Y2UoLTDOzY2O2d7akeWEt8k1JWTHrTNJ5kr4Na5oPhT8evYGxQFYY/xNF1ahim/ZhjXVq+Dn9IOnuwp9L+DorjGN1GNfZMdsbFn4eT0n6RdIcSXsn+AyeBk6JeX0K8FShOE+XNDfc5neSzg2X7wC8HXOcv4bxDZM0UtIzkn4GTlPBrpLjJC2QtGP4+jBJ30vaNUGsFZOZ+SNFDyAH6A28CtwSLjsLGBc+35mgVnAykAkcH76uG64fBywC2ofrq4bL5gEtgNrAl8A34X4yCb5Q/42J4SSCGlgmQXPye6BGuG4Y8Ez4vClgQGahY6hKUJO7LXx9KTARaAhUBx4Bng/XtQN+BQ4K190NbAR6F/HZbA/kAgfH+fx6ASuBzuH2HgA+jllvwGhgJ4Ka4QqgX7iuJ7AkpmyB17F/n/D5BODk8HlNYL+iPhfgY+BfQA1gr3CfvWI+zz+B/gQthNuAiXGOz4AOwA/hMdQJn3cALKbcgPDvLaAHwQ9k5zjHNQzYABxJUBHaLvZvHZZ5Fngi/LexDBgY9fclqofXFKNxI3BxEb/EA4BvzexpM9toZs8DXwGHx5R5wszmhOs3hMv+a2bzzWwtQU1hvpn9z8w2Ai8DnfLebGbPmNmq8P13ESSX1tsQ+/3AL0Bere884DozW2Jm6wi+bIPDmtRgYLSZfRyuuwHYtJXt1iH4wi6Ps+8TgcfNbHq4vb8C3WJrwsDtZrbGzBYBHxIkquLYALSUtIuZ/WpmEwsXkNQI2B/4PzP708xmAo9RsKb3qZm9ZUEf5NNAxwT7/RMYBRwXPt4Ml+UzszHh39vM7CPgPaDIGn+MCWb2upltMrM/ilh/IcGPzjhglJmNTrC9CsuTYgTMbDZBjeaaQquygIWFli0EsmNeLy5ikz/EPP+jiNc1815IujJseq2VtIagdrlLMnGHzbSewAlmlpfcmgCvhc3VNcBcghpfvfB48uM1s9+AVVvZ/E8ECbNBnBAKfD5m9mu4vdjPJ/bM8u/EHPs2OhNoBXwVdkEM3Eo8q83sl5hlhf9eheOpkUSf3lMEiXWLpjPkN28nhk32NQQ10UR/w6L+3eQzszUEP6AdgLsSbKtC86QYnb8BZ1PwC7SMIMnEagwsjXld7GGNwv7Dq4FjgTpmthOwlqAZlsx7bwaOMLOfY1YtBg4zs51iHjXMbClBra9RzDa2J2iebcHMfidosh4TJ4wCn0/Yh1aXgp9Psn4jaLLnbasKkF9zN7Nvzex4YDfgH8BIxVw1EBPPzpJqxSwr/Pcqjk8IfhzqAZ/GrpBUHXgF+CdQL/wbvsXmv+HW/n3E/XcjaS/gDOB5gtZApeVJMSJmNg94EbgkZvFbQCtJJ4QnOI4j6JcrraZMLYI+vRVApqQbgR0TvSlsJr4EnGJm3xRa/W9guKQmYdldJR0RrhsJDJR0gKRqwE3E/zd3NcFJgKsk1Q2311HSC+H654HTJe0VJodbgUlmlpPwyLf0DUGtbYCkqsD1BF0Jecd8kqRdwxrxmnBxgaa/mS0GPgNuk1RD0p4ENcxnihFP7HaNoMtkUPg8VrUwzhXARkmHAX1j1v8A1JVUO9n9SaoRxnwtcDqQLemCEhxCueZJMVo3Afm1DzNbBQwkOAGyiiBJDDSzlaW0v3eBdwgSwkKCvqq4zarQIQS1lpExZzXnhOvuI+j3ek/SLwQnXfYNj2cOQV/VcwS1xp+ArV5DZ2afEfRr9QK+k7QaGEHwY4GZ/Y+gX/KVcHstgKHJHnyhfa0FLiDoA1xKUHOMja0fMEfSr+ExDt1KX9zxBCdflgGvAX8L4yyRsN94ThHLfyH4IX2J4PM8geDzz1v/FcGPx3dhl0ZW4W0U4TZgsZk9HPbVngTcImn3kh5HeaQtf4icc67y8pqic87F8KTonHMxPCk658otSf0kfR3eTVT4Ere8ATw+lDRD0ueS+ifcpvcpOufKo/Ayqm+APgQnyaYAx5vZlzFlRgAzzOxhSe2At8ysabzt+o3hZUhVtzNVT/rKiHKvbfNkTnRWLDWqVq7G1vTp01aaWancE11lxyZmG4s6ob+Z/bHiXTPrt5XV+wDzzOw7gPDSrSMIbnXN3wSbLzurTXCVQFyeFMuQqtemeodTEhesIJ5/+YaoQ0i5Vg1qJS5UgWxXVYXvuCo22/gH1VsfG7fMnzMfaiNpasyiEWY2InyeTcFLypYQXg4WYxjB5WIXE1z+lnAsT0+KzrloSJCRcKzelWaWaGSheI4nGC/gLgVD0z0tqUPMbapb8KTonItOyYYEXUrMbaQEIzUVvsXyTIIL8TGzCeHdO7sAWx1Xs3J1iDjn0khYU4z3iG8KsLukZuFtpEOJubsntIjgjiwktSUY4m1FvI16TdE5Fx0lHItkq8xso6SLCG5frUIwrNwcSTcBU83sTYJbZh+VdDnBSZfTirifvABPis65aIiSNp8xs7cI742PWXZjzPMvCca8TJonRedcRJI60ZJynhSdc9EpQfO5rHhSdM5FI7lLclLOk6JzLjppOEuvJ0XnXETkSdE55/IJqOLNZ+ec28xPtDjnXB4/0eKccwV5n6JzzoX8khznnCvE+xSdcy6PX5LjnHObCW8+O+fcZulZU0y/iJxzlUfJBplNZorTeyTNDB/fSFqTaJteU3TORacEJ1rCKU4fImaKU0lvxk5xamaXx5S/GOiUaLteUyxH+uzXilkvXsXsl6/mypN7brH+jksPZ+JTlzHxqcv4/KWrWD727/nrhl/Un2nPXcGMF/7CXVcMSmHUJTN+3FgG9ezMwAM78p+H7t5i/bRJ4zmu/4F0blaHsWNeL7DunuE3cNQh+3Bkr725/carKA9znL/37jvs2b417du05M47bt9i/bp16zjphONo36YlB3bfl4U5OQXWL1q0iF12qsk9d/8zRRGXgMLmc7xHfPlTnJrZeiBvitOtOR54PtFGvaZYTmRkiHuvPIoBlzzK0h/X8ul/L2b0J1/yVc7m+Xeuvm9U/vPzh3SnY6tsAPbbownd9mxK15PuAeCDRy7gwM7N+WT6d6k9iG2Um5vLrdf/hUeefYN6DbI54fCe9OzTnxat2uSXqZ/VkJvvepgnH7m/wHtnTp3EzKkTGfneBABOO6YvUyd+StduB6b0GLZFbm4ul11yIWPeHkt2w4YcsF9XBg4cRNt27fLLPPH4f6izUx3mfDWPl158geuu/T+eee7F/PX/d9UV9O13WBThF4sySlQvS2aK02A/UhOgGfBBoo16TbGc6NquEfOXrCRn2Wo2bMzl5bGzGHhQ+62WP7bPXrw0diYAZkb1aplUq1qF6lUzycyswo+rf01V6MU2e+ZUGjVtTsMmzaharRr9Dj+Gce+NKVAmu1ETWrXtQEahL5cU1Ko2bFjP+vXr2LhhI3V32S2V4W+zKZMn06JFS5o1b061atUYctxQRo96o0CZ0aPe4MSTTwXg6GMGM+6D9/NrwG++8TpNmzajXbut/7tIJwIkxX0Au0iaGvM4p5i7GwqMNLPcRAW9plhOZO1amyU/rs1/vfTHtezTvlGRZRvX34kmWTszbuo8ACbNXsTH0+azYPQNSPDvkZ/xdc5WZ3hMGz9+v5z6WQ3zX+/WIIsvZk6N847NOnbZl67dD6T33q0wM4aeeg7Nd29dVqGWimXLltKw4ea/aXZ2QyZPnrRlmUZBmczMTHasXZtVq1ZRo0YN7rrzH4x5Zyz3loemM4CEMhL2Kcab9zmZKU7zDAUuTCasSlVTlJQjaZeSlkl3Q/rsxesffsGmTUENonnDurRuuhstBw2nxeHD6dmlJft3bBptkGVsUc58Fsz7mvcmzWXs5K+Y/NlHTJ/0WdRhlZlbbhrGxZdeTs2aNaMOZZskUVOMJ5kpTpHUBqgDTEgmJq8plhPLVqyl4W61819n71abpSt+LrLs4N4dufyfm086HNGjA5NnL+K3P9YD8O6Er9l3jyaMn5VTpjGX1G71G/D9siX5r39cvox69bKSeu8H74xmj05d2X6HIEns37MPs6ZPpvO+3csk1tKQlZXNkiWbu8iWLl1Cdnb2lmUWL6Zhw4Zs3LiRn9eupW7dukyZPInXXh3JdX+9mrVr1pCRkUGN6jU4/8KLUn0Y26Rwt8e2SHKKUwiS5QuJpjbNj6nYEaU5Sa9LmiZpTuF+CElNJX0l6VlJcyWNlLR9TJGLJU2X9EX4K4OkfSRNkDRD0meSUtoWmzp3CS0b7UKTBnWomlmFIX06MuaTL7co16rJrtTZcTsmfrEwf9niH9ZwYOfmVKmSQWaVDA7s1KzACZp01b5jFxYt+I4li3LYsH4974x6hR59+if13vpZDZk2cTwbN25kw4YNTJs4nmYt07v5vHfXrsyb9y05Cxawfv16Xn7xBQYMLHilwICBg3j26ScBePWVkfQ4uBeSeH/cJ3w9L4ev5+Vw0SWXcdU116Z9Qgw6FRM8EjCzt8yslZm1MLPh4bIbYxIiZjbMzLa4hnFrKnJN8QwzWy1pO4Lrl14ptL41cKaZjZf0OHABkNcZs9LMOku6ALgSOAv4Cjgw/HXqDdwKHJOaQ4Hc3E1c/s83GHXfWVTJyODJ0VOYu+AHbji7L9O/WpKfIIf02YuXx84q8N5XP/icHl1aMPXZyzGDsRO/5q1P56Yq9GLLzMzkrzffyfknH8Wm3FyOPO5kWrZuy0N33UL7PTrTs29/Zs+axuVnn8jPa9fw0f/e5l9338pr70+mz4AjmfzZxwzuux9CdO/Zm5590vusbGZmJvfc9yCHDziU3NxcTj3tDNq1b89Nw26kc5e9GXj4IE4740zOOO1k2rdpSZ06O/P0sy9EHXaxiaSayCmn8nDtVnFIGgYcFb5sChxKcB3T3kBN4GMzaxyW7QVcYmZHSsoB9jezpZL2BYabWW9JjYD7gd0BA6qaWRsKCWulQc202o5danQ6t8yOMd1MevmGqENIuVYNakUdQkptV1XT4pz42CaZdZvbjv1viVvmp2dOLLX9JatCNp8l9QR6A93MrCMwA6hRqFjhX4PY1+vC/+eyuTZ9M/ChmXUADi9ie8FGzEaY2d5mtreqblf8g3CuEijhiZYyUSGTIlAb+MnMfg/7BPcrokxjSd3C5ycAnyaxzbzT/aeVSpTOVWYCZSjuIwoVNSm+A2RKmgvcDkwsoszXwIVhmTrAwwm2eQdwm6QZVOy+WOdSIq9PMd1qihXyy21m64CietWbAkiqCWw0s5OKeG/TmOdTgZ7h8wlAq5ii15dawM5VUul4oqVCJkXnXDkQNp/TTaVMimaWA3SIOg7nKjuvKTrnXEioRHe0lBVPis656KRfRdGTonMuIirZvc9lxZOicy4y3qfonHOx0i8nelJ0zkVD8hMtzjlXQDo2n9MvTTvnKo2S3vucaN7nsMyxkr4Mx1Z9LtE2vabonItMSWqKycz7LGl34K8EwwH+JCnh7GVeU3TORUMlHjosmXmfzwYeMrOfAMws4ZDznhSdc5EI7miJ/yD+FKdFzftccFKbYBCXVpLGS5ooqV+iuLz57JyLTBKt53hTnCYjk2C0/J4EU6B+LGkPM1sT7w3OOZd6Iq82WFzJzPu8BJhkZhuABZK+IUiSU7a2UW8+O+ciIUim+RxPMvM+v044Jmo4n3sr4Lt4G/Wk6JyLjBT/EY+ZbQTy5n2eC7yUN++zpLy5Yd8FVkn6EvgQuMrMVsXbrjefnXPRKHnzGTN7C3ir0LIbY54bcEX4SIonRedcJER63tHiSdE5F5Gk+g1TzpOicy4yXlN0zrmQSqFPsSx4UnTORSYNK4qeFJ1z0fHmcyXTrkUWI1/7W9RhpEyn00dEHULKzXv+gqhDKL+8+eycc5sFl+REHcWWPCk65yLil+Q451wB3qfonHN5kri/OQqeFJ1zkQhGyUm/MWk8KTrnIuM1Reecy1PeLsmRtGO8N5rZz6UfjnOushBJTU6VcvFqinMAI2j658l7bUDjMozLOVcJVClhTTGciOo+oArwmJndXmj9acCdbJ6m4EEzeyzeNreaFM2s0dbWOedcaShJRTGZeZ9DL5rZRcluN6lTP5KGSro2fN5QUpdkd+Ccc0VRauZ93mYJk6KkB4GDgZPDRb8D/y7pjp1zrkqG4j4o+bzPAMdI+lzSSEkJW8DJnH3ubmadJc0AMLPV4cxZzjlXIimY93kU8LyZrZN0LvAk0CveG5JpPm+QlEFwcgVJdYFNJQjSOecQUEWK+0gg4bzPZrbKzNaFLx8DEnb9JZMUHwJeAXaV9HfgU+AfSbzPOee2LkF/YhJ9ignnfZbUIOblIIKpUONK2Hw2s6ckTQN6h4uGmNnsRO9zzrlESnL22cw2Ssqb97kK8HjevM/AVDN7E7gknAN6I7AaOC3RdpO9o6UKsIGgCZ1+Nys658odUfLrFJOY9/mvwF+3ZZvJnH2+DngeyCJosz8naZt24pxzRSlh87lMJFNTPAXoZGa/A0gaDswAbivLwJxzFZtU8ppiWUgmKS4vVC4zXOaccyWSfikx/oAQ9xD0Ia4G5kh6N3zdl+Csj3POFVtp9CmWhXg1xbwzzHOAMTHLJ5ZdOM65SiPCfsN44g0I8Z9UBuKcq3zSMCcmdfa5haQXwnsHv8l7pCI4V9AnH47lsAM6cWj3PXn0gbu2WD9l4qcc3Xd/OjSqzbujXyuwbtmSxZw5dBADDurMwB5dWLp4YarCLpE+XZow67FTmf346Vx5bNct1t9xTg8mPnQiEx86kc8fO43lI88HoPFutfjswROY+NCJTHvkFM7qv2eqQy+WD//3Hgftswf7d2nHg/feucX6iZ99Qr+e+9Fk1x0Y/carBdadOPhw2jWtx6lDj0pVuCWS13xOcO9zyiVzouUJ4Bbgn8BhwOmEt/y51MnNzeXma6/gPy+8Sb0G2Rzb/yAOPrQ/LVu1zS+Tld2I2+59hMf/fd8W77/m0rM595Kr2b9HL3777VcylP6Xm2ZkiHsv7MWAa19l6cpf+PT+Exg9cT5fLVqdX+bqER/lPz9/0F50bLErAMtX/0bPy19k/YZcdqhRlWmPnMyYifNZvvq3lB9HsnJzc7n+6kt57tUxNMhqyIBD9qdvv4G0arP5b5zdsBF3P/Qojzx4zxbvP//iy/njjz945om4wwWmlXRsPifzzdjezN4FMLP5ZnY9QXJ0KfT5jKk0btqcRk2aUa1aNfofMZgP3h1ToEx2oya0btdhi8mA5n0zl9yNuezfI7gPfocdarLd9tunLPbi6tq6PvOXryHn+7Vs2LiJlz/6moHdWmy1/LE9W/PSuK8B2LBxE+s35AJQvWoVMtLwy1fYzGlTaNqsBU2aNqdatWoccfQQ3nt7VIEyjRo3pV37PYqc8OmAHr3YoWbNVIVbYlKJ730uE8kkxXXhgBDzJZ0n6XCgVhnH5Qr58ftl1M9qmP+6XoNsfli+LKn35syfR63atbn4zOM5uk937rzpOnJzc8sq1FKTVbcmS1b8kv966cpfya5b9Je+8W61aFK/NuNmbR5JquEuNZn88El8+/RZ3PXy1LSuJQIsX76MBtmb/8b1s7JZnuTfuLyS4j+ikExSvBzYAbgE2B84GzijuDuUlCNpl5KWKcZ+f93G8kdKaleaMUQlN3cj0yZ9xtU33spLb3/M4kULeO3FZ6IOq1QN6dGa1z/5hk2bNvfsLFn5K/uc/wwdzvgvJ/Vux247pX/tuLJJxztaEiZFM5tkZr+Y2SIzO9nMBpnZ+FQEVxyShoXzMpTUkUDaJMXd6mfx/bIl+a9/WL6Ueg2yknpvvQbZtGm/B42aNCMzM5ND+h3Ol1/MLKtQS82yVb/ScNfNjZLsXWqydFXRv22De2xuOhe2fPVvzMlZyf4dihp/NH00aJDF8qWb/8bfL1tKgyT/xuWRiH+SJaoTLVtNipJek/Tq1h7JbFzS65KmSZpTaMRcJDWV9JWkZyXNDUfFjf0pv1jSdElfSGoTvmcfSRMkzZD0maTW23Kwku4JY3lf0q7hshaS3gnj/ERSG0ndCYYZulPSzLDM2ZKmSJol6ZVCsZa5PfbqwsIF81myKIf169fz1hsjObhv/6Tf+8vPa1m9agUAkz79iBat2pRluKVi6tff0zKrDk3q7UjVzAyG9GjNmInfbVGuVcM61KlVnYlzN99olb1LTWpUqwLATjWr0719Nt8sWb3Fe9NJx857s+C7eSxauID169fzxqsv06ffwKjDKjsJms5RNZ/jnX1+sBS2f0Y4Uvd2BJPKvFJofWvgTDMbL+lx4AKCs9wQjLjbWdIFwJXAWcBXwIHhkEG9gVuBY5KMZQeC4YQul3Qj8DfgImAEcJ6ZfStpX+BfZtZL0pvAaDMbCSBpjZk9Gj6/BTgTeKDwTsLkfw4EZ4NLS2ZmJtcPv4uzTjiSTbm5HD30ZHZv3Y7777iZDh070+vQAXwxcxoXn3k8P69Zw4dj3+aBfw5n9LipVKlShatuuJXTjx2ImdF+z04MOfH0UoutrORuMi7/1weMGn40VTLEk+/NYe7CVdxwcjemf/tDfoIc0rM1L48reJVY60Y7c/s5B2EWfLnufWUac3JWRXEYScvMzOTmO+7lxMGHsyk3l+NOPJXWbdtx561/p2OnLvQ9bCAzp0/lrJOPY+3anxj7zlvcffvNfDBhBgBH9+/FvG+/4bfffmXv9i345/3/puchfSI+qviiOpkSj8zK7uoaScOAvIummgKHEkwuszdQE/jYzBqHZXsBl5jZkZJygP3NbGmYqIabWe9wfoX7gd0JLguqamZtJO0BPB3upz6wnuD2RIBDzGyVpFygephQmwOvAgcAK4DYdld1M2sr6QkKJsUeBJcm7RTG/q6ZnRfv+Dt07Gwj3/lk2z60cqzT6SOiDiHl5j1/QdQhpFTDnWtMK+H0APnqtexgx/1zZNwyDxzVNu7+lGCK05hyxwAjga5mNjXePpMdT3GbSepJMDBtNzP7XdI4oEahYoUzcuzrvCHEc9kc583Ah2Z2lKSmwDgAM/sC2Cvc7zAgx8yeSBBi3tiQa8xsryQO6QngSDObFfZZ9kziPc65OErSbagkpziVVAu4FJiUVEzFDymh2sBPYUJsA+xXRJnGkrqFz08gmOog0Tbz5mA4bRvjyQAGx+7LzH4GFkgaAqBAx7DMLxS89KgWsFxSVeDEbdy3c66QvKHDSnCiJdkpTm8mmELlz2TiSjopSqqebNnQO0CmpLnA7RQ9kMTXwIVhmTrAwwm2eQdwm4KZBbe1lvsbsI+k2QSzed0ULj8ROFPSLILBL/I+1BeAq8KTOi2AGwh+acYT9G0650ooQ/EflHCKU0mdgUZmVvBOhzgSJhZJ+wD/IailNQ5rUmeZ2cXx3hfOoFXUnS9Nw+3WBDaa2UlFvLdpzPOphE1VM5sAtIopen0R7x22lXiKvOrXzBYA/YpYPp6Cl+Q8TOKk7ZxLUpJDhxV7itPwppO72cZWZTI1xfuBgcAqADObBRy8jfE559wWMhI8Ekg0xWktoAMwLjx5ux/wpqS4STaZJmiGmS0sdHV5ie8RM7McgoCdc5WQVOILtPOnOCVIhkMJzhcAYGZrgfw748KTvVcmOvucTE1xcdiENklVJF0G+NBhzrkSK8nF22a2keBa43cJ5nN+KW+KUwXTmhZLMjXF8wma0I2BH4D/hcucc65ESnonX6IpTgst75nMNhMmRTP7kaBa6pxzpaY8ztECgKRHKWJQWTM7p4jizjmXHJW8plgWkmk+/y/meQ2C2/YWb6Wsc84lRaTnvc/JNJ9fjH0t6WkS33ninHMJldeaYmHNgHqlHYhzrvKJaiDZeJLpU/yJzX2KGQSjz1xTlkE55yq+4N7nqKPYUtykqCCNd2TzVeKbrCzHGnPOVSrpOKFY3KRoZibpLTPzO0+cc6UquCQn6ii2lExIMyV1KvNInHOVjMhI8IjCVmuKkjLD22g6EQzeOJ9g+C0RVCI7pyhG51wFVB77FCcDnQkmcHLOuVJX3voUBWBm81MUi3OuEhHRzdgXT7ykuKukK7a20szuLoN4nHOVSHm797kKwax16Re1c67cE2U7SVRxxUuKy83spjjrnXOu+JSefYrxEnX6ReucqzBEkBTjPRJuQ+on6WtJ8yRtcaedpPMkfSFppqRPJbUrajux4iXFQxJG5JxzJaAEj7jv3Tzv82EEk8wdX0TSe87M9gjndr+DYCKruLaaFM1sdaI3O+dc8YmMjPiPBBLO+xzO7Z5nB4oYG7aw4oyS45xzJZbkiZZdJMVONDXCzEaEz4ua93nfLfYjXQhcAVQjmPM9Lk+KzrnIJNFvWOx5n/OY2UPAQ5JOIJgr/tR45T0plqFqmRk0qrt91GFPc1YjAAAWQklEQVSkzKwnzo06hJRr2esvUYdQfqnE4ykmmve5sBeAhxNtNB0vE3LOVQJ50xHEeySQP++zpGoEE+y9WWAf0u4xLwcA3ybaqNcUnXORKUk90cw2Ssqb97kK8HjevM/AVDN7E7hIUm9gA/ATCZrO4EnRORehkl67nWjeZzO7dFu36UnROReJcjubn3POlQ2hNLxxzpOicy4SXlN0zrlYKn/jKTrnXJnypOiccyFvPjvnXCF+osU552Kk4yCznhSdc5EIBpmNOooteVJ0zkUjydG1U82TonMuMumXEj0pOucikjdHS7rxpOici0wa5kRPis656KTjJTk+yKxzLjIZiv9IJIkpTq+Q9KWkzyW9L6lJwpiKdyjOOVcKSjDHaZJTnM4A9jazPYGRBNOcxuVJ0TkXiSDvxf8vgWSmOP3QzH4PX04kmMclLk+KzrloJGg6J9F8LmqK0+w45c8E3k60UT/R4pyLTuLEF2/e5+R3I50E7A30SFTWa4rlyNh336FThzbs2XZ37rrz9i3Wr1u3jlNOHMqebXen5wH7sTAnB4CFOTnsUnt7unXtRLeunbjkwvNSHHnxffzBexx6wF706bYHIx745xbrp0z4lKP6dKddwx15Z/RrBda1za7FEb3344je+3HeqUNSFXKJ9Onellmv3cDsN/7Glaf32WJ9o/p1eGfEJUx4/v+Y/OJfOfSAzV1oV57Rl9lv/I1Zr91A725tUxl2MQV3tMR7EM77HPOITYhJTXEaTlx1HTDIzNYlispriuVEbm4uV1x6EW++9R7ZDRtyUPd96D9wEG3bbv5SPPnf/7DTTjvx+dxvefmlF7jhumt46tkXAGjWvAUTpsyIKvxiyc3N5aZrr+C/L46iXoNsBh92IL36DqBl681f+AYNG3HbfY/w+MP3bfH+GjW2443/TUxlyCWSkSHuveZYBpz/IEt/WMOnz17F6I++4Kvvvs8v839n9eOVsdN59OVPadO8Pq8/cD5tBvyNNs3rM+TQznQePJwGu9bmrX9fxB5H3sSmTRbhEcWXxLmURPKnOCVIhkOBEwrsQ+oEPAL0M7Mfk9mo1xTLialTJtO8RUuaNW9OtWrVGHzscYwZ9UaBMmNGvcmJJwczOB519GDGffg+Zun7pUjk8xlTadK0OY2aNKNatWoMOGIw7787ukCZho2a0KbdHmRklP9/yl07NGX+4pXkLF3Fho25vPzudAb23LNAGTNjxx1qAFC75nYsX7EWgIE99+Tld6ezfsNGFi5bxfzFK+naoWmqD2HbleDss5ltBPKmOJ0LvJQ3xamkQWGxO4GawMuSZkp6cyuby+c1xXJi2bKlNGy0+cRZdnZDpkyetGWZhkFrIjMzk9o71mbVqlUALMxZQPd9OlNrxx25cdjN7H/AgakLvph++H4Z9bM3H3O9Btl8PmNqnHcUtG7dnxx96AFkZmZyzkV/ofdhh5dFmKUma7faLPnhp/zXS3/4iX0KJbbhj7zFqH9dxPlDe7D9dtUZcN4DAGTvWptJX+Rsfu+PP5G1W+1UhF0iJb3NL4kpTntv6zYrTVKU1BQYbWYdkix/GvCemS0rw7BSon6DBsydt5C6desyY/o0hg45iikzZrPjjjtGHVqZ+nDKV9RrkMXihQs4dXB/WrVtT+OmzaMOq0SO7bc3z4yayH1Pf8C+ezbjP7ecQpfBt0YdVrGl3/0s3nyO5zQgK+og8mRlZbNk8ZL810uXLiErO3vLMkuCKxQ2btzI2p/XUrduXapXr07dunUB6NS5C82at2Det9+kLvhiqlc/i++Xbj7mH5YvpV79Bsm/v0Hw52vUpBn7dD+QL2fPKvUYS9OyH9fSsF6d/NfZ9eqwNGwe5zn1yG688t50ACZ9voAa1aqyy047sHTFWhrWj3nvbnVY9mPB96YdgaS4jyhUtqSYKelZSXMljZS0vaQukj6SNE3Su5IaSBpMcPr+2bAfYjtJN0qaImm2pBFK8V+sy95dmT/vW3IWLGD9+vWMfOlF+g8cVKBM/4GH8+zTTwLw2qsj6dGzF5JYsWIFubm5ACz47jvmz/uWps3Sv8a0x15dyFkwn8WLcli/fj1j3hhJr0MHJPXetWt+Yv264ETj6lUrmT5lIi13b1OW4ZbY1DkLadl4V5pk1aVqZhWGHNqZMeM+L1Bm8fer6blPawBaN6tHjepVWfHTr4wZ9zlDDu1MtaqZNMmqS8vGuzJldk4ER5E8EQwIEe8RhUrTfA61Bs40s/GSHgcuBI4CjjCzFZKOA4ab2RmSLgKuNLOpAJIeNLObwudPAwOBUYV3IOkc4ByARo0bl1rgmZmZ3HXvAxw5sB+5ubmcfNrptGvXnpv/fiOdO+/NgMMHcerpZ3LW6aewZ9vdqbPzzjzx9PMAjP/0Y275+9+oWrUqGRkZ3PfAw+y8886lFltZyczM5MZb7+Ks448gNzeXY4aewu6t23HfHTfToWNnDjl0AJ/PnMZFZwzl5zVr+HDs2zxw53DGfDSV+d9+zd+uvhhlZGCbNnH2RX8pcNY6HeXmbuLyf7zEqH9dSJUM8eQbE5n73ffccP4Apn+5iDEffcE1d7/Gv244notPOhgzOPvGpwGY+933vPLeDGa8ch0bczdx2e0vpfWZ5zzpOEqOyvPZyW0R9il+bGaNw9e9gGsJbhX6LixWBVhuZn0ljaNgUjwGuBrYHtgZeMDMtrxYMEbnLnvbJxOmlP7BpKmlq/+IOoSU63jY1VGHkFJ/znxompntXRrb6tCxs41859O4Zdpm7VBq+0tWZaspFv4F+AWYY2bd4r1JUg3gXwQ3li+WNAyoUTYhOld5pGNNsbL1KTaWlJcATyC4QXzXvGWSqkpqH67/BagVPs9LgCsl1QQGpypg5yqydOxTrGxJ8WvgQklzgTrAAwQJ7h+SZgEzge5h2SeAf0uaCawDHgVmE1woWnnaxM6VkVIYJadMVJrms5nlAEWdfpwJHFRE+VeAV2IWXR8+nHOlIcmBZFOt0iRF51wa8qTonHN5omsix+NJ0TkXiWCK06ij2JInRedcdDwpOufcZiUdJacseFJ0zkUm/VJi5btO0TmXLkphlJwk5n0+SNJ0SRvDgV4S8qTonItESUfJSXLe50UEwwA+l2xc3nx2zkWmhM3n/HmfASTlzfv8ZV6B8KYNJG1KdqOeFJ1zkUniREu8KU6Lmvd535LG5EnRORedxFXFlT50mHOuUlDJ731Oat7nbeUnWpxzkSnhKDn58z5LqkYw73PCKUwT8aTonItOGc/7LKmrpCXAEOARSXMSheTNZ+dcZEp673MS8z5PIWhWJ82TonMuIj5KjnPO5cu7eDvdeFJ0zkXGk6JzzuWRj5LjnHP5kjjBHAlPis656KRhVvSk6JyLjDefnXMuRvqlRE+KzrkIJTOQbKrJzKKOocKStAJYGMGudwFWRrDfqFS244XojrmJme1aGhuS9A7BccSz0sz6lcb+kuVJsQKSNDXVwy1FqbIdL1TOY04VHxDCOedieFJ0zrkYnhQrphGJi1Qole14oXIec0p4n6JzzsXwmqJzzsXwpOicczE8KTrnXAxPis45F8OTonMVhNLxnrlyyJNiBZf3RZFUJepY0lFRiaS8JJfCcZpfSlIq/JKcCkySzMwkDSCYE/cHYIyZfRhxaGkh5vPpCXQmmCZzqpmtyFsXbYRbFxufpOMJZqybDHxhZqsjDa6c85piBRZ+4fsDNwMPAzsDj0k6AspPjaishJ/PQOAeIJdgDuFrJTUO16Xt5xOTEC8BLgDWA7cC50lqFGVs5Z0nxQpMUnWgA3AiUBfYHbgDuEvSgHSuCZWV2EQnqQFwLHAE8B3QmGCIv6skNUzHz0dSRszzdsC+wMHhompAFnCKpOwIwqsQvPlcwRRu9knaHqgDPAdcbGafS3ofaAPsRTA0U6X4RyBpO6ChmX0rqQ3wC7AdUAN4EjiG4HO5CxgPXGJmf0YVb2GS6gI7mNkiSb2ATwiG3moHXGtmh0g6D7gUeBy4x8w2Rhdx+eSDzFYQknYws9/CZl9fgi/KMjN7SdIGgnEdN0o6CPiS4Au/IsqYI9ACOCr8oTgROMjM5oWfySwzywlrWJMIEkraJMRQK+AcSWuAgcDeZrZcUm/g+7DMWmAi8IQnxOLx5nMFIKkm8LGkvpJ2Bx4iaEYNkvSgmW0g+NJcR1Bj/J+ZzYku4miY2WyCJuYVwAgzywlXzQWOlvQU8BIw0szmRhPl1pnZBIK+w3OBv5rZ2nDVGOBgSaOBm4A7KuEPXqnx5nMFIekU4HrgA+ANM3s7TJA3AkvM7K+SdgR2NbP56X52tTQVOlPbBjgE6AiMBT4ysx8lNSTon1tsZpOji7agIrpD9gH2AfoRNPM/M7N1knYB9gNmxyR7VwzefC7nwo53M7OnJP0K3AcsBt4G5gN/B/4h6REzOxf4GSrPNW0xl90cBnQh+Dd/K8HJlYHAr2FNuz1wm5mtiy7aLcUk83OA3cLFtwPrgKuB68NLimqZ2bAoYqxoPCmWY+EXfpOklpJWm9mrkn4CHpc0PawtzgeuAbaPONxIhAmxH0Gz8gqC2lVDMztLUh1gANALuCHdEmKe8LKbQQSXVt0LbDKzW8O+0cuA1gRNalcKPCmWUzE1oEMJrrObF06UdQdwBvCIpKpm9ibwbZSxpoEDCU6stCXok7sFwMxG5J2dN7Ol6dKlICkj/LHLi6cRcBhwCbCc4JKqqmZ2n6SdAMxsTYQhVyjep1iOSeoOnA/cDfxB0Kc0BDgdOChc3qUydboXldgk3QU0BWoD54Z9qscQXKr0uJltSn2kiUlqRnAX0rNAdYIm87FmtkHSucAPZvZ6lDFWRH72uRyR1FTSX8Ln1Qn6lHqY2Qwz+woYDXwN9DOzkcB+lSkhQoE+uE7hoxHwAHAA8FaYEA8g6Fecl04JUVJ3SUPD5xcDrwLDCc6YdwaeCRPiaQTN5i+iirUi8+Zz+WLAXyRVD/uULgFGSRphZueY2UpJvxCcWYWgllEphHenPG9mPSXtBbxBcHFzA4LLkHoBr0vaE9gDuMLMxkUV71bUAW6T1J7gmsrB4f8XEfQJPxjex74HcIyZzY8s0grMk2I5ISnTzBZK6gaMllTNzIaFX5LnJL0HPAj0J7gMBzPLjTDklAovYv5V0hfA+8AQM5skaQ/gMWApwSU32wM1zGxehOEWyczGSFpP0Ec8K6zVLiG4mqAB4SU4QJXK1gJIJe9TLAdiTqq0BlYT9C+9DbxgZsPDa+xGAVWAk8Jb+TIryx0Nkqrk/QBI+jdBn2p3M5sWLjsZaG9m10QYZtIUDNjxBHCemb0YLnsDeDrsFnFlyGuK5UCYEA8nOGuaA8wDrgLukZRrZrdLGgTcD1xIcDKhUiRECGrEYZfCOjM7T1Jt4BmCs80Q9J23Dc/Ybogu0uSY2RthIr9fUltgJtAcmBFtZJWDJ8VyQNJ+BE3iPuFjBPAncDkwIqwV3iLpMuBuSfXN7Putb7HikLS7mX0be42hmR0vaaSkpQQ/FAcAj5aHhJjHzEZLygReITiBdoSZfRdxWJWCN5/LgbB53ICgI/4W4ATgEYL7mUcCq83so7BsZWo29ye4Ze9VMxsv6VKgtpndFK5/nuDi7G5mNiddrkPcFpJ6AAv91r3U8aRYjkgaDvwYXrR7CsEQUUdZMJRUufvCl1R4xvlkYAPBABj7E1zHtySmTPvKOPiFKz5vPpcvXwDnSqoKHE0w/NciqDz3MucJT64sB+6Q9BrBfc2X5yXEvP7DvIRYGX80XPH4xdvly1sEZyW7AcPNbHy04UQn5mzz+QRdC88CjSUdEq7fUKi8J0SXFG8+l0N5/YaVqfYTc1lS7OU3jQlGBTo7vHB9GMF0Avea2U8RhuvKMU+K5VBlSoYQf1ZCSduZ2R9huV0JhtDys7Su2Lz5XA5VpoQIcWclHGRmf0jKCGuQKzwhupLypOjSnrY+K+HdCmYl3FSZbml0Zcubzy4tFe4ikM9K6FLEL8lxaUU+K6GLmDefXdqQz0ro0oA3n11akc9K6CLmNUWXFsIzyDKzpwgm2hpAMNo0bJ6VsJWCWQl/zhtg1ROiK21eU3SRi7kOsSXB4BarJR0MPA5cENYWBbQEtjezWZEG7Co0T4ouUjEJMX9WQiBvVsIsgtGArrRgVkLnypwnRRc5+ayELo14UnQpJ6kpwcRLd4UXZr8IdDazxuH6XYBrgZlm9pSkLDNbFlnArlLxEy0uCnmzEl4bjph9CfCTpBEAZrYSqJSzErroeVJ0KZU3KyHB8GfHSxoWjgk5AGgj6b1wvpn+wP+gcs1K6KLnzWeXMj4roSsPPCm6lCpiVsKxBGednwxnJWxEMNnUj2Z2bmSBukrLm88uZQrNSvgScB5wIMGshBdIut7MFgOXAbtIqh9ZsK7S8pqiSxmfldCVB15TdCljZkvMbArQA3jWzOYBTxIM/zXNzD4K71zBE6KLig8d5qLgsxK6tOVJ0UXhLYIzz4Oo5LMSuvTjfYouMpVxVkKX/rxP0UUpF7zJ7NKL1xSdcy6G1xSdcy6GJ0XnnIvhSdE552J4UnTbTFKupJmSZkt6OZyTubjb6ilpdPh8kKRr4pTdSdIFxdjHMElXJru8UJknJA3ehn01lTR7W2N06cOToiuOP8xsLzPrAKwnuIc5nwLb/G/LzN40s9vjFNkJ2Oak6Ny28KToSuoToGVYQ/pa0lPAbKBROH/zBEnTwxplTQBJ/SR9JWk6wR0thMtPk/Rg+LyepNckzQof3YHbgRZhLfXOsNxVkqZI+lzS32O2dZ2kbyR9CrROdBCSzg63M0vSK4Vqv70lTQ23NzAsX0XSnTH79hF9KghPiq7YJGUChxHctgewO/AvM2sP/EYwf3NvM+sMTAWukFQDeBQ4HOgCbG0knPuBj8ysI8FUp3MIpj6dH9ZSr5LUN9znPsBeQBdJB0nqAgwNl/UHuiZxOK+aWddwf3OBM2PWNQ33MQD4d3gMZwJrzaxruP2zJTVLYj8uzfltfq44tpM0M3z+CfAfgpn3FprZxHD5fkA7YHw4xkM1YALB4A8LzOxbAEnPAOcUsY9ewCmQP/L2Wkl1CpXpGz5mhK9rEiTJWsBrZvZ7uI9kZgLsIOkWgiZ6TeDdmHUvmdkm4FtJ34XH0BfYM6a/sXa472+S2JdLY54UXXH8YWZ7xS4IE99vsYuAsWZ2fKFyBd5XQgJuM7NHCu3jsmJs6wngSDObJek0oGfMusJ3OFi474vNLDZ55k3K5coxbz67sjIR2D+c4B5JO0hqBXwFNJXUIix3/Fbe/z7BtKd5/Xe1CSazqhVT5l3gjJi+ymxJuwEfA0dK2k5SLYKmeiK1gOXhyD0nFlo3RFJGGHNz4Otw3+eH5ZHUStIOSezHpTmvKboyYWYrwhrX8wqmMQW43sy+kXQOMEbS7wTN71pFbOJSYISkMwnukT7fzCZIGh9e8vJ22K/YFpgQ1lR/JZjbZbqkF4FZwI/AlCRCvgGYBKwI/x8b0yJgMrAjcJ6Z/SnpMYK+xunhGJArgCOT+3RcOvN7n51zLoY3n51zLoYnReeci+FJ0TnnYnhSdM65GJ4UnXMuhidF55yL4UnROedi/D92InqZdTXwtAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"Classification metrics:\\n\")\n",
    "print(metrics.classification_report(expected, predicted))\n",
    "\n",
    "cm = metrics.confusion_matrix(expected, predicted)\n",
    "mlutils.plot_confusion_matrix(cm, classifier.classes_, normalize=True, title='Normalized Confusion Matrix')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAEYCAYAAAA+mm/EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XeYFFXWx/Hvb4YMQ5As0ZVkWhUQFROK66ogYI6IimIOq65ZV11dw7rmiKuvcQ2LAQUMiBEUJAgKi0oQBCRKECRIOO8fdQeacbpnmFQ90+fzPPVQfau6+vTonLn31q17ZWY455z7vay4A3DOuXTlCdI555LwBOmcc0l4gnTOuSQ8QTrnXBKeIJ1zLglPkK5USKou6W1JKyT9txjXOVXS+yUZWxwkvSOpX9xxuG3jCTLDSTpF0jhJqyTND7/I+5fApY8DGgP1zez4ol7EzF40s8NKIJ6tSOomySS9kad891D+cSGvc7OkFwo6z8yOMLNnixiui4knyAwm6XLgfuAfRMmsJfAo0LsELt8K+N7MNpTAtUrLYmBfSfUTyvoB35fUByjiv2fllZn5loEbUAdYBRyf4pyqRAn0p7DdD1QNx7oBc4ErgEXAfODMcOwW4DdgffiM/sDNwAsJ124NGFApvD4DmAmsBH4ATk0oH5nwvq7AWGBF+LdrwrGPgb8Do8J13gcaJPluufE/DlwYyrKBecBNwMcJ5z4AzAF+AcYDB4Tyw/N8z0kJcdwe4lgDtAllZ4fjjwGvJVz/LmAEoLj/v/Bt683/smWufYFqwBspzrke2AfYA9gd6ALckHC8CVGibUaUBB+RVM/M/kZUK33FzGqZ2VOpApFUE3gQOMLMcoiS4MR8ztsOGBrOrQ/cCwzNUwM8BTgTaARUAa5M9dnAc8DpYf/PwGSiPwaJxhL9DLYD/gP8V1I1M3s3z/fcPeE9fYEBQA4wO8/1rgB2k3SGpAOIfnb9LGRLlz48QWau+sASS90EPhW41cwWmdliopph34Tj68Px9WY2jKgW1b6I8WwCdpVU3czmm9mUfM7pAUwzs+fNbIOZvQR8CxyVcM7/mdn3ZrYGeJUosSVlZp8D20lqT5Qon8vnnBfM7Ofwmf8iqlkX9D2fMbMp4T3r81xvNdHP8V7gBeBiM5tbwPVcDDxBZq6fgQaSKqU4Z3u2rv3MDmWbr5Enwa4Gam1rIGb2K3AicB4wX9JQSR0KEU9uTM0SXi8oQjzPAxcBB5NPjVrSlZKmhjvyy4lqzQ0KuOacVAfNbAxRl4KIErlLQ54gM9cXwDqgT4pzfiK62ZKrJb9vfhbWr0CNhNdNEg+a2Xtm9iegKVGt8MlCxJMb07wixpTreeACYFio3W0WmsBXAScA9cysLlH/p3JDT3LNlM1lSRcS1UR/Ctd3acgTZIYysxVENyMekdRHUg1JlSUdIenucNpLwA2SGkpqEM4vcEhLEhOBAyW1lFQHuDb3gKTGknqHvsh1RE31TflcYxjQLgxNqiTpRGBnYEgRYwLAzH4ADiLqc80rB9hAdMe7kqSbgNoJxxcCrbflTrWkdsBtwGlETe2rJKXsCnDx8ASZwUJ/2uVEN14WEzULLwLeDKfcBowDvga+ASaEsqJ81nDglXCt8Wyd1LJCHD8BS4mS1fn5XONnoCfRTY6fiWpePc1sSVFiynPtkWaWX+34PeBdoqE/s4G1bN18zh0E/7OkCQV9TujSeAG4y8wmmdk04DrgeUlVi/MdXMmT3zhzzrn8eQ3SOeeS8ATpnHNJeIJ0zrkkPEE651wSqQYJu2JSpeqmKjlxh1FmOuzYrOCTKpiqVbLjDqFMTZwwfomZNSyJa2XXbmW2YU3Kc2zN4vfM7PCS+Lyi8ARZilQlh6rtT4g7jDLz3Ou3xx1CmWvXNHP+AALUqZ6d90mmIrMNawr8/Vg78ZGCnlgqVZ4gnXPxkCArvWvgniCdc/FJ86kyPUE652LiNUjnnEtOKvicGHmCdM7FQ3gT2znn8udNbOecS86b2M45lw8f5uOccyl4H6RzzuVHniCdcy5fArLTu4md3unbOVexSam3Ql1CsyR9I2mipHGhbDtJwyVNC//WC+WS9KCk6ZK+ltQx1bU9QTrnYhJu0qTaCu9gM9vDzDqH19cAI8ysLTAivAY4AmgbtgHAY6ku6gnSORcfZaXeiq438GzYf5Ytyxv3Bp6zyGigrqSmyS7iCdI5Fw+VWA3SgPcljZc0IJQ1NrP5YX8B0DjsN2PrVSnnhrJ8+U0a51x8Cu5nbJDbrxgMNLOBec7Z38zmSWoEDJf0beJBMzNJRVq+1ROkcy4mhRrmsyShXzFfZjYv/LtI0htAF2ChpKZmNj80oReF0+cBLRLe3jyU5cub2M65eIhiN7El1ZSUk7sPHAZMBt4C+oXT+gGDw/5bwOnhbvY+wIqEpvjveA3SOReTEhko3hh4Q1FTvRLwHzN7V9JY4FVJ/YHZQO7aDsOAI4HpwGrgzFQX9wTpnItPMZ/FNrOZwO75lP8MdM+n3IALC3t9T5DOufik+Ww+3gdZDn079BbGvnodo1++hpEvXrXVsUv7HsKarx6mft2aANSuVY1B95/LmFeuYfyg6+nba584Qi6WW6+6kMP2asOJh++7ueyxe2/j5CO6ckqP/bno9KNZvDDqRjIz7rnlKo4+eE9OPqIr306eGFfYJWLunDn0/HN3uuy5K3t33I3HHn4QgDtuu4UOf2jB/nt3ZP+9O/L+u8NijrQIpNIcB1kivAZZTh0+4AF+Xv7rVmXNG9el+z478eP8pZvLzj3hQL6duYDjLnuCBvVqMemNG3l52FjWb9hY1iEXWc/jTuGE08/hb1eev7ms7zmXcP7lNwDw8jOP8+8H7+ba2+/j84+H8+Osmbz+4QQmTxzHnTdewTNvjIgr9GKrVKkSt935T/bYsyMrV67koK57cXD3QwG44OLLuOQvV8QcYfEoK/4kmEp6R+e2yd1XHsv1D7xJ1M0SMaBWzaoA1KxelWUrVrNh46aYIiyajl32o3bdeluV1cqpvXl/zerVhE56PvlgGD2OPglJ7LbnXqz8ZQVLFi0o03hLUpOmTdljz+hx4ZycHNp36MBPPyUdlVKuCJCUcoubJ8hyyMx4+9GLGPXiVZx1zH4A9Oy2Gz8tWs4332/9y/P4y5/QYYcmzHz/dsb99zqu/OegrRJoefboPX+nx3678O5b/+Xcv1wHwOIF82ncdMuDEY2abM+iBUlHcZQrs2fP4uuJE+m8194APPn4I3Tdaw8uPLc/y5Ytizm6IpBQVuotbhmVIMOsHw2Ke07cup95H11PuYs+Fz3KuScewH4dd+Sqs/7MrY8N/d25f+q6E19/N5c/HHY9e590B/ddczw5NavFEHXJu+DKGxk6agqH9zqeV5/L+3BFxbJq1Sr6nnw8d/zzXmrXrk3/c85j4v+mMXLMBBo3acoN11wZd4hF4jVIV+J+WrwCgMXLVvHWh19zQKe2tGpWny9fuZZvh95Cs0Z1+eI/V9O4fg59e+3D4A8nATBzzhJmzfuZ9q0bp7p8uXNE7+P58L23AWjYpCkL52+pRS9a8BONmiSdi6BcWL9+PX1PPo4TTjyFXn2OAaBR48ZkZ2eTlZVFv7POZvy4sTFHWTRZWVkpt7jFH0EpkfRmeHh9SsID7LnHWkv6VtKLkqZKGiSpRsIpF0uaEOaY6xDe00XSF5K+kvS5pPZl+oWCGtWqUKtG1c37h+7bgfFTZtOq+7V06PE3OvT4G/MWLWffU+5i4c8rmbNgGd26RKE22i6Hdq0b88O8JXGEXqJ+/GHG5v1PPhhG6z+0BeDA7kcw9I2XMTO++WostXJq06BRk7jCLDYz46LzzqZ9+5246NK/bC5fMH9Lt8GQwW+y0867xBFe8agQW8wq8l3ss8xsqaTqwFhJr+U53h7ob2ajJD0NXADcE44tMbOOki4ArgTOBr4FDjCzDZIOBf4BHFs2X2WLRvVzeOXecwColJ3NK++MY/jnU5Oef+eT7zLwltMY++p1SHD9A4N/d/c73V1/SX/GjxnJ8mU/06Przgy49BpGfTyc2T9MJ0uiSbMWXHvbfQDsd/BhjPp4OEcfvCfVqtXgprsfiTn64hn9+She/s8L7LLrbuy/d3Sz5qZbbmPQqy/zzdeTkETLVq24/6HHY45024n0aEanoorSYZ+XpJuBo8PL1sCfgZeBzkAt4FMzaxnOPQS4xMz6SJoF7BdmB9kbuN3MDpXUAniQaKJNAyqbWYd8PncA0UScULlWp2q79Mt7SoX12eu3xx1CmWvXNCfuEMpUnerZ4wuaPKKwKtX/g9U+8raU5yx74dQS+7yiqJBNbEndgEOBfc1sd+ArIO+dibx/GRJfrwv/bmRLLfvvwEdmtitwVD7Xiy5iNtDMOptZZ1WqXvQv4VwG8Js08agDLDOz1aEPMb/HR1pKyn004xRgZCGumdv7f0aJROlcJhM+zCcm7wKVJE0F7gRG53POd8CF4Zx6FLA2BXA3cIekr6jYfbfOlYncPsh0rkFWyF90M1tHtDhPXq0BJNUCNpjZafm8t3XC/jigW9j/AmiXcOoNJRawcxkqHZJgKhUyQTrnyoHQxE5nGZkgzWwWsGvccTiX6bwG6Zxz+RBKi6dlUvEE6ZyLT3pXID1BOudiIrwG6ZxzyXgfpHPOJZPe+dETpHMuHpLfpHHOuaS8ie2cc0n4QHHnnEsi3WuQ6d0B4JyruFQy051Jyg4z/Q8Jr3eQNEbSdEmvSKoSyquG19PD8dYFXdsTpHMuFtGTNKm3QroUSJxW/y7gPjNrAywD+ofy/kTTILYB7gvnpeQJ0jkXGyn1VvD71RzoAfw7vBZwCDAonPIs0Cfs9w6vCce7q4BqqvdBOufiIQpTS2wgaVzC64FmlrjG7/3AVUDu2hf1geVmtiG8ngvkLpTeDJgDENaWWhHOT7qKnSdI51wsRKES5JJka9JI6gksMrPxYZmVEucJ0jkXm2LexN4P6CXpSKI1omoDDwB1JVUKtcjmbFkqZR7QApgrqRLRMio/p/oA74N0zsUjNLGLepPGzK41s+ZhFYCTgA/N7FTgI+C4cFo/YHDYfyu8Jhz/0ApY1tUTpHMuFqLUVjW8Grhc0nSiPsanQvlTQP1QfjlwTUEX8ia2cy4m2zSUJyUz+xj4OOzPBLrkc85a4Phtua4nSOdcbNL9SRpPkM65WKhww3xi5QnSORebNK9AeoJ0zsXHm9gZbNd2LRgy4l9xh1Fmrh7yv7hDKHMPHO2rBxeZN7Gdcy5/0TCfuKNIzROkcy4mJTfMp7R4gnTOxcb7IJ1zLj+FnNIsTp4gnXOxiGbzSe+nnT1BOudi4zVI55zLT3ke5iOpdqo3mtkvJR+Ocy5TiGLN2FMmUtUgpwBG1FWQK/e1AS1LMS7nXAbILq81SDNrUZaBOOcyT5pXIAs3Ya6kkyRdF/abS+pUumE55yo6ldC62KWpwAQp6WHgYKBvKFoNPF6aQTnnMkN2llJucSvMXeyuZtZR0lcAZrZUUpVSjss5lwHSoJKYUmES5HpJWUQ3ZpBUH9hUqlE55yo8AdlpniEL0wf5CPAa0FDSLcBI4K5Sjco5V/EV0P+YDn2QBdYgzew5SeOBQ0PR8WY2uXTDcs5lgjTIgSkV9kmabGA9UTM7vR+edM6VCyL9x0EW5i729cBLwPZAc+A/kq4t7cCccxVfuW9iA6cDe5rZagBJtwNfAXeUZmDOuYpNSv8aZGES5Pw851UKZc45VyzpnR5TT1ZxH1Gf41JgiqT3wuvDgLFlE55zrqIqiT5ISdWAT4GqRPlskJn9TdIOwMtAfWA80NfMfpNUFXgO6AT8DJxoZrOSXT9VDTL3TvUUYGhC+egifhfnnNuiZPoZ1wGHmNkqSZWBkZLeAS4H7jOzlyU9DvQHHgv/LjOzNpJOIhqyeGKyi6earOKp4kbunHOpFDc/mpkBq8LLymEz4BDglFD+LHAzUYLsHfYBBgEPS1K4zu8U2AcpaUfgdmBnoFpCYO227au40rBixXKuvvR8vp86BST++dATPP34w8yc/j0Av6xYTu06dXnnky9jjrRoKmeJmw5vS+WsLLKzYMzs5QyatICGtapwyQGtqVW1Ej8sXc0jI2ezcZPRoGZlzu3aitrVKrFq3QYeGTmbpavXx/01im3jxo0cdtA+NGnajBf/+yZmxh1/v4m333iN7Oxs+vU/l3POvyjuMLdJIZvYDSSNS3g90MwGbnUdKZuoGd2G6MGWGcByM9sQTpkLNAv7zYA5AGa2QdIKomb4kvw+vDA3aZ4BbgPuAY4AziQ8dujid8u1V3BQ9z/x+DMv8dtvv7FmzWoeeeqFzcf/fuPV1K6dcu7jtLZ+k3Hb+9NZt2ET2YKbD2/HxHm/0GPnRgybuogvZi2n/94tOLhNfT74fgmndmrGZzOW8unMpezSpBYn7bk9j46aHffXKLYnH3uItu06sHLlSgBefvE5fpo7l1HjJ5OVlcXixYtijrBoCtHEXmJmnVOdYGYbgT0k1QXeADqUUHiFGvRdw8zeC4HMMLMbiBKli9kvv6xgzBcjOem0MwGoUqUKderU3XzczBj65iB6HZO0i6VcWLchevQ/d4YXA3ZpksOY2csB+HTGz3RuWQeA5nWrMXlBlESmLFhFpxZ1Yom5JP00by7D33uHU/udtbnsmX8/wRVXX7950auGDRvFFV6RSdGz2Km2bWFmy4GPgH2BupJyK4DNgXlhfx7QIvp8VQLqEN2syVdhEuS6MFnFDEnnSToKyNmmyF2pmDN7FvXrN+TKi87hiG57c9Wl57H61183H//yi5E0aNiYHXZsE2OUxSfBHT3b88QJu/HN/JUsWrmOX3/byKbQjvl59Xq2q14ZgNnL1tClZfRHYq+WdahRJZtaVbPjCr1E3HjNFdx06x1brQA4+4eZvPn6fznsoH04+ZijmDl9WowRFp2Ueiv4/WoYao5Iqg78CZhKlCiPC6f1AwaH/bfCa8LxD5P1P0LhEuRfgJrAJcB+wDnAWSnfkYKkWZIaFPecInzuqoLP2ur8PpJ2LskYStrGDRuY/PVXnHbmAN75eAw1atTk0Qf+ufn4W6+9Sq9jT4gxwpJhBtcO+Y4LB01hxwY12L5OtaTnvjhuHjs1rsUdPduzU+Na/Pzrb2wqx3NPvf/OUBo0aMTue3bcqnzdb+uoVq0a738ymtPOOIvLLhwQU4TFUwJP0jQFPpL0NdHww+FmNgS4Grhc0nSiPsbcm85PAfVD+eXANakuXpjJKsaE3ZVsmTQ3bUm6GZhlZs8U81J9gCHA/4obU2lpsn0zmm7fjD07dwHgyF5H8+gD9wCwYcMG3h06mCEjPo8zxBK1ev1G/rdgFW0b1qRmlWyyBJsM6teozNI10Y2YZWs2cN8nPwBQtVIWXVrWZfX6jXGGXSxfjvmc994Zwojh77J27VpWrfyFC87ux/bbN+PIo/oAcORRfbj0gnNijnTbieJPimtmXwN75lM+E+iST/la4PjCXj9pDVLSG5JeT7YV5uKS3pQ0XtIUSQPyHGst6VtJL0qaKmmQpBoJp1wsaYKkbyR1CO/pIukLSV9J+lxS+8J+0fD++0IsIyQ1DGU7Sno3xPmZpA6SugK9gH9KmhjOOUfSWEmTJL2WJ9ZYNGrchKbNmjNjWnTHetSnH9G2/U4AjPzkQ3Zs246mzZrHGWKx5VStRI3KURO5crbYrWkO85avZcqClezdKmpKH7hjfcbPWRHOz978dEbvXRvz8fSk3Uvlwg03387Eb39g3ORpPPF/L7DfgQfz6L+f5fCevRj12ScAfD7yU3bcsW3MkRZBAc3rNHgUO2UN8uESuP5ZYQby6sBYSa/lOd4e6G9moyQ9DVxAdLccortXHSVdAFwJnA18CxwQbs8fCvwDOLaQsdQExpnZXyTdBPwNuAgYCJxnZtMk7Q08amaHSHoLGGJmgwAkLTezJ8P+bUQDTh/K+yHhD8EAgGbNS3/ds1vuvI9Lzz2D9et/o2WrHbjn4WgExNuvv1rub84A1KteifP3b0WWhIDRs5fz1bxfmLdiLRcf2JoT9tieWUtX89G0KBHu1DiHkzo2BWDqwlX835i5MUZfei75y1VccHY/nnjkAWrWrMW9D5fPVVDSfcJcpeifLP7Fo+bu0eFla+DPRI//dAZqAZ+aWctw7iHAJWbWR9IsYD8zmxeS1u1mdqikFsCDQFuioUaVzayDpN2A58PnNAF+I3pEEqC7mf0saSNQNSTXPwCvA/sDi4HvEsKuamY7SXqGrRPkQUTDneqG2N8zs/NSff8/7tHJhnxYcZq4Bbl6SNr2RpSaB47eNe4QylTj2lXGFzTsptDXarOrnXjPoJTnPHT0TiX2eUVR2Pkgt5mkbkST7O5rZqslfUzCQPMgb3ZOfL0u/LuRLXH+HfjIzI6W1Br4GMDMvgH2CJ97M4Xrg8yd23K5me1RiK/0DNDHzCZJOgPoVoj3OOdSSPPJfEp18ts6RM88rg59iPvkc05LSfuG/VOIlnMo6Jq545nO2MZ4sthy2/8UYKSZ/QL8IOl4AEV2D+esZOvhTDnA/PC856nb+NnOuTxypztL51UNC50gwywY2+JdoJKkqcCd5D/JxXfAheGcekTPSqZyN3CHohUWt7X2+yvQRdJkouc0bw3lpwL9JU0impijdyh/GfhruCG0I3AjMAYYRdQX6pwrpiyl3uJWmGexuxCNHapDVOPbHTjbzC5O9T4zW0f+T9y0DtetBWwws9PyeW/rhP1xhOasmX0BJD4DfkM+7705STy1kpT/AByeT/kooufPcz1GwQncOVdIFWLJBaKbIj0Jj+OY2STg4NIMyjmXGbIK2OJWmGZqlpnNzjOqvdgjb8MklZl1C9A5t5mUHv2MqRQmQc4JzWwL0wpdDHxfumE55zJBmg+DLFSCPJ+omd0SWAh8EMqcc65Y0rwCWahnsRcBJ5VBLM65DFIebtIU5i72k+QzQa6Zlc/pQ5xz6SFNhvKkUpgm9gcJ+9WIHh2cUzrhOOcyhUj/Z7EL08R+JfG1pOcp+IkX55wrUEWoQea1A9C4pANxzmWeQk6KG5vC9EEuY0sfZBbRLDkpZ+F1zrmCRM9ixx1FaikTpKL0vjtbJojYlGr9Buec2xZZ5bkGaWYmaZiZ+RMvzrkSFQ3ziTuK1AoT3kRJv1vzwTnnikdkFbDFLWkNUlIlM9tAtCDOWEkziKYME1HlsmOy9zrnXEHKex/kl0BHosWrnHOuxJXnPkgBmNmMMorFOZdBRPmerKKhpMuTHTSze0shHudcBkn3Z7FT9QBkE63el5Nkc865IhPFnzBXUgtJH0n6X1jz/tJQvp2k4ZKmhX/rhXJJelDSdElfS0p5LyVVDXK+md2a4rhzzhWdSqQPcgNwhZlNkJQDjJc0nGhRvxFmdqeka4gebrmaaBmYtmHbm2gZlb2TXTxVkk7vuq9zrlwTUYJMtRXEzOab2YSwvxKYCjQjWnzv2XDas0CfsN8beM4io4G6kpomu36qBNm9wOicc64YVMAGNJA0LmFLOs2ipNZEwxLHAI3NbH44tIAt80c0Y+vZyOaGsnwlbWKb2dLkX8s554pLZBV8k2aJmXUu8ErRKqmvAZeZ2S+Jk2CEJwKL9Ih0mg/TdM5VVCVxkwZAUmWi5Piimb0eihfmNp3Dv4tC+TygRcLbm7Nlronf8QTpnItNcfsgw4Q6TwFT8ww9fAvoF/b7AYMTyk8Pd7P3AVYkNMV/pyjzQbpCqpwtGtWuGncYZeax4/4YdwhlbtTMJXGHUH6pROaD3A/oC3wjaWIouw64E3hVUn9gNnBCODYMOBKYDqwGzkx1cU+QzrlYlMSSC2Y2kuQjbn53ozlM13hhYa/vCdI5F5t0H0voCdI5F5vy/Cy2c86VmgqxqqFzzpUOoTRvZHuCdM7FwmuQzjmXjLwP0jnnkvIE6Zxz+fAmtnPOpeA3aZxzLonyvGiXc86VmmjC3LijSM0TpHMuHoWcsSdOniCdc7FJ7/ToCdI5F5PcNWnSmSdI51xs0jw/eoJ0zsXHh/k451wSfhfbOeeS8QTpnHO/F619nd4Z0hOkcy4e8ia2c84l5wnSlaZzzz6Ld4YNoWGjRoyfOBmApUuX0veUE5k9exatWrXmhZdepV69ejFHWjLWrl1Lj8O6sW7db2zcuIFefY7h2htu5uLzz+GrCeMxM9q0bcsjTzxNrVq14g63SO6/8TLGfjqcOts14NE3PgHgrisHMHfWDAB+XbmCmjl1eGjQCBbO+5Hzex9Is9Y7AtD+j5246Ka7Y4t926T/kzRZcQfgiqdvvzMYPOTdrcruuftOuh3SnclTp9HtkO7cc/edMUVX8qpWrcrgYR8wcswEPv1iPCOGv8fYL0dz+13/YuSYCYz68iuaN2/Bk48/EneoRXZo7xO55bGXtiq7+p6BPDRoBA8NGkHXQ3vQtfuRm481adFq87Hykxxz+yBTb3HzBFnO7X/AgWy33XZblQ15ezCn9e0HwGl9+/H2W2/GEVqpkLS5Zrh+/XrWr9+AJGrXrg2AmbFm7dqSWJA+Nrt23pecOnXzPWZmjHzvbQ488ugyjqqUpHmG9ARZAS1auJCmTZsC0KRJExYtXBhzRCVr48aNHLBPJ9q1bkq3Q7rTea+9Abjw3P6036EZ077/lgHnXxRzlKVjyvjR1K3fgGat/rC5bOG8H7nk+EO55ow+TB4/Osbotl1WmLAi2Ra3jEmQklpLmrwN558hafvSjKksSCrXtan8ZGdn89no8Uz5fjYTxo/lf1Oi/6yPPPEUU2fMoV37nXhj0KsxR1k6Pnnnja1qj9s1bMz/vT+eB//7AWf/9RbuufoCVq9aGWOE26a4FUhJT0talPi7LWk7ScMlTQv/1gvlkvSgpOmSvpbUsaDrZ0yCLIIzgHKZIBs1bsz8+fMBmD9/Pg0bNYo5otJRp25dDjiwGyOGv7e5LDs7m2OOO4G3Br8eY2SlY+OGDXzxwTAO/HPvzWWVq1Sldt2oi6XNLrvTpEUr5s2eEVeI20Zb/oAn2wrhGeDwPGXXACPMrC0wIrwGOAJoG7YBwGMFXTzTEmQlSS9KmippkKQakjpJ+kQbOSdBAAAQGklEQVTSeEnvSWoq6TigM/CipImSqku6SdJYSZMlDVQaV8t69OzFC88/C8ALzz9Lz6N6F/CO8mPJ4sWsWL4cgDVr1vDRhx/Qpl07Zs6YDkR9dO8OfZt27drHGWapmDj6U5rv0IYGTbb83V6xdAkbN24EYMGc2fz04w80ad4qrhC3iYgmq0i1FcTMPgWW5inuDTwb9p8F+iSUP2eR0UBdSU1TXT/Thvm0B/qb2ShJTwMXAkcDvc1ssaQTgdvN7CxJFwFXmtk4AEkPm9mtYf95oCfwdt4PkDSA6K8TLVq2LPUvdPppJ/PZJx+zZMkSdmzdnBtvuoUrr7qG004+gWf/7ylatmzFCy9VnObmggXzuWDAWWzcuJFNmzZx9LHH8efDe3DEnw5i5S8rMTN23e2P/OuB8nsX++6rzuObsZ/zy/Kl9Ou+J6de+FcOO+YUPn3nzd/dnJk8fjQvPnI32ZUqk5WVxYU33k1OnfIzpKsQSbCBpHEJrwea2cAC3tPYzOaH/QVA47DfDJiTcN7cUDafJGRmBUZYEUhqDXxqZi3D60OA64AuwMxwWjYw38wOk/QxWyfIY4GrgBrAdsBDZpZy/EynTp1t1JhxqU6pUNb+tjHuEMrcqJlL4g6hTPXcrcl4M+tcEtfadfeONujdkSnP2Wn7mgV+XvjdHmJmu4bXy82sbsLxZWZWT9IQ4E4zGxnKRwBX5/6O5yfTapB5/xqsBKaY2b6p3iSpGvAo0NnM5ki6GahWOiE6lzlKqaNqoaSmZjY/NKEXhfJ5QIuE85qHsqQyrQ+ypaTcZHgKMBpomFsmqbKkXcLxlUBO2M9Nhksk1QKOK6uAnavIitsHmcRbQL+w3w8YnFB+eribvQ+wIqEpnq9Mq0F+B1wY+h//BzwEvAc8KKkO0c/jfmAK0d2xxyWtAfYFngQmE/VpjC370J2rWEpiNh9JLwHdiPoq5wJ/A+4EXpXUH5gNnBBOHwYcCUwHVgNnFnT9jEmQZjYL6JDPoYnAgfmc/xrwWkLRDWFzzpWEEpjNx8xOTnKoez7nGtGN2ULLmATpnEtDaTtYLuIJ0jkXE/mEuc45l59o2de4o0jNE6RzLj6eIJ1zLn/pMGNPKp4gnXOxSe/06AnSOReXMJtPOvME6ZyLRe5sPunME6RzLjZpnh89QTrn4uM3aZxzLpn0zo+eIJ1z8VAJPItd2jxBOudi448aOudcMumdHz1BOufi401s55zLl8/m45xz+fKB4s45l4InSOecy498oLhzzuVLpP1NbE+QzrkYpXmG9ATpnIuNN7Gdcy6J9E6PniCdczFK9wlzFa2l7UqDpMXA7Bg+ugGwJIbPjUumfV+I7zu3MrOGJXEhSe8SfY9UlpjZ4SXxeUXhCbICkjTOzDrHHUdZybTvC5n5neOQFXcAzjmXrjxBOudcEp4gK6aBcQdQxjLt+0Jmfucy532QzjmXhNcgnXMuCU+QzjmXhCdI55xLwhOkc84l4QnSuQpC6f7cXjnkCbKCy/2lkZQddyzpKL+kUl4STd44zYeklDgf5lOBSZKZmaQewEnAQmComX0Uc2hpIeHn0w3oCEwFxpnZ4txj8UaYXGJ8kk4GmgNfAt+Y2dJYg6tAvAZZgYVf/iOBvwOPAdsB/5bUG8pPTam0hJ9PT+A+YCNwEXCdpJbhWNr+fBKS4yXABcBvwD+A8yS1iDO2isQTZAUmqSqwK3AqUB9oC9wN/EtSj3SuIZWWxKQnqSlwAtAbmAm0JJqi8K+Smqfjz0dSVsL+zsDewMGhqAqwPXC6pGYxhFfheBO7gsnbNJRUA6gH/Ae42My+ljQC6ADsQTSdVEb8TyCpOtDczKZJ6gCsBKoD1YBngWOJfi7/AkYBl5jZ2rjizUtSfaCmmf0o6RDgM6LpwnYGrjOz7pLOAy4FngbuM7MN8UVc/vmEuRWEpJpm9mtoGh5G9Evzk5m9Kmk90byUGyQdCPyP6Jd/cZwxx2BH4OjwR+NU4EAzmx5+JpPMbFaoeY0hSi5pkxyDdsAAScuBnkBnM5sv6VBgQThnBTAaeMaTY/F5E7sCkFQL+FTSYZLaAo8QNbV6SXrYzNYT/QJdT1ST/MDMpsQXcTzMbDJRM/RyYKCZzQqHpgLHSHoOeBUYZGZT44kyOTP7gqiv8VzgWjNbEQ4NBQ6WNAS4Fbg7A//4lQpvYlcQkk4HbgA+BAab2TshWd4EzDWzayXVBhqa2Yx0v0tbkvLc8e0AdAd2B4YDn5jZIknNifrz5pjZl/FFu7V8uky6AF2Aw4m6Aj43s3WSGgD7AJMTEr8rJm9il3Oh097M7DlJq4AHgDnAO8AM4BbgLklPmNm5wC+QOWPmEobyHAF0Ivp//h9EN2Z6AqtCDXwX4A4zWxdftL+XkNgHAI1C8Z3AOuAq4IYwTCnHzG6OI8aKzBNkORZ++TdJaiNpqZm9LmkZ8LSkCaEWOQO4BqgRc7ixCMnxcKKm5+VEta7mZna2pHpAD+AQ4MZ0S465wlCeXkTDte4HNpnZP0Jf6mVAe6JmtythniDLqYSa0Z+JxvFND4uE3Q2cBTwhqbKZvQVMizPWNHAA0U2ZnYj68G4DMLOBuXf5zWxeunQ7SMoKf/hy42kBHAFcAswnGqZV2cwekFQXwMyWxxhyheV9kOWYpK7A+cC9wBqiPqjjgTOBA0N5p0zqsM8vyUn6F9AaqAOcG/pgjyUa/vS0mW0q+0gLJmkHoqefXgSqEjWrTzCz9ZLOBRaa2ZtxxljR+V3sckRSa0lXhP2qRH1QB5nZV2b2LTAE+A443MwGAftkUnKErfrs9gxbC+AhYH9gWEiO+xP1Q05Pp+Qoqaukk8L+xcDrwO1Ed947Ai+E5HgGUdP6m7hizRTexC5fDLhCUtXQB3UJ8LakgWY2wMyWSFpJdIcWotpHRghPxbxkZt0k7QEMJhpI3ZRoaNMhwJuS/gjsBlxuZh/HFW8S9YA7JO1CNGbzuPDvj0R9yA+H5+p3A441sxmxRZohPEGWE5IqmdlsSfsCQyRVMbObwy/MfyS9DzwMHEk0tAcz2xhjyGUqDJheJekbYARwvJmNkbQb8G9gHtEwnhpANTObHmO4+TKzoZJ+I+pTnhRqu3OJRiU0JQzrAbIzrWUQF++DLAcSbsi0B5YS9Ue9A7xsZreHMXxvA9nAaeFxwkqZ8iSFpOzcPwaSHifqg+1qZuNDWV9gFzO7JsYwC03RZCLPAOeZ2SuhbDDwfOg6cWXEa5DlQEiORxHdfZ0FTAf+CtwnaaOZ3SmpF/AgcCHRjYiMSI4Q1ZRDt8M6MztPUh3gBaK71hD1te8U7vyujy/SwjGzwSGpPyhpJ2Ai8Afgq3gjyzyeIMsBSfsQNZv/FLaBwFrgL8DAUFu8TdJlwL2SmpjZguRXrDgktTWzaYljGM3sZEmDJM0j+qOxP/BkeUiOucxsiKRKwGtEN996m9nMmMPKON7ELgdCE7opUSf+bcApwBNEz1cPApaa2Sfh3ExqWh9J9Njg62Y2StKlQB0zuzUcf4loIPi+ZjYlXcY5bgtJBwGz/fHBeHiCLEck3Q4sCgOETyea1upoi6a/Kne//MUV7lz3BdYTTc6xH9E4wbkJ5+ySiRNzuJLhTezy5RvgXEmVgWOIpiz7ETLn2epc4cbMfOBuSW8QPWf9l9zkmNvfmJscM/EPiCs+Hyhevgwjuru5L3C7mY2KN5z4JNy1Pp+o++FFoKWk7uH4+jzne3J028yb2OVQbj9jJtWKEoY6JQ7paUk0e9E5YZD8zURLJtxvZstiDNdVEJ4gy6FMSoyQenVGSdXNbE04ryHRtF9+t9eVCG9il0OZlBwh5eqMvcxsjaSsULNc7MnRlSRPkC7tKfnqjPcqWp1xUyY9VunKjjexXVrK240gX53RxcCH+bi0Il+d0aURb2K7tCFfndGlGW9iu7QiX53RpRGvQbq0EO5Ey8yeI1pkrAfRLNqwZXXGdopWZ/wld7JYT46uNHkN0sUuYZxjG6KJN5ZKOhh4Grgg1CIFtAFqmNmkWAN2GcMTpItVQnLcvDojkLs64/ZEsxZdadHqjM6VKU+QLnby1RldmvIE6cqcpNZEi079KwwCfwXoaGYtw/EGwHXARDN7TtL2ZvZTbAG7jOU3aVwccldnvC7MBH4JsEzSQAAzWwJk5OqMLr14gnRlKnd1RqIp206WdHOY07IH0EHS+2F9nSOBDyCzVmd06cWb2K7M+OqMrrzxBOnKVD6rMw4nunv9bFidsQXRQluLzOzc2AJ1Dm9iuzKUZ3XGV4HzgAOIVme8QNINZjYHuAxoIKlJbME6h9cgXRny1RldeeM1SFdmzGyumY0FDgJeNLPpwLNEU5aNN7NPwhMzeHJ06cCnO3Nx8NUZXbngCdLFYRjRHexeZPjqjC69eR+ki00mrs7oyhfvg3Rx2gjerHbpy2uQzjmXhNcgnXMuCU+QzjmXhCdI55xLwhOk22aSNkqaKGmypP+GNauLeq1ukoaE/V6Srklxbl1JFxThM26WdGVhy/Oc84yk47bhs1pLmrytMbr05AnSFcUaM9vDzHYFfiN6pnozRbb5/y0ze8vM7kxxSl1gmxOkc0XlCdIV12dAm1Bz+k7Sc8BkoEVY3/oLSRNCTbMWgKTDJX0raQLRkzSE8jMkPRz2G0t6Q9KksHUF7gR2DLXXf4bz/ipprKSvJd2ScK3rJX0vaSTQvqAvIemccJ1Jkl7LUys+VNK4cL2e4fxsSf9M+GyfeagC8gTpikxSJeAIokcHAdoCj5rZLsCvROtbH2pmHYFxwOWSqgFPAkcBnYBkM/Y8CHxiZrsTLf86hWg52Bmh9vpXSYeFz+wC7AF0knSgpE7ASaHsSGCvQnyd181sr/B5U4H+Ccdah8/oATwevkN/YIWZ7RWuf46kHQrxOa4c8UcNXVFUlzQx7H8GPEW0AuFsMxsdyvcBdgZGhfknqgBfEE1M8YOZTQOQ9AIwIJ/POAQ4HTbPKL5CUr085xwWtq/C61pECTMHeMPMVofPKMyKiLtKuo2oGV8LeC/h2KtmtgmYJmlm+A6HAX9M6J+sEz77+0J8lisnPEG6olhjZnskFoQk+GtiETDczE7Oc95W7ysmAXeY2RN5PuOyIlzrGaCPmU2SdAbQLeFY3qcpLHz2xWaWmEhzFyRzFYQ3sV1pGQ3sJ6kNgKSaktoB3wKtJe0Yzjs5yftHEC0Fm9vfV4doIa+chHPeA85K6NtsJqkR8CnQR1J1STlEzfmC5ADzwwxDp+Y5drykrBDzH4DvwmefH85HUjtJNQvxOa4c8RqkKxVmtjjUxF5StLQrwA1m9r2kAcBQSauJmug5+VziUmCgpP5Ez2yfb2ZfSBoVhtG8E/ohdwK+CDXYVURr2UyQ9AowCVgEjC1EyDcCY4DF4d/EmH4EvgRqA+eZ2VpJ/ybqm5wQ5rBcDPQp3E/HlRf+LLZzziXhTWznnEvCE6RzziXhCdI555LwBOmcc0l4gnTOuSQ8QTrnXBKeIJ1zLon/BzmPMCnSGBE6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mlutils.plot_confusion_matrix(cm, classifier.classes_, normalize=False, title='Confusion Matrix')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "The three classification methods: SVM, Logistic Regression, and Neural Network perform about the same on the test dataset. We have not optimized any parameters. We leave this as an excercise for the reader.\n",
    "\n",
    "For all three methods, the prediction of the mixed class: alpha+beta has the lowest precision and recall. \n",
    "\n",
    "**Limitations of the Model**\n",
    "\n",
    "The feature vectors are created by summing ProtVectors for all 3-grams of a protein sequence and this process averages individual contributions. This step may contribute to the lower performance for the alpha+beta class, because alpha and beta related features are averaged together. In addition, the beta class is underrepresented.\n",
    "\n",
    "The limiting factor appears to the expressiveness of the feature vector using the ProtVec model. Furthermore, we used a cutoff of 25% alpha and/or beta content to define the fold classes. This means that the model will not perform well on protein sequences with minimal alpha or beta content.\n",
    "\n",
    "Alternative feature vectors can be easily explored by replacing the CalculateFeatures step with another method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the Classification Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['./intermediate_data/classifier']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(classifier, \"./intermediate_data/classifier\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Step\n",
    "After you saved the classification model here, run the next step in the workflow [4-Predict.ipynb](./4-Predict.ipynb) or go back go back to [0-Workflow.ipynb](./0-Workflow.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "**Authors:** [Peter W. Rose](mailto:pwrose.ucsd@gmail.com), Shih-Cheng Huang, UC San Diego, October 1, 2018\n",
    "\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}