ocr_project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mini-Project OCR Code\n",
    "\n",
    "This is a walkthrough of the process we went through to develop an OCR model that recognizes handwriting.\n",
    "\n",
    "Our libraries used are listed in the README, we will utilize requirements.txt to load them all at once.\n",
    "\n",
    "**IMPORTANT** Make sure to use `python3.6` (the version we use for class projects) to run our program with little issues. Information can be found [here](https://courses.cs.vt.edu/cs4804/Spring24/projects/project0.html#python-installation) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "hide_output"
    ]
   },
   "outputs": [],
   "source": [
    "# Install libraries\n",
    "%pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load libraries\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import cv2\n",
    "import keras\n",
    "\n",
    "from keras import layers, models\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelBinarizer\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.model_selection import KFold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helper functions for loading dataset\n",
    "\n",
    "We need to load our dataset. We will create a helper function for the model OCR. This function will load the English Handwritten Characters dataset that should be in given path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_eng_dataset(datasetPath):\n",
    "\t # initialize the list of data and labels\n",
    "    data = []\n",
    "    labels = []\n",
    "\n",
    "    # loop over the rows of the A-Z handwritten digit dataset\n",
    "    for row in open(datasetPath):\n",
    "        # Skip the first row\n",
    "        if row == \"image,label\\n\":\n",
    "            continue\n",
    "\n",
    "        # parse the label and image from the row\n",
    "        row = row.split(\",\")\n",
    "        imagePath = \"eng_dataset/\" + row[0] # hardcode the path\n",
    "        try:\n",
    "            image = cv2.imread(imagePath)\n",
    "            image = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)\n",
    "        except cv2.error as e:\n",
    "            print(\"[ERROR] loading image \", row[0], \" fail\")\n",
    "            continue\n",
    "        \n",
    "        label = row[1][:-1] if len(row[1]) > 1 else row[1] # remove '\\n' at end\n",
    "\n",
    "        # update the list of data and labels\n",
    "        data.append(image)\n",
    "        labels.append(label)\n",
    "\n",
    "    # convert the data and labels to NumPy arrays\n",
    "    data = np.array(data)\n",
    "    labels = np.array(labels, dtype=\"U1\")\n",
    "\t# return a 2-tuple of the English Handwritten Characters data and labels\n",
    "    return (data, labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset Pre-Processing\n",
    "\n",
    "Next we will pre-process the dataset in order to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_dataset(data, labels):\n",
    "    \"\"\"\n",
    "    Help function to pre-process the dataset for ready to train model.\n",
    "    \"\"\"\n",
    "    # the architecture we're using is designed for 32x32 images,\n",
    "    # so we need to resize them to 32x32\n",
    "    data = [cv2.resize(image, (32, 32)) for image in data]\n",
    "    data = np.array(data, dtype=\"float32\")\n",
    "\n",
    "    # add a channel dimension to every image in the dataset and \n",
    "    # data = np.expand_dims(data, axis=-1)\n",
    "\n",
    "    # scale the pixel intensities of the images from [0, 255] down to [0, 1]\n",
    "    data /= 255.0\n",
    "\n",
    "    # convert the labels from integers to vectors\n",
    "    le = LabelBinarizer()\n",
    "    labels = le.fit_transform(labels)\n",
    "\n",
    "    # account for skew in the labeled data\n",
    "    classTotals = labels.sum(axis=0)\n",
    "    classWeight = {}\n",
    "    # loop over all classes and calculate the class weight\n",
    "    for i in range(0, len(classTotals)):\n",
    "        classWeight[i] = classTotals.max() / classTotals[i]\n",
    "\n",
    "    return data, labels, classWeight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verification\n",
    "\n",
    "To verify that the dataset looks correct, let's plot the first 25 images from the training set and display the class name below each image.\n",
    "\n",
    "We define the helper function here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_train_data(train_images, train_labels):\n",
    "    class_names = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', \n",
    "                   'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'G', 'K', 'L', 'M', \n",
    "                   'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z',\n",
    "                   'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'g', 'k', 'l', 'm', \n",
    "                   'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n",
    "\n",
    "    plt.figure(figsize=(10,10))\n",
    "    for i in range(25):\n",
    "        plt.subplot(5,5,i+1)\n",
    "        plt.xticks([])\n",
    "        plt.yticks([])\n",
    "        plt.grid(False)\n",
    "        plt.imshow(train_images[i])\n",
    "        # The CIFAR labels happen to be arrays, \n",
    "        # which is why you need the extra index\n",
    "        index = np.where(train_labels[i] == 1)[0][0]\n",
    "        plt.xlabel(class_names[index])\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and Pre-Process\n",
    "\n",
    "First, we need to load and pre-process the data. We will use the functions we defined previously.\n",
    "\n",
    "Make sure the English Handwritten Characters dataset is loocated in `/eng_dataset` in the following format:\n",
    "\n",
    "```\n",
    ".\n",
    "├── ocr_project.ipynb\n",
    "└── eng_dataset\n",
    "    ├── english.csv\n",
    "    └── Img\n",
    "        ├── imgXXX-XXX.png\n",
    "        └── ...\n",
    "```\n"
   ]