PYNQ Computer Vision demo: 2D Filter and Dilate

filter2d_and_dilate_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PYNQ Computer Vision demo: Filter2D and Dilate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is a modified version of the [filter2d_and_dilate.ipynb](https://github.com/Xilinx/PYNQ-ComputerVision/blob/master/notebooks/computer_vision/filter2d_and_dilate.ipynb) notebook which is part of the [Xilinx PYNQ Computer Vision](https://github.com/Xilinx/PYNQ-ComputerVision) project. Results of this notebook running on the PYNQ-Z1 can be viewed on this [Youtube video](https://www.youtube.com/watch?v=tzQlyEj71Us).\n",
    "\n",
    "To run this notebook on your PYNQ-Z1:\n",
    "1. Follow the install instructions for the computer vision project\n",
    "2. Copy this notebook to your PYNQ-Z1 file system\n",
    "3. Make the connections described in this [demo video](https://www.youtube.com/watch?v=tzQlyEj71Us)\n",
    "\n",
    "Read more about the demo on this [FPGA Developer blog post](http://www.fpgadeveloper.com/2018/03/pynq-computer-vision-demo-2d-filter-and-dilate.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Program overlay\n",
    "\n",
    "Here we program the overlay and load the pynq python libraries for a memory manager and the accelerator drivers.\n",
    "NOTE: All overlay and python libraries should be loaded prior to assigning the HDMI input/outputs. This is necessary right now to ensure correct functionality but will be enhanced in future releases. For now, please copy this block as is when using it in your own designs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pynq.lib.video import *\n",
    "from pynq_computervision import BareHDMIOverlay\n",
    "base = BareHDMIOverlay(\"/opt/python3.6/lib/python3.6/site-packages/\"\n",
    "                       \"pynq_computervision/overlays/computer_vision/\"\n",
    "                       \"xv2Filter2DDilate.bit\")\n",
    "from pynq import Xlnk\n",
    "mem_manager = Xlnk()\n",
    "import pynq_computervision.overlays.computer_vision.xv2Filter2DDilate as xv2\n",
    "\n",
    "hdmi_in = base.video.hdmi_in\n",
    "hdmi_out = base.video.hdmi_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup and configure HDMI drivers \n",
    "\n",
    "~15 seconds to initialize HDMI input/output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<contextlib._GeneratorContextManager at 0x2f2970f0>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hdmi_in.configure(PIXEL_GRAY)\n",
    "hdmi_out.configure(hdmi_in.mode)\n",
    "\n",
    "hdmi_in.start()\n",
    "hdmi_out.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup up HDMI input/output parameters \n",
    "\n",
    "These parameters are referenced in later function calls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My mode: VideoMode: width=1280 height=720 bpp=8\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'VideoMode: width=1280 height=720 bpp=8'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mymode = hdmi_in.mode\n",
    "print(\"My mode: \"+str(mymode))\n",
    "\n",
    "height = hdmi_in.mode.height\n",
    "width = hdmi_in.mode.width\n",
    "bpp = hdmi_in.mode.bits_per_pixel\n",
    "\n",
    "str(hdmi_out.mode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run software Filter2D + Dilate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frames per second:  5.319235093125142\n",
      "Frames per second:  4.503973190394415\n",
      "Frames per second:  4.627154958597521\n",
      "Frames per second:  4.627049294217074\n",
      "Frames per second:  5.090251790582316\n",
      "Frames per second:  5.158016825525103\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import cv2\n",
    "\n",
    "kernel_g = {\n",
    "        'laplacian high-pass':  np.array([[0.0, 1.0, 0],[1.0, -4, 1.0],[0, 1.0, 0.0]],np.float32),\n",
    "        'gaussian high-pass': np.array([[-0.0625,-0.125,-0.0625],[-0.125,0.75,-0.125],[-0.0625,-0.125,-0.0625]],np.float32),\n",
    "        'average blur':  np.ones((3,3),np.float32)/9.0,\n",
    "        'gaussian blur': np.array([[0.0625,0.125,0.0625],[0.125,0.25,0.125],[0.0625,0.125,0.0625]],np.float32),\n",
    "        'Sobel Ver': np.array([[1.0,0.0,-1.0],[2.0,0.0,-2.0],[1.0,0.0,-1.0]],np.float32),\n",
    "        'Sobel Hor': np.array([[1.0,2.0,1.0],[0.0,0.0,0.0],[-1.0,-2.0,-1.0]],np.float32)}\n",
    "\n",
    "kernelD = np.ones((3,3),np.uint8)\n",
    "buf     = np.ones((height,width),np.uint8)\n",
    "\n",
    "numframes = 100\n",
    "\n",
    "def test_filter(kernelName):\n",
    "    start = time.time()\n",
    "    for _ in range(numframes):\n",
    "        inframe = hdmi_in.readframe()\n",
    "        outframe = hdmi_out.newframe()\n",
    "        cv2.filter2D(inframe, -1, kernel_g.get(kernelName, np.ones((3,3),np.float32)/9.0), dst=buf)\n",
    "        cv2.dilate(buf, kernelD, dst=outframe, iterations=1)\n",
    "        inframe.freebuffer()\n",
    "        hdmi_out.writeframe(outframe)\n",
    "    end = time.time()\n",
    "    print(\"Frames per second:  \" + str(numframes / (end - start)))\n",
    "\n",
    "# Test the different filters one by one\n",
    "test_filter('laplacian high-pass')\n",
    "test_filter('gaussian high-pass')\n",
    "test_filter('average blur')\n",
    "test_filter('gaussian blur')\n",
    "test_filter('Sobel Ver')\n",
    "test_filter('Sobel Hor')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run hardware accelerated filter2D + dilate "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frames per second:  60.013129245613996\n",
      "Frames per second:  60.01214034583174\n",
      "Frames per second:  60.016618561934905\n",
      "Frames per second:  60.01684542447948\n",
      "Frames per second:  60.0161190409898\n",
      "Frames per second:  60.014465242299245\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "kernel_g = {\n",
    "        'laplacian high-pass':  np.array([[0.0, 1.0, 0],[1.0, -4, 1.0],[0, 1.0, 0.0]],np.float32),\n",
    "        'gaussian high-pass': np.array([[-0.0625,-0.125,-0.0625],[-0.125,0.75,-0.125],[-0.0625,-0.125,-0.0625]],np.float32),\n",
    "        'average blur':  np.ones((3,3),np.float32)/9.0,\n",
    "        'gaussian blur': np.array([[0.0625,0.125,0.0625],[0.125,0.25,0.125],[0.0625,0.125,0.0625]],np.float32),\n",
    "        'Sobel Ver': np.array([[1.0,0.0,-1.0],[2.0,0.0,-2.0],[1.0,0.0,-1.0]],np.float32),\n",
    "        'Sobel Hor': np.array([[1.0,2.0,1.0],[0.0,0.0,0.0],[-1.0,-2.0,-1.0]],np.float32)}\n",
    "\n",
    "def loop_hw2_app(kernelName):\n",
    "    kernelVoid = np.zeros(0)\n",
    "    \n",
    "    xFbuf = mem_manager.cma_array((height,width),np.uint8)\n",
    "    \n",
    "    numframes = 1200\n",
    "\n",
    "    start=time.time()\n",
    "    for _ in range(numframes):\n",
    "        inframe = hdmi_in.readframe()\n",
    "        outframe = hdmi_out.newframe()\n",
    "        xv2.filter2D(inframe, -1, kernel_g.get(kernelName, np.ones((3,3),np.float32)/9.0), xFbuf, (-1,-1), 0.0,borderType=cv2.BORDER_CONSTANT)\n",
    "        xv2.dilate(xFbuf, kernelVoid, dst=outframe, borderType=cv2.BORDER_CONSTANT)\n",
    "        inframe.freebuffer()\n",
    "        hdmi_out.writeframe(outframe)\n",
    "    end=time.time()\n",
    "    print(\"Frames per second:  \" + str(numframes / (end - start)))\n",
    "\n",
    "# Test the different filters one by one\n",
    "loop_hw2_app('laplacian high-pass')\n",
    "loop_hw2_app('gaussian high-pass')\n",
    "loop_hw2_app('average blur')\n",
    "loop_hw2_app('gaussian blur')\n",
    "loop_hw2_app('Sobel Ver')\n",
    "loop_hw2_app('Sobel Hor')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean up hdmi drivers\n",
    "\n",
    "NOTE: This is needed to reset the HDMI drivers in a clean state. If this is not run, subsequent executions of this notebook may show visual artifacts on the HDMI out (usually a shifted output image)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "hdmi_out.close()\n",
    "hdmi_in.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

Hello! That run on the Pynq Z1, but what happen with the Pynq Z2? I found that xv2 doesn't work in the lastest version of Pynq Z2... So, what can I do to run dilate and erosion on this board?

Thanks!