Machine Learning: Doodle Recognition with Convolutional Neural Network - Part 1

Machine Learning: Doodle Recognition with Convolutional Neural Network, Part 1 – Introduction

Part 1 of the step by step video tutorial series on making a game like “Quick, Draw!“. It is an online game that challenges players to draw a doodle and then artificial intelligence guesses what the drawings represent.

To show the process of developing this game, we will create a complete HTML5 Doodle Predictor game that runs directly in the browser using Phaser 2 framework. To implement Machine Learning for recognizing doodles, we will build a Convolutional Neural Network (CNN) using Tensorflow library and train it on the small subset of the Quick Draw Dataset.

The first part gives an overview of the whole project providing a list of topics and a short video trailer. Regarding source code, we will start easy with developing the basic game structure.


1. Video Trailer

To get a quick insight into the game we are going to make, watch the following video:



2. Tutorial Topics

This project is divided into 8 parts. Each part includes a video and fully-commented source code so you can follow game development step by step. Here is a list of all topics:

  • Part 1: introduction – creating a new Phaser game
  • Part 2: getting and preparing training and test data
  • Part 3: building a convolutional neural network model
  • Part 4: training, testing and evaluating the model
  • Part 5: predicting a batch of samples
  • Part 6: implementing the Painter class to allow users to draw doodles
  • Part 7: recognizing doodles by using the trained model
  • Part 8: adding more doodle categories

All source code will be available for free download on Github:

The picture below shows a screenshot of the completed Doodle Predictor game:

Screenshot of the completed Doodle Predictor game


3. Libraries

Now, let’s get started with developing the game. Here are the links to the libraries used in this project:

  • Phaser.js – a free 2D game framework for making HTML5 games for desktop and mobile.
  • Tensorflow.js – a JavaScript library for training and deploying Machine Learning models in the browser.


4. Main Program (main.js)

In the first part of this tutorial series, we just create a new Phaser Game object and build the basic game structure. The main program is consisted of two major sections:

  • Section 1: Game Setup – used to create a new Phaser Game object.
  • Section 2: Main State – used to implement the game logic.

4.1. Game Setup

In Phaser, the first thing we need to do is to setup our game following these steps:

  1. on window load event create a new game object as an instance of the Phaser.Game
  2. add all states to the previously created game object
  3. start the default state

4.2. Main State

In Phaser, all the action occurs around Phaser States. To keep things simple, we will add only the Main State and implement the following three Phaser.State methods:

  • preload() – the first method called to load all assets.
  • create() – the method called only once immediately after all assets are loaded in memory; used to create all game objects such as sprites, graphics, text fields and so on.
  • update() – after all objects are created, this method will be called on every game tick representing the main game loop. So the entire game logic goes here!

To keep the source code clearer and easier to read when it later becomes bigger, it’s good to break the Main State into smaller parts using a simple finite state machine. So we can define the following sub-states inside the Main State:

Sub-state Constant Description
MODE_INIT = 1 to initialize the game
MODE_OPEN_FILE = 2 to open a dataset file and start loading it
MODE_LOAD_FILE = 3 to wait until a dataset file is loaded
MODE_START_TRAIN = 4 to start training the model
MODE_DO_TRAIN = 5 to wait until the model is trained
MODE_START_PREDICT = 6 to start predicting the samples
MODE_DO_PREDICT = 7 to wait until all samples are predicted
MODE_DRAW = 8 to draw a doodle and predict it

4.3. Pseudo Code of the Main Program

Setup Game on Window Load {
	create a new game using Phaser.Game
	add MainState to the game 
	start MainState

MainState {
		load all assets
		create all objects
		on MODE_INIT: 
			initialize the game
			start loading dataset
			wait to load dataset
			start training model
			wait to train model
			start predicting the samples
			wait to predict all samples
			draw a doodle and predict it


5. User Interface (ui.js)

To allow users an interaction with the game we need to create an User Interface (UI) with buttons, messages, head-up display and so on. For now, we will only create a new UI class with the following objects:

  1. “Play More Game” button which opens this Web Site
  2. Text for showing messages in the status bar

During game development, we will add all other UI elements.


6. Running Doodle Predictor

After completing the first part, the game looks pretty miserable, but this is just beginning!



7. Source Code

The complete source code for Doodle Predictor – Part 1 is available on Github:

In the next part, we will see how to get and prepare data. So stay tuned!



Leave a Reply

Your email address will not be published. Required fields are marked *