Machine Learning: Doodle Recognition with Convolutional Neural Network - Part 1

Machine Learning: Doodle Recognition with Convolutional Neural Network, Part 1 – Project Setup

Part 1 of the step by step video tutorial series on making a game like “Quick, Draw!“. It is an online game that challenges players to draw a doodle and then artificial intelligence guesses what the drawings represent.

This tutorial series shows the whole process of developing this game.

Using Phaser 2 framework and Tensorflow.js library, we will create a complete HTML5 Doodle Predictor game that runs directly in the browser and recognizes doodles.

To classify drawings, we will implement an Artificial Intelligence (AI) based on Machine Learning (ML) and Convolutional Neural Network (CNN).

To train the model, we will use a small subset of the Quick Draw Dataset.


1. Video Trailer

To get a quick insight into the game we are going to make, you can check out the video trailer here:



2. Tutorial Topics

This tutorial series is divided into the following 8 parts:

  • Part 1: project setup
  • Part 2: getting and preparing training and test data
  • Part 3: building a convolutional neural network model
  • Part 4: training, testing and evaluating the model
  • Part 5: predicting a batch of samples
  • Part 6: implementing the Painter class to allow users to draw doodles
  • Part 7: recognizing doodles by using the trained model
  • Part 8: adding more doodle categories

Each part includes a video and fully-commented source code so you can follow game development step by step.

The complete source code is available for free download on Github:

The picture below shows a screenshot of the completed Doodle Predictor game:

Screenshot of the completed Doodle Predictor game



3. Libraries

Let’s get started with developing the game. Our first task is to set up the whole project.

Using the Phaser framework we will create a new Phaser Game object and build the basic game structure with the main loop.

Later, we will use Tensorflow.js library to implement machine learning.

So we need the following libraries:

  • Phaser.js
    – a free 2D game framework for making HTML5 games for desktop and mobile.
    – note: the game uses Phaser 2, because at the time of development that was the current stable version!
  • Tensorflow.js
    – a JavaScript library for training and deploying Machine Learning models in the browser.


4. Main Program (main.js)

The main program is consisted of two major sections:

  1. Game Setup – used to create a new Phaser Game object.
  2. Main State – used to implement the game logic.


4.1. Game Setup

In Phaser, the first thing we need to do is to set up the game as follows:

  1. on window load event create a new game object as an instance of the Phaser.Game
  2. create all states and attach them to the previously created game object
  3. start the default state


4.2. Main State

In Phaser, all the action occurs around Phaser States.

To keep things simple, we will add only the Main State and implement the following three Phaser.State methods:

  • preload() – the first method called to load all assets.
  • create() – the method called only once immediately after all assets are loaded in memory; used to create all game objects such as sprites, graphics, text fields and so on.
  • update() – after all objects are created, this method will be called on every game tick representing the main game loop. So the entire game logic goes here!

To keep the source code clearer and easier to read when it later becomes bigger, it’s fine to break the Main State into smaller parts using a simple finite state machine.

So we can define the following sub-states inside the Main State:

Sub-state Constant Description
MODE_INIT = 1 to initialize the game
MODE_OPEN_FILE = 2 to open a dataset file and start loading it
MODE_LOAD_FILE = 3 to wait until a dataset file is loaded
MODE_START_TRAIN = 4 to start training the model
MODE_DO_TRAIN = 5 to wait until the model is trained
MODE_START_PREDICT = 6 to start predicting the samples
MODE_DO_PREDICT = 7 to wait until all samples are predicted
MODE_DRAW = 8 to draw a doodle and predict it

4.3. Pseudo Code of the Main Program

Setup Game on Window Load {
	create a new game using Phaser.Game
	add MainState to the game 
	start MainState

MainState {
		load all assets
		create all objects
		on MODE_INIT: 
			initialize the game
			start loading dataset
			wait to load dataset
			start training model
			wait to train model
			start predicting the samples
			wait to predict all samples
			draw a doodle and predict it


5. User Interface (ui.js)

To allow users an interaction with the game we need to create an User Interface (UI) with buttons, messages, head-up display and so on.

For now, we will only create a new UI class with the following objects:

  1. “Play More Game” button which opens this Web Site
  2. Text for showing messages in the status bar

During game development, we will add all other UI elements.


6. Running Doodle Predictor Part 1

After completing the first part, the game looks pretty miserable, but this is just beginning!



7. Source Code

The complete source code for Doodle Predictor (Part 1) is available on Github:

In the next part, we will see how to get and prepare data. So stay tuned!


Leave a Reply

Your email address will not be published. Required fields are marked *