Welcome to a complete HTML5 tutorial with demo of a machine learning algorithm for the Flappy Bird video game. The aim of this experiment is programming an artificial intelligence game controller using neural networks and a genetic algorithm.
Hence, we want to create an AI robot which can learn how to optimally play the Flappy Bird game. As a result, our little bird should be able to fly safely through a number of barriers. In the best scenario, it should never die.
When you read all theory behind this project, you can download the source code at the end of this tutorial. All code is written in HTML5 using Phaser framework. Also, we used Synaptic Neural Network library to implement neural network instead of making it from scratch.
First of all, let’s start with the live demo to see the algorithm in action:
The Video Presentation
Additionally to the previous live demo, here you can watch a short video with a simple presentation of the algorithm. So this should be great for those who like to fast forward things!
What Is Machine Learning Algorithm
According to Arthur Samuel in 1959, machine learning is the science of getting computers to act without being explicitly programmed. Generally speaking, this is a fine tuning process of learning that incrementally improves an initial random system.
Therefore, here is the goal to achieve an artificial intelligence which can find a proper solution from a bad system by fine tuning model parameters. To do that, machine learning algorithm uses a number of different approaches.
Specifically for this project, the main approach of machine learning algorithm (ML) is based on the NeuroEvolution (or neuro-evolution). This form of machine learning uses evolutionary algorithms such as a genetic algorithm (GA) to train artificial neural networks (ANN).
So for this example, we can say ML = GA + ANN.
Artificial Neural Network
An artificial neural network is a subset of machine learning algorithm. It is inspired by the structure and functions of biological neural networks. These networks are made out of many neurons which send signals to each other.
Therefore, to create an artificial brain we need to simulate neurons and connect them to form a neural network.
A generic artificial neural network consists of an input layer, one or more hidden layers and an output layers. Each layer has a number of neurons. Input and output neurons are connected directly to an external environment. Finally, hidden neurons are connected between them.
In this project, each unit (bird) has its own neural network used as its AI brain for playing the game. It consists of 3 layers as follows:
- 1) an input layer with 2 neurons representing what a bird sees:
- horizontal distance to the closest gap
- height difference to the closest gap
- 2) a hidden layer with 6 neurons
- 3) an output layer with 1 neuron which provides an action as follows:
- if output > 0.5 then flap else do nothing
The picture below shows the neural network architecture for this example:
When we talked about machine learning algorithm, we said that a genetic algorithm is used to train and improve neural networks.
Genetic algorithm is a search-based optimization technique inspired by the process of natural selection and genetics. It uses the same combination of selection, crossover and mutation to evolve initial random population.
Here are the main steps of our genetic algorithm implementation:
- 1. create initial population of 10 units (birds) with random neural networks
- 2. let all units play the game simultaneously by using their own neural networks
- 3. for each unit calculate its fitness function to measure its quality
(for more details see fitness function below)
- 4. when all units died, evaluate the current population to the next one by using genetic operators
(for more details see replacement strategy below)
- 5. go back to the step 2
In addition to the genetic algorithm (step 3), here we go with more details about fitness function – what it is and how to define it.
Since we want to evolve a population by using the best units, we need to define a fitness function.
Generally, the fitness function is the metrics to measure quality of an object. While we have a quality measure for each bird, we can select the fittest units and use them to reproduce the next population.
In this project, we reward a bird equally to its travelled distance. Also, we penalize it by its current distance to the closest gap. So in that way, we are making a difference between birds which travelled the same distance.
To conclude, our fitness function is the difference between the total distance covered by a bird and its current distance to the closest gap.
In addition to the genetic algorithm (step 4), here are the steps for applying natural evolution on dying population. Basically, the best units survive and their children replace the worst units in this way:
- 1. sort the units of the current population by their fitness ranking
- 2. select the top 4 units (winners) and pass them directly on to the next population
- 3. create 1 offspring as a crossover product of two best winners
- 4. create 3 offsprings as a crossover products of two random winners
- 5. create 2 offsprings as a direct copy of two random winners
- 6. apply random mutations on each offspring to add some variations
The Source Code
Finally, here is the link for downloading the source code:
Machine Learning Algorithm for Flappy Bird – Conclusion
In this tutorial we have successfully implemented AI robot for learning how to play the Flappy Bird game. As a result of several iterations, we can get an almost invincible player. To achieve that goal we have used two approaches of machine learning algorithms: artificial neural networks and genetic algorithm.
As a future work, you can try to change some parameters in code and see what happens. For instance, you can change the number of neurons in hidden layer or number of units in population. Also, you can try to change the fitness function somehow. Furthermore, change some physical parameters such as the distance between barriers, gravity and so on!
Also, try to apply the same idea of evolution to some other games!