Machine Learning Algorithm for Flappy Bird

Machine Learning Algorithm for Flappy Bird using Neural Network and Genetic Algorithm

Welcome to a complete HTML5 tutorial with demo of a machine learning algorithm for the Flappy Bird video game. The aim of this experiment is programming an artificial intelligence game controller using neural networks and a genetic algorithm.

Hence, we want to create an AI robot which can learn how to optimally play the Flappy Bird game. As a result, our little bird should be able to fly safely through a number of barriers. In the best scenario, it should never die.

When you read all theory behind this project, you can download the source code at the end of this tutorial. All code is written in HTML5 using Phaser framework.  Also, we used Synaptic Neural Network library to implement neural network instead of making it from scratch.

 

The Demo

First of all, let’s start with the live demo to see the algorithm in action:


 

 

The Video Presentation

Additionally to the previous live demo, here you can watch a short video with a simple presentation of the algorithm. So this should be great for those who like to fast forward things!

 

 

What Is Machine Learning Algorithm

According to Arthur Samuel in 1959, machine learning is the science of getting computers to act without being explicitly programmed. Generally speaking, this is a fine tuning process of learning that incrementally improves an initial random system.

Therefore, here is the goal to achieve an artificial intelligence which can find a proper solution from a bad system by fine tuning model parameters. To do that, machine learning algorithm uses a number of different approaches.

Specifically for this project, the main approach of machine learning algorithm (ML) is based on the NeuroEvolution (or neuro-evolution). This form of machine learning uses evolutionary algorithms such as a genetic algorithm (GA) to train artificial neural networks (ANN).

So for this example, we can say ML = GA + ANN.

 

Artificial Neural Network

An artificial neural network is a subset of machine learning algorithm. It is inspired by the structure and functions of biological neural networks. These networks are made out of many neurons which send signals to each other.

Therefore, to create an artificial brain we need to simulate neurons and connect them to form a neural network.

A generic artificial neural network consists of an input layer, one or more hidden layers and an output layers. Each layer has a number of neurons. Input and output neurons are connected directly to an external environment. Finally, hidden neurons are connected between them.

In this project, each unit (bird) has its own neural network used as its AI brain for playing the game. It consists of 3 layers as follows:

  • 1) an input layer with 2 neurons representing what a bird sees:

    • horizontal distance to the closest gap
    • height difference to the closest gap

  • 2) a hidden layer with 6 neurons
  • 3) an output layer with 1 neuron which provides an action as follows:

    • if output > 0.5 then flap else do nothing

The picture below shows the neural network architecture for this example:

Machine Learning Algorithm for Flappy Bird - Neural Network
 

 

Genetic Algorithm

When we talked about machine learning algorithm, we said that a genetic algorithm is used to train and improve neural networks.

Genetic algorithm is a search-based optimization technique inspired by the process of natural selection and genetics. It uses the same combination of selection, crossover and mutation to evolve initial random population.

Here are the main steps of our genetic algorithm implementation:

  • 1. create initial population of 10 units (birds) with random neural networks
  • 2. let all units play the game simultaneously by using their own neural networks
  • 3. for each unit calculate its fitness function to measure its quality
    (for more details see fitness function below)
  • 4. when all units died, evaluate the current population to the next one by using genetic operators
    (for more details see replacement strategy below)
  • 5. go back to the step 2

 

Fitness Function

In addition to the genetic algorithm (step 3), here we go with more details about fitness function – what it is and how to define it.

Since we want to evolve a population by using the best units, we need to define a fitness function.

Generally, the fitness function is the metrics to measure quality of an object. While we have a quality measure for each bird, we can select the fittest units and use them to reproduce the next population.

In this project, we reward a bird equally to its travelled distance. Also, we penalize it by its current distance to the closest gap. So in that way, we are making a difference between birds which travelled the same distance.

To conclude, our fitness function is the difference between the total distance covered by a bird and its current distance to the closest gap.

Machine Learning Algorithm for Flappy Bird - Fitness Function
 

 

Replacement Strategy

In addition to the genetic algorithm (step 4), here are the steps for applying natural evolution on dying population. Basically, the best units survive and their children replace the worst units in this way:

  • 1. sort the units of the current population by their fitness ranking
  • 2. select the top 4 units (winners) and pass them directly on to the next population
  • 3. create 1 offspring as a crossover product of two best winners
  • 4. create 3 offsprings as a crossover products of two random winners
  • 5. create 2 offsprings as a direct copy of two random winners
  • 6. apply random mutations on each offspring to add some variations

 

The Source Code

Finally, here is the link for downloading the source code:

https://github.com/ssusnic/Machine-Learning-Flappy-Bird

 

Machine Learning Algorithm for Flappy Bird – Conclusion

In this tutorial we have successfully implemented AI robot for learning how to play the Flappy Bird game. As a result of several iterations, we can get an almost invincible player. To achieve that goal we have used two approaches of machine learning algorithms: artificial neural networks and genetic algorithm.

As a future work, you can try to change some parameters in code and see what happens. For instance, you can change the number of neurons in hidden layer or number of units in population. Also, you can try to change the fitness function somehow. Furthermore, change some physical parameters such as the distance between barriers, gravity and so on!

Also, try to apply the same idea of evolution to some other games!

 

 


4 thoughts on “Machine Learning Algorithm for Flappy Bird using Neural Network and Genetic Algorithm

  1. Hi

    Nice article. But what I didn’t get is how did you train your ANN using GA? If I am not mistaken, did you use the outcome of GA as training set for ANN?

    1. Each unit starts with its own random ANN. When all units from the current population are killed we are using GA to select the best ones and make crossover to get the offsprings from the best parent. Here, parents and their offsprings are improved ANN in fact. So we are using GA operators to produce a new population with units which have better ANN than the units from the previous population.

  2. Can you explain what is the mutation done between? For example if there are 2 top neural networks each having a set of weights. Now if I a to form a new network from these two, how should I select the weight? Is it average of the weights of the original top 2 performers or say I select the weights of 1st layer from first network, and others from thee 2nd network or something else. Thanks

    1. Inside the genetic algorithm, there is defined mutation rate to always take about 20% offsprings from the new population for mutation. To mutate these units, their connection weights are just a little bit changed using some random value.

Leave a Reply

Your email address will not be published. Required fields are marked *