Machine Learning Algorithm for Flappy Bird

Machine Learning Algorithm for Flappy Bird using Neural Network and Genetic Algorithm

Welcome to a complete HTML5 tutorial with demo of a machine learning algorithm for the Flappy Bird video game. The aim of this experiment is programming an artificial intelligence game controller using neural networks and a genetic algorithm.

Hence, we want to create an AI robot which can learn how to optimally play the Flappy Bird game. As a result, our little bird should be able to fly safely through a number of barriers. In the best scenario, it should never die.

When you read all theory behind this project, you can download the source code at the end of this tutorial. All code is written in HTML5 using Phaser framework.  Also, we used Synaptic Neural Network library to implement neural network instead of making it from scratch.

 

The Demo

First of all, let’s start with the live demo to see the algorithm in action:


 

 

The Video Presentation

Additionally to the previous live demo, here you can watch a short video with a simple presentation of the algorithm. So this should be great for those who like to fast forward things!

 

 

What Is Machine Learning Algorithm

According to Arthur Samuel in 1959, machine learning is the science of getting computers to act without being explicitly programmed. Generally speaking, this is a fine tuning process of learning that incrementally improves an initial random system.

Therefore, here is the goal to achieve an artificial intelligence which can find a proper solution from a bad system by fine tuning model parameters. To do that, machine learning algorithm uses a number of different approaches.

Specifically for this project, the main approach of machine learning algorithm (ML) is based on the NeuroEvolution (or neuro-evolution). This form of machine learning uses evolutionary algorithms such as a genetic algorithm (GA) to train artificial neural networks (ANN).

So for this example, we can say ML = GA + ANN.

 

Artificial Neural Network

An artificial neural network is a subset of machine learning algorithm. It is inspired by the structure and functions of biological neural networks. These networks are made out of many neurons which send signals to each other.

Therefore, to create an artificial brain we need to simulate neurons and connect them to form a neural network.

A generic artificial neural network consists of an input layer, one or more hidden layers and an output layers. Each layer has a number of neurons. Input and output neurons are connected directly to an external environment. Finally, hidden neurons are connected between them.

In this project, each unit (bird) has its own neural network used as its AI brain for playing the game. It consists of 3 layers as follows:

  • 1) an input layer with 2 neurons representing what a bird sees:

    • horizontal distance to the closest gap
    • height difference to the closest gap

  • 2) a hidden layer with 6 neurons
  • 3) an output layer with 1 neuron which provides an action as follows:

    • if output > 0.5 then flap else do nothing

The picture below shows the neural network architecture for this example:

Machine Learning Algorithm for Flappy Bird - Neural Network
 

 

Genetic Algorithm

When we talked about machine learning algorithm, we said that a genetic algorithm is used to train and improve neural networks.

Genetic algorithm is a search-based optimization technique inspired by the process of natural selection and genetics. It uses the same combination of selection, crossover and mutation to evolve initial random population.

Here are the main steps of our genetic algorithm implementation:

  • 1. create initial population of 10 units (birds) with random neural networks
  • 2. let all units play the game simultaneously by using their own neural networks
  • 3. for each unit calculate its fitness function to measure its quality
    (for more details see fitness function below)
  • 4. when all units died, evaluate the current population to the next one by using genetic operators
    (for more details see replacement strategy below)
  • 5. go back to the step 2

 

Fitness Function

In addition to the genetic algorithm (step 3), here we go with more details about fitness function – what it is and how to define it.

Since we want to evolve a population by using the best units, we need to define a fitness function.

Generally, the fitness function is the metrics to measure quality of an object. While we have a quality measure for each bird, we can select the fittest units and use them to reproduce the next population.

In this project, we reward a bird equally to its travelled distance. Also, we penalize it by its current distance to the closest gap. So in that way, we are making a difference between birds which travelled the same distance.

To conclude, our fitness function is the difference between the total distance covered by a bird and its current distance to the closest gap.

Machine Learning Algorithm for Flappy Bird - Fitness Function
 

 

Replacement Strategy

In addition to the genetic algorithm (step 4), here are the steps for applying natural evolution on dying population. Basically, the best units survive and their children replace the worst units in this way:

  • 1. sort the units of the current population by their fitness ranking
  • 2. select the top 4 units (winners) and pass them directly on to the next population
  • 3. create 1 offspring as a crossover product of two best winners
  • 4. create 3 offsprings as a crossover products of two random winners
  • 5. create 2 offsprings as a direct copy of two random winners
  • 6. apply random mutations on each offspring to add some variations

 

The Source Code

Finally, here is the link for downloading the source code:

https://github.com/ssusnic/Machine-Learning-Flappy-Bird

 

Machine Learning Algorithm for Flappy Bird – Conclusion

In this tutorial we have successfully implemented AI robot for learning how to play the Flappy Bird game. As a result of several iterations, we can get an almost invincible player. To achieve that goal we have used two approaches of machine learning algorithms: artificial neural networks and genetic algorithm.

As a future work, you can try to change some parameters in code and see what happens. For instance, you can change the number of neurons in hidden layer or number of units in population. Also, you can try to change the fitness function somehow. Furthermore, change some physical parameters such as the distance between barriers, gravity and so on!

Also, try to apply the same idea of evolution to some other games!

 

 


33 thoughts on “Machine Learning Algorithm for Flappy Bird using Neural Network and Genetic Algorithm

  1. Hi

    Nice article. But what I didn’t get is how did you train your ANN using GA? If I am not mistaken, did you use the outcome of GA as training set for ANN?

    1. Each unit starts with its own random ANN. When all units from the current population are killed we are using GA to select the best ones and make crossover to get the offsprings from the best parent. Here, parents and their offsprings are improved ANN in fact. So we are using GA operators to produce a new population with units which have better ANN than the units from the previous population.

      1. So, you have a bunch of bird with their own neural networks. My question is, what exactly does your genetic algorithm evolve – weights? number of hidden nodes?

  2. Can you explain what is the mutation done between? For example if there are 2 top neural networks each having a set of weights. Now if I a to form a new network from these two, how should I select the weight? Is it average of the weights of the original top 2 performers or say I select the weights of 1st layer from first network, and others from thee 2nd network or something else. Thanks

    1. Inside the genetic algorithm, there is defined mutation rate to always take about 20% offsprings from the new population for mutation. To mutate these units, their connection weights are just a little bit changed using some random value.

  3. I love what you have shown us , the application of machine learning in playing games.

    But I wonder how do you get the needed data from the game, like the distance or game states, and feed them into your programme?

    Actually can you also tell me other than this particular game, what are the usual ways for people to get data from a flash game or video game or html game? Is it from screen capture followed by examination into the pixels? or there are kind of backdoor APIs to get the data?

    Thanks!

    1. No, there is nothing like getting data from screen capture. The neural networks need only two input parameters: current distance to the closest gap (gap_position_X – bird_position_X) and height difference to the closest gap (gap_position_Y – bird_position_Y).

    1. Here is a quick guide (probably you will need to do some more configuration steps after installing a web-server):

      1. Install some web-server on your machine (Xampp or Wamp)
      2. copy and paste all files in a web-server folder with “permits public access” (it’s \htdocs folder on Xampp so you can put all files in Xampp\htdocs\flappy folder)
      3. Start web-server (click on Start Apache button in Xampp Control Panel)
      4. open a web browser and enter the link http://localhost/flappy to run the application

      Or you can try without a web server by copy&paste all files in some folder on your disk and then open a web browser and just drag and drop index.html in it (I found Firefox is the best choice if you are using this method, it seems other browsers don’t want to start html5 apps in that way)

    1. Well, it’s a bit of a mix between art and science. You can use less or more than 6 neurons, it is your choice. I used 6 neurons because I found it is an optimal number of hidden neurons for this example – with less neurons the system needs more time to produce a good population, with more neurons the system needs to do more calculations in each iteration.

    1. The crossover algorithm swaps ‘bias’ information between both parents as follows:
      1. get a random crossover cutting point
      2. copy from the first parent its left side to the crossover point
      3. copy from the second parent its right side after the crossover point

    1. Just download all files from Github and put them to a local webserver on your machine (for instance you can use Xampp or Wamp server). The project contains index.html file and a number of javascript files including the source code and frameworks. To run the application start the webserver and then open a proper link pointing to the index.html in your web browser.

  4. How exactly does the input get changed by the connections and become the hidden layer, and how do the hidden layers become the output? What operations are used, and in what way? Thank you, and great article.

    1. In this example, the neural network is implemented by using Synaptic Neural Network library. So I didn’t make this part from the scratch but used a complete neural network as a black box. I am just sending inputs to it and reading its output.

  5. I muѕt thank you for the efforts you have put in writing this site.
    I really hope to check out the same high-grade content by you later on as well.
    Іn fact, your creativе writing abilities has motivated me to get my very own website now ;
    )

  6. Today I found you on youtube, and immediately became a huge fan of you, and this became my new favorite website.
    Open source Machine learning code written with javascript it’s like a dream come true.

  7. Hello , why the output need 0.5 – if (outputs[0] > 0.5) bird.flap(); , and how I get this value?

    Thanks

    Sorry my english no is very well

    1. There are two possible actions: flap yes or flap no. Because we have only one output from the neural network which can be in the range from 0.0 to 1.0 we must define the bird will flap only if output>0.5.

      1. I pass as input the x and y on the activation, how does he know I need to fly or not ?, why when searching for deltaX or deltaY have to multiply by 200? var named SCALE_FACTOR.

        1. 1. The bird knows when to fly or not because its neural network is adjusted (by using genetic algorithms from generation to generation) to produce the right output action for the X and Y inputs.
          2. Experimenting with the Synaptic Neural Network library, I found that if I multiply inputs by factor 200 then I get something better output results. But probably, it’s not necessary.

  8. Fun example, really enjoyed it
    I added
    if(bird.fitness_curr > 3000){
    this.state = this.STATE_GAMEOVER;
    }
    to the bird iteration in STATE_PLAY to see how long it would take for only “winning” birds to dominate the population
    So typically it takes 12 GEN to create a “winning” bird
    It took for me 48 gens to get 9 birds “winning”

    Which is interesting, how much mutation do you need to speed up evolving a winning bird vs. mutation that prevents the entire population from becoming winners? Fun experiment for thought and coding

    1. Nice to hear that you are interested to play with this. I’m not sure if I know the right answer on your question. I think it’s pretty hard to say exactly how much mutation is needed to speed up evolving a winning bird.

  9. Hi Srdjan, this project is really cool! It is very inspiring… and nice to look at! 🙂
    Just a question about the fitness function; I think that the image that explain that could be a little bit misleading. The “distance to the closest gap” should be painted as an inclined segment between the bird and the “target point” (I got that from the sources). The target point should be in the middle between the two trees. Right?

    Anyway, thank you for sharing this wonderful project!

    1. Look at the genetic.js source code – crossOver() and mutation() funtions. The neurons bias information and connections weights are modified there to form a new generation.

  10. Hi! First of all, congratulations for you awesome project!

    I would like to know which functions did you use at each neurons and why… Was it logistic function?

    Thanks in advance!

Leave a Reply

Your email address will not be published. Required fields are marked *