Welcome to a complete HTML5 tutorial with demo of a machine learning algorithm for the Flappy Bird video game. The aim of this experiment is programming an artificial intelligence game controller using neural networks and a genetic algorithm.
Hence, we want to create an AI robot which can learn how to optimally play the Flappy Bird game. As a result, our little bird should be able to fly safely through a number of barriers. In the best scenario, it should never die.
When you read all theory behind this project, you can download the source code at the end of this tutorial. All code is written in HTML5 using Phaser framework. Also, we used Synaptic Neural Network library to implement neural network instead of making it from scratch.
The Demo
First of all, let’s start with the live demo to see the algorithm in action:
The Video Presentation
Additionally to the previous live demo, here you can watch a short video with a simple presentation of the algorithm. So this should be great for those who like to fast forward things!
What Is Machine Learning Algorithm
According to Arthur Samuel in 1959, machine learning is the science of getting computers to act without being explicitly programmed. Generally speaking, this is a fine tuning process of learning that incrementally improves an initial random system.
Therefore, here is the goal to achieve an artificial intelligence which can find a proper solution from a bad system by fine tuning model parameters. To do that, machine learning algorithm uses a number of different approaches.
Specifically for this project, the main approach of machine learning algorithm (ML) is based on the NeuroEvolution (or neuro-evolution). This form of machine learning uses evolutionary algorithms such as a genetic algorithm (GA) to train artificial neural networks (ANN).
So for this example, we can say ML = GA + ANN.
Artificial Neural Network
An artificial neural network is a subset of machine learning algorithm. It is inspired by the structure and functions of biological neural networks. These networks are made out of many neurons which send signals to each other.
Therefore, to create an artificial brain we need to simulate neurons and connect them to form a neural network.
A generic artificial neural network consists of an input layer, one or more hidden layers and an output layers. Each layer has a number of neurons. Input and output neurons are connected directly to an external environment. Finally, hidden neurons are connected between them.
In this project, each unit (bird) has its own neural network used as its AI brain for playing the game. It consists of 3 layers as follows:
- 1) an input layer with 2 neurons representing what a bird sees:
- horizontal distance to the closest gap
- height difference to the closest gap
- 2) a hidden layer with 6 neurons
- 3) an output layer with 1 neuron which provides an action as follows:
- if output > 0.5 then flap else do nothing
The picture below shows the neural network architecture for this example:
Genetic Algorithm
When we talked about machine learning algorithm, we said that a genetic algorithm is used to train and improve neural networks.
Genetic algorithm is a search-based optimization technique inspired by the process of natural selection and genetics. It uses the same combination of selection, crossover and mutation to evolve initial random population.
Here are the main steps of our genetic algorithm implementation:
- 1. create initial population of 10 units (birds) with random neural networks
- 2. let all units play the game simultaneously by using their own neural networks
- 3. for each unit calculate its fitness function to measure its quality
(for more details see fitness function below) - 4. when all units died, evaluate the current population to the next one by using genetic operators
(for more details see replacement strategy below) - 5. go back to the step 2
Fitness Function
In addition to the genetic algorithm (step 3), here we go with more details about fitness function – what it is and how to define it.
Since we want to evolve a population by using the best units, we need to define a fitness function.
Generally, the fitness function is the metrics to measure quality of an object. While we have a quality measure for each bird, we can select the fittest units and use them to reproduce the next population.
In this project, we reward a bird equally to its travelled distance. Also, we penalize it by its current distance to the closest gap. So in that way, we are making a difference between birds which travelled the same distance.
To conclude, our fitness function is the difference between the total distance covered by a bird and its current distance to the closest gap.
Replacement Strategy
In addition to the genetic algorithm (step 4), here are the steps for applying natural evolution on dying population. Basically, the best units survive and their children replace the worst units in this way:
- 1. sort the units of the current population by their fitness ranking
- 2. select the top 4 units (winners) and pass them directly on to the next population
- 3. create 1 offspring as a crossover product of two best winners
- 4. create 3 offsprings as a crossover products of two random winners
- 5. create 2 offsprings as a direct copy of two random winners
- 6. apply random mutations on each offspring to add some variations
The Source Code
Finally, here is the link for downloading the source code:
https://github.com/ssusnic/Machine-Learning-Flappy-Bird
Machine Learning Algorithm for Flappy Bird – Conclusion
In this tutorial we have successfully implemented AI robot for learning how to play the Flappy Bird game. As a result of several iterations, we can get an almost invincible player. To achieve that goal we have used two approaches of machine learning algorithms: artificial neural networks and genetic algorithm.
As a future work, you can try to change some parameters in code and see what happens. For instance, you can change the number of neurons in hidden layer or number of units in population. Also, you can try to change the fitness function somehow. Furthermore, change some physical parameters such as the distance between barriers, gravity and so on!
Also, try to apply the same idea of evolution to some other games!
Hi
Nice article. But what I didn’t get is how did you train your ANN using GA? If I am not mistaken, did you use the outcome of GA as training set for ANN?
Each unit starts with its own random ANN. When all units from the current population are killed we are using GA to select the best ones and make crossover to get the offsprings from the best parent. Here, parents and their offsprings are improved ANN in fact. So we are using GA operators to produce a new population with units which have better ANN than the units from the previous population.
So, you have a bunch of bird with their own neural networks. My question is, what exactly does your genetic algorithm evolve – weights? number of hidden nodes?
It evolves weights of connections and bias information.
can you explain how the game procedures works, i have a research.
Hello, I need a Forex neural network, can you help?
Can you explain what is the mutation done between? For example if there are 2 top neural networks each having a set of weights. Now if I a to form a new network from these two, how should I select the weight? Is it average of the weights of the original top 2 performers or say I select the weights of 1st layer from first network, and others from thee 2nd network or something else. Thanks
Inside the genetic algorithm, there is defined mutation rate to always take about 20% offsprings from the new population for mutation. To mutate these units, their connection weights are just a little bit changed using some random value.
I love what you have shown us , the application of machine learning in playing games.
But I wonder how do you get the needed data from the game, like the distance or game states, and feed them into your programme?
Actually can you also tell me other than this particular game, what are the usual ways for people to get data from a flash game or video game or html game? Is it from screen capture followed by examination into the pixels? or there are kind of backdoor APIs to get the data?
Thanks!
No, there is nothing like getting data from screen capture. The neural networks need only two input parameters: current distance to the closest gap (gap_position_X – bird_position_X) and height difference to the closest gap (gap_position_Y – bird_position_Y).
How do I get the program running?
Here is a quick guide (probably you will need to do some more configuration steps after installing a web-server):
1. Install some web-server on your machine (Xampp or Wamp)
2. copy and paste all files in a web-server folder with “permits public access” (it’s \htdocs folder on Xampp so you can put all files in Xampp\htdocs\flappy folder)
3. Start web-server (click on Start Apache button in Xampp Control Panel)
4. open a web browser and enter the link http://localhost/flappy to run the application
Or you can try without a web server by copy&paste all files in some folder on your disk and then open a web browser and just drag and drop index.html in it (I found Firefox is the best choice if you are using this method, it seems other browsers don’t want to start html5 apps in that way)
Hi, can you please tell me why you used 6 hidden neurons, why not 4, why not 8? what’s special about 6?
Well, it’s a bit of a mix between art and science. You can use less or more than 6 neurons, it is your choice. I used 6 neurons because I found it is an optimal number of hidden neurons for this example – with less neurons the system needs more time to produce a good population, with more neurons the system needs to do more calculations in each iteration.
What’s the crossover strategy?
The crossover algorithm swaps ‘bias’ information between both parents as follows:
1. get a random crossover cutting point
2. copy from the first parent its left side to the crossover point
3. copy from the second parent its right side after the crossover point
Any specific reason why you swap only the bias weights and not all weights?
Well, during the development, I noticed it works fine with swapping only the bias weights so I used only that.
can you make a video doing this in real time? please! 🙂 to learn even how to install the framework. Thanks!
Just download all files from Github and put them to a local webserver on your machine (for instance you can use Xampp or Wamp server). The project contains index.html file and a number of javascript files including the source code and frameworks. To run the application start the webserver and then open a proper link pointing to the index.html in your web browser.
How exactly does the input get changed by the connections and become the hidden layer, and how do the hidden layers become the output? What operations are used, and in what way? Thank you, and great article.
In this example, the neural network is implemented by using Synaptic Neural Network library. So I didn’t make this part from the scratch but used a complete neural network as a black box. I am just sending inputs to it and reading its output.
I muѕt thank you for the efforts you have put in writing this site.
I really hope to check out the same high-grade content by you later on as well.
Іn fact, your creativе writing abilities has motivated me to get my very own website now ;
)
Today I found you on youtube, and immediately became a huge fan of you, and this became my new favorite website.
Open source Machine learning code written with javascript it’s like a dream come true.
Hello , why the output need 0.5 – if (outputs[0] > 0.5) bird.flap(); , and how I get this value?
Thanks
Sorry my english no is very well
There are two possible actions: flap yes or flap no. Because we have only one output from the neural network which can be in the range from 0.0 to 1.0 we must define the bird will flap only if output>0.5.
I pass as input the x and y on the activation, how does he know I need to fly or not ?, why when searching for deltaX or deltaY have to multiply by 200? var named SCALE_FACTOR.
1. The bird knows when to fly or not because its neural network is adjusted (by using genetic algorithms from generation to generation) to produce the right output action for the X and Y inputs.
2. Experimenting with the Synaptic Neural Network library, I found that if I multiply inputs by factor 200 then I get something better output results. But probably, it’s not necessary.
Fun example, really enjoyed it
I added
if(bird.fitness_curr > 3000){
this.state = this.STATE_GAMEOVER;
}
to the bird iteration in STATE_PLAY to see how long it would take for only “winning” birds to dominate the population
So typically it takes 12 GEN to create a “winning” bird
It took for me 48 gens to get 9 birds “winning”
Which is interesting, how much mutation do you need to speed up evolving a winning bird vs. mutation that prevents the entire population from becoming winners? Fun experiment for thought and coding
Nice to hear that you are interested to play with this. I’m not sure if I know the right answer on your question. I think it’s pretty hard to say exactly how much mutation is needed to speed up evolving a winning bird.
Hi Srdjan, this project is really cool! It is very inspiring… and nice to look at! 🙂
Just a question about the fitness function; I think that the image that explain that could be a little bit misleading. The “distance to the closest gap” should be painted as an inclined segment between the bird and the “target point” (I got that from the sources). The target point should be in the middle between the two trees. Right?
Anyway, thank you for sharing this wonderful project!
Yes, you are right. I simplified things on that image a little bit, just for better understanding.
what is the genome representation that you used for this problem?
Thank you ?
Look at the genetic.js source code – crossOver() and mutation() funtions. The neurons bias information and connections weights are modified there to form a new generation.
Hi! First of all, congratulations for you awesome project!
I would like to know which functions did you use at each neurons and why… Was it logistic function?
Thanks in advance!
An impressive share! I’ve just forwarded this onto a friend who has been doing a
little homework on this. And he in fact ordered me dinner because I stumbled upon it for
him… lol. So allow me to reword this….
Thanks for the meal!! But yeah, thanx for spending some time to discuss
this subject here on your internet site.
Hi. Is it possible to run one bird at time? Or it always have to be 10 at the same time?
I’m afraid there must be at least some number of birds in population to be able to make genetic operations with them.
Hi, really interesting and fun.
Quick question does the trees are randomly created? Best
Yes of course!
Hi, Great project, congratulations. Could you also show how the genetic algorithm uses neural networks to create this machine learning in pseudocode ?, I’d like to learn how to implement this in other projects, but I’m not understanding the right concept. thank you.
I think the pseudocode is already there in tutorial and you have the complete source code on Github to learn how it is implemented. Just to know that neural networks are not implemented by me, but by using Synaptic library!
Nice article
its really cool.
ihave a question, since human see about 3 trees foreword, is Iit possible and more efficient to add more inputs parameters for more trees into ANN? A real boy to play flap bird and he can see more trees to adjust tictac by human brian.
sorry for my poor english.
another question, would pls give some commets.
if there is a parameter to adjust gap between trees from x or y two dimension. then we change these parameters. for a already always winner flap bird, does it will failed for different situation?
thansk
I got this web site from my buddy who told
me about this web page and at the moment this time
I am browsing this web page and reading very informative content at this place.
Hi! Awesome project! Congratulations. I just have one question: I was looking around your code (Specifically the GA script) and something i doesn´t got clear is; when/where are you using the fitness function, I know that you explained that is the subtraction of total distance traveled minus distance to the closest gap; but in code, when/where is reflected. That´s my question. Thank you
Exactly the kind of article i needed !
Simple but complete.
Thanks !
Hello，this is a great project，can i use python to instead javascrip for this project?
I really want to communicate with you, but I do not know what channels can be used?
I come from China.
The genetic could be saved? And used later?
can someone tell me how to get this project running !?
It’s awesome in favor of me to have a site,
which is good designed for my experience. thanks admin
Can we save the generation output?
So, every time we run this project it will start from the last completed generation.
For example, we calculate till generation 11 till the first run and after closing this project and after rerun the project it will start with generation 12?
Unfortunately, there is no that feature.
Thanks for sharing, this is a fantastic blog post. Really thank you! Really Great. Simona Duff Natika