Road Artery Traffic Light Optimization with Use of the Reinforcement Learning
Abstract
The basic principle of optimal traffic control is the appropriate real-time response to dynamic traffic flow changes. Signal plan efficiency depends on a large number of input parameters. An actuated signal system can adjust very well to traffic conditions, but cannot fully adjust to stochastic traffic volume oscillation. Due to the complexity of the problem analytical methods are not applicable for use in real time, therefore the purpose of this paper is to introduce heuristic method suitable for traffic light optimization in real time. With the evolution of artificial intelligence new possibilities for solving complex problems have been introduced. The goal of this paper is to demonstrate that the use of the Q learning algorithm for traffic lights optimization is suitable. The Q learning algorithm was verified on a road artery with three intersections. For estimation of the effectiveness and efficiency of the proposed algorithm comparison with an actuated signal plan was carried out. The results (average delay per vehicle and the number of vehicles that left road network) show that Q learning algorithm outperforms the actuated signal controllers. The proposed algorithm converges to the minimal delay per vehicle regardless of the stochastic nature of traffic. In this research the impact of the model parameters (learning rate, exploration rate, influence of communication between agents and reward type) on algorithm effectiveness were analysed as well.
References
Anžek M, Kavran Z, Badanjak D. Adaptive Traffic Control as Function of Safety. 12th World Congress on Intelligent Transport Systems and Services. San Francisco, 2005.
Robertson DI. Research on the TRANSYT and SCOOT Methods of Signal Coordination. ITE Journal. 1986;56(1):36-40.
Hunt PB, Roberetson DI. Bretherton RD, Winton RI. A traffic responsive method of coordinating signals. Crowthorne, Berkshire: Transport and Road Research Laboratory; 1981.
Lowrie PR. Scats, Sydney co-ordinated adaptive traffic system: a traffic responsive method of controlling urban traffic. Darlinghurst, NSW Australia: Roads and Traffic Authority NSW; 1990.
Gartner N. Opac: A demand-responsive strategy for traffic signal control. Transportation Research Board; 1983.
Veljanovska K, Bombol K, Maher T. Reinforcement Learning Technique in Multiple Motorway Access Control Strategy Design. PROMET - Traffic&Transportation. 2010;22(2):117-123.
Sen S, Head K. Controlled optimization of phases at an intersection. Transportation science. 1997;31(1):5-17.
Bingham E. Neurofuzzy Traffic Signal Control [Master’s thesis]. Helsinki, Finland: Dept. of Engineering Physics and Mathematics, Helsinki Univ. of Technology; 1998.
Wiering M. Multi-agent reinforcement learning for traffic light control. ICML ‘00 Proceedings of the 17th International Conference on Machine Learning; 2000.
Thorpe TL, Anderson CW. Traffic Light Control Using SARSA with Three State Representations. IBM Corporation; 1996.
Abdulhai B, Pringle R, Karakoulas GJ. Reinforcement Learning for True Adaptive Traffic Signal Control. Journal of Transportation Engineering. 2003;129(3):278–285.
Gregoire PL, Desjardins C, Laumonier J, Chaib-draa B. Urban Traffic Control Based on Learning Agents. IEEE Intelligent Transportation Systems Conference. IEEE; 2007. p. 916-921.
Lu S, Liu X, Dai S. Q-Learning for Adaptive Traffic Signal Control Based on Delay Minimization Strategy. 2008 IEEE International Conference on Networking, Sensing and Control; 2008. p. 687-91.
Arel I, Liu C, Urbanik T, Kohls AG. Reinforcement learning-based multi-agent system for network traffic signal control. IET Intelligent Transport Systems. 2010:4(2):128-135.
Prashanth, L.A., Bhatnagar, S.: Reinforcement Learning With Function Approximation for Traffic Signal Control. IEEE Transactions on Intelligent Transportation Systems. 2011;12(2):412-421.
Sutton RS, Barto A. Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press; 1998.
Abdulhai B, Kattan L. Reinforcement learning: Introduction to theory and potential for transport applications. Canadian Journal of Civil Engineering. 2003;30(6):981–991.
Perez-Uribe A. Introduction to reinforcement learning [Internet]; 1998. Available from: http://ape.iict.ch/teaching/AIGS/AIGS_Labo/Labo5-RL/sarsa.html
Sutton RS. Learning to predict by the methods of temporal differences. Machine Learning. 1988;3:9–44
Wiering M, van Veenen J, Vreeken J, Koopman A. Intelligent traffic light control. Technical Report. Utrecht University, Institute of Information and Computing Sciences; 2004. 31 p.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
