Road Artery Traffic Light Optimization with Use of the Reinforcement Learning

  • Rok Marsetič University of Ljubljana, Faculty of Civil and Geodetic Engineering
  • Darja Šemrov University of Ljubljana, Faculty of Civil and Geodetic Engineering
  • Marijan Žura University of Ljubljana, Faculty of Civil and Geodetic Engineering
Keywords: reinforcement learning, Q learning, road artery, traffic control, traffic lights,

Abstract

The basic principle of optimal traffic control is the appropriate real-time response to dynamic traffic flow changes. Signal plan efficiency depends on a large number of input parameters. An actuated signal system can adjust very well to traffic conditions, but cannot fully adjust to stochastic traffic volume oscillation. Due to the complexity of the problem analytical methods are not applicable for use in real time, therefore the purpose of this paper is to introduce heuristic method suitable for traffic light optimization in real time. With the evolution of artificial intelligence new possibilities for solving complex problems have been introduced. The goal of this paper is to demonstrate that the use of the Q learning algorithm for traffic lights optimization is suitable. The Q learning algorithm was verified on a road artery with three intersections. For estimation of the effectiveness and efficiency of the proposed algorithm comparison with an actuated signal plan was carried out. The results (average delay per vehicle and the number of vehicles that left road network) show that Q learning algorithm outperforms the actuated signal controllers. The proposed algorithm converges to the minimal delay per vehicle regardless of the stochastic nature of traffic. In this research the impact of the model parameters (learning rate, exploration rate, influence of communication between agents and reward type) on algorithm effectiveness were analysed as well.

Author Biographies

Rok Marsetič, University of Ljubljana, Faculty of Civil and Geodetic Engineering

Rok MarsetiÄ has more than 9 years’ experience with road-planning, road designing, ITS (intelligent transport systems) and applying geographical information system in traffic engineering. Presently he is the assistant, field of Traffic Engineering, at University of Ljubljana, Faculty of Civil and Geodetic Engineering.

Darja Šemrov, University of Ljubljana, Faculty of Civil and Geodetic Engineering
Darja Šemrov dealt primarily with road transportation planning at the beginning of her career, but in the last three years her researches and work focus mainly on railway transport and railway infrastructure. Presently she is the assistant at University of Ljubljana, Faculty of Civil and Geodetic Engineering.
Marijan Žura, University of Ljubljana, Faculty of Civil and Geodetic Engineering

Dr. Marijan Žura has 26 years experience with transportation planning. He  was a leader or member of team which performed several transportation studies of cities in Slovenia and abroad. He  has also 21 years experience with implementation of geographic information system, specially in transportation. With his work he is using state of the art computer software packages as ARC/INFO, ArcView, Oracle, MS Access, etc. He was the leader of many GIS project and also works as GIS consultant. Also he participates in other researches and project performed at the Traffic Technical Institute. Presently he is associate professor at University of Ljubljana, Faculty of Civil and Geodetic Engineering.

References

Anžek M, Kavran Z, Badanjak D. Adaptive Traffic Control as Function of Safety. 12th World Congress on Intelligent Transport Systems and Services. San Francisco, 2005.

Robertson DI. Research on the TRANSYT and SCOOT Methods of Signal Coordination. ITE Journal. 1986;56(1):36-40.

Hunt PB, Roberetson DI. Bretherton RD, Winton RI. A traffic responsive method of coordinating signals. Crowthorne, Berkshire: Transport and Road Research Laboratory; 1981.

Lowrie PR. Scats, Sydney co-ordinated adaptive traffic system: a traffic responsive method of controlling urban traffic. Darlinghurst, NSW Australia: Roads and Traffic Authority NSW; 1990.

Gartner N. Opac: A demand-responsive strategy for traffic signal control. Transportation Research Board; 1983.

Veljanovska K, Bombol K, Maher T. Reinforcement Learning Technique in Multiple Motorway Access Control Strategy Design. PROMET - Traffic&Transportation. 2010;22(2):117-123.

Sen S, Head K. Controlled optimization of phases at an intersection. Transportation science. 1997;31(1):5-17.

Bingham E. Neurofuzzy Traffic Signal Control [Master’s thesis]. Helsinki, Finland: Dept. of Engineering Physics and Mathematics, Helsinki Univ. of Technology; 1998.

Wiering M. Multi-agent reinforcement learning for traffic light control. ICML ‘00 Proceedings of the 17th International Conference on Machine Learning; 2000.

Thorpe TL, Anderson CW. Traffic Light Control Using SARSA with Three State Representations. IBM Corporation; 1996.

Abdulhai B, Pringle R, Karakoulas GJ. Reinforcement Learning for True Adaptive Traffic Signal Control. Journal of Transportation Engineering. 2003;129(3):278–285.

Gregoire PL, Desjardins C, Laumonier J, Chaib-draa B. Urban Traffic Control Based on Learning Agents. IEEE Intelligent Transportation Systems Conference. IEEE; 2007. p. 916-921.

Lu S, Liu X, Dai S. Q-Learning for Adaptive Traffic Signal Control Based on Delay Minimization Strategy. 2008 IEEE International Conference on Networking, Sensing and Control; 2008. p. 687-91.

Arel I, Liu C, Urbanik T, Kohls AG. Reinforcement learning-based multi-agent system for network traffic signal control. IET Intelligent Transport Systems. 2010:4(2):128-135.

Prashanth, L.A., Bhatnagar, S.: Reinforcement Learning With Function Approximation for Traffic Signal Control. IEEE Transactions on Intelligent Transportation Systems. 2011;12(2):412-421.

Sutton RS, Barto A. Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press; 1998.

Abdulhai B, Kattan L. Reinforcement learning: Introduction to theory and potential for transport applications. Canadian Journal of Civil Engineering. 2003;30(6):981–991.

Perez-Uribe A. Introduction to reinforcement learning [Internet]; 1998. Available from: http://ape.iict.ch/teaching/AIGS/AIGS_Labo/Labo5-RL/sarsa.html

Sutton RS. Learning to predict by the methods of temporal differences. Machine Learning. 1988;3:9–44

Wiering M, van Veenen J, Vreeken J, Koopman A. Intelligent traffic light control. Technical Report. Utrecht University, Institute of Information and Computing Sciences; 2004. 31 p.

Published
2014-04-26
How to Cite
1.
Marsetič R, Šemrov D, Žura M. Road Artery Traffic Light Optimization with Use of the Reinforcement Learning. PROMET [Internet]. 2014Apr.26 [cited 2019Dec.14];26(2):101-8. Available from: http://traffic.fpz.hr/index.php/PROMTT/article/view/1318
Section
Articles