EVALUATING THE LEARNING PERFORMANCE OF EMERGENCY PROCEDURES FOR OPERATORS IN METRO BASED ON A COGNITION MODEL

With the rapid development of the automated metro, valid emergency procedures play a significant role for operators in metro emergency response and recovery. Also, the operators have a challenge to learn different emergency procedures under different automation grades of the metro. Hence, the paper aims to evaluate the learning performance of emergency procedures with regard to the operator. Based on the ACT-R cognitive theory, two decision patterns of the operators are proposed to predict the operator’s learning process for emergency procedures, and a cognition model including 16 production rules and 32 chunks to realize the perceptual encoding and the corresponding determining parts is built. After that, an experiment is further implemented to validate the model results.


INTRODUCTION
An increasing number of metro lines are being built, and the new metro lines are being automated in different cities around the world. The automated metro has several advantages, such as cost-effectiveness, high-traffic frequency and flexibility. Under the automated system, the driver's tasks have been gradually replaced by the complex and automatic functions of a control system. However, the operators are required to recognize emergency situations and handle their abnormal aspects. Automated systems are still man-machine systems, and both technical and human factors are important [1].
The emergencies of a metro system could create extremely serious consequences. For example, the Daegu metro fire accident in Korea in February 2003 caused death of 192 people and 151 people were injured. As a result, considerable efforts have been devoted to improve the safety of the automated metro by metro operators and suppliers. Some projects, such as SECUTR-ED [2], MOD safe and Secure Metro [3], which concern organizational and technological measures, have addressed some types of integrated solutions to increase the effectiveness of the responses to emergencies.
Nevertheless, the existing operational experiences have continuously addressed that human performance is one of the dominant factors affecting the safety of the system process [4][5][6][7][8]. During emergencies, the operators are required to follow highly prescribed emergency procedures that have the specific functions to provide correct and complete instructions in an emergency [7]. The emergency procedures play a further decisive role in the operators' practical processes under challenging environments, such as significant time pressure or uncertain conditions [9]. Therefore, it is essential to successfully provide useful emergency procedures for the operators to accomplish their tasks.
Many qualitative guidelines or checklists have been employed to confirm whether emergency procedures satisfy the necessary requirements, such and distractions [17]. In the nuclear field, a cognitive model of the operator's high-level activities has been employed to describe a situational assessment and response-planning process with the concept of modelling decision-making in an emergency [18][19][20], and in the aviation domain, a computational cognitive model of pilots has been used to diagnose possible sources of human performance and error prediction [21]. From these studies, it is demonstrated that the cognition theory is a significant contributor to the research of human performance.
The concept of higher automation makes the metro staff to be in a more alert state for emergencies. In other words, a higher grade of metro system automation has more requirements from the operators to emergency procedures. For this reason, the objective of this paper is to discover how the variances of automation grades affect the operators to learn different emergency procedures and how the operators store and cognize the changes with the evolution of the metro system. The remaining part of this paper is organized as follows: Section 2 and Section 3 briefly introduce the cognition theory adopted in this study and the background about the study. After that, a cognitive model based on ACT-R theory, simulating operator knowledge and how operators think about emergency procedures, is proposed in Section 4. Additionally, a GUI object is created to provide the interface of the external world with the cognitive model in Section 5, and in Section 6, a corresponding experiment is designed to validate the result of the cognitive model. The discussions about the comparison between the simulation results and the experimental data are analysed in Section 7. Finally, some conclusions are drawn in Section 8.

BRIEF INTRODUCTION OF ACT-R
The ACT-R theory was presented by Anderson [22] to model and understand the human higher level of cognition. This theory has been applied to a broad domain such as mathematical problem solving in the classroom, driver behaviour prediction in car driving, and human memory and other tasks. ACT-R is such a comprehensive theory that can model human thinking, receiving and processing of information, as well as the learning and forgetting processes.
There exist two assumptions representing human knowledge in the ACT-R theory: declarative knowledge and procedural knowledge. The former is used as "emergency procedures should be prepared for operators to easily and clearly understand the operational context" or "emergency procedures should be designed appropriately for operators to accomplish their tasks within an acceptable workload" [10][11][12]. However, a qualitative method has several drawbacks, such as a subjective way of working or ambiguous requirements that cannot give a quantitative evaluation of the emergency procedures.
For the drawbacks of the qualitative methods, some researchers have devoted to applying mockup test technologies on the evaluation of emergency procedures. For example, the emergency procedures for airport transport were tested by a mock-test method to analyse the relative efficiencies of four specific configurations [13]. Nevertheless, the mock-up test method cannot perform the emergency procedures frequently and thoroughly, and it is always time-consuming.
Furthermore, an extensive effort has been made to quantitatively study how the emergency procedures affect human performance. Park et al. [14] proposed a task complexity measure for the operating procedures of nuclear power plants and further validated the appropriateness of this measure by comparing task complexity scores with two sets of operator response time data. In this method, the abstract hierarchy (AH), which considers the amount of knowledge required to recognize the place of a problem in accomplishing the task, is one of the innovate points of the complexity measure for the task [15].
The emergency procedures are designed as stepby-step instructions that can cover all the emergency situations [16]. As the executor of the emergency procedures, the operators play a significant role in handling the emergency process; thus, they should fully understand the context of the emergency procedures. Although the task complexity measure can provide a quantitative evaluation of emergency procedures, it is insufficient in exploring the intrinsic correlation of the human factors and emergency procedures with the cognitive aspects.
Generally, the cognitive theory is presented to provide a powerful tool for understanding how a human thinks and predicting how a human will behave. For this advantageous feature, many studies have applied the cognitive theory to their specific domains. A rigorous computational model of driver behaviour has been explored in a cognitive architecture to predict and recognize the driver's behaviour 0 to 4 [24]. Among these GOAs, GOA 3 and GOA 4 metro systems have fully automatic train control system to drive the train, ensure safe movement of trains, and supervise the guideway, and the systems of GOA 0 to GOA 2 require a driver to accomplish the necessary train driving tasks.
With the development of the metro system automation, the emergency procedures should be adjusted correspondingly with respect to the organization structure and system configuration. Although the higher GOA system provides more automatic functions and even replaces the role of the train driver in GOA 4, the automatic functions cannot cover all the emergency scenarios, and hence, the operators in Operating Control Center (OCC) should take a more dominant position in the higher GOA systems than in the lower ones.
The emergency procedures of a metro system involve detailed guidance on how the operators should check the emergency situation, what the operators should do, and with whom the operators should communicate, such as "the traffic operator contacted the driver to confirm a fire in case of a train fire occurring between stations" in GOA 2 and "the traffic operator contacted the vehicle train attendance to confirm a fire in case of a train fire occurring between stations" in GOA 3. The behaviours of a traffic operator will be changed along with different emergency procedures, and the changes will become the key factors to affect the emergency performance. The corresponding reasonable responsibilities for different elements, such as the passengers, train, signaling system, to explain the things that we are aware of, and the latter is the knowledge we could display in human behaviour but that we are not conscious of [23].
Based on these assumptions, the ACT-R architecture consists of a set of modules. Each module in the ACT-R model performs a respective cognitive function and is capable of other modules independently, as shown in Figure 1.
These modules communicate with each other through the buffers. One of these modules, storing all of the chunks that represent declarative knowledge, is the declarative module. Declarative memory represents the knowledge that we already know and have learned. The procedural module is employed to hold all the procedural knowledge to decide how we behave in the world.

BACKGROUND
One view of the operators in emergencies is that the operators are capable of reading and following the steps of the emergency procedures [7]. In the metro system, the behaviour of operators under the guide of emergency procedures can also make a substantial impact on the recovering ability from a disturbed circumstance to safe operation. Alternatively, insufficient understanding and incorrect cognition of emergency procedures can weaken the emergency performance and even lead to unsafe operational recovery of the metro system.
Based on the functions automatically implemented by the system equipment, the metro system is defined into Grade of Automation (GOA) from  tions and organizational structures. Therefore, we could simplify the case of emergency procedures and select the key information that the traffic operator should know, as shown in the following items: -When a fire alarm occurs, contact the driver to confirm whether there is a fire; -When a fire alarm occurs, contact the driver to confirm whether the train can be moved; -When a train fire occurs and the train can be moved, contact the driver to continuously drive the train to the next the station; -…… With a description of the emergency task, the traffic operator should always analyse the emergency condition and decide to contact the respective "object" to confirm the next action. For this causal relationship, the pattern of making a decision is related to three elements that are abstracted as "condition", "object" and "action". Based on this, two decision patterns are employed to simulate the decision process. As shown in Figure 2, Pattern 1 indicates that all the three elements in the imaginal module will be the retrieval indices, and Pattern 2 means that the correct decision chunk in the declarative module can be retrieved by chunks "condition" and "object" in the imaginal module.
To provide a retrieval, the chunks in the imaginal buffer spread activation to the chunks in the declarative module, and the slots in the imaginal chunks spread an amount of activation based on their relationships to other chunks in the declarative module. For example, it is assumed that the two slots which are "condition" and "object" are involved in the imaginal buffer. As a source, the two slots provide a context to retrieve the matched chunk, and therefore, the total activation A i of the potential chunk i is defined in Equations 1 and 2 [25]: station and infrastructure, should be assigned to the operators of different GOAs. The detailed responsibilities of operators from the role-function are summarized in Table 1. The different positions of operators indicate the different context and interactive patterns for an emergency task, and Table 2 shows the sort of operators involved in GOA 2 to 4 systems.

MODEL
Under the distinct emergency procedures of the GOA systems, it is a challenge whether the operators can understand the emergency tasks and how the operators can master the new context of the emergency-handling process. For this purpose, a cognitive model has been designed to simulate the adaptive capability of operators for learning different GOAs' emergency procedures.
Having the dominant and core role in the OCC organization, the traffic operator is modelled to focus on human performance in the emergency case of a train fire occurring between stations as a typical application. As aforementioned, the traffic operator should contact the corresponding operator or other staff to inquire about the information and make different decisions with different system configura- Environment operator Managing and supervising electromechanical devices, water supply and drainage system.

Power operator
Managing and supervising third rail power supply devices.
Passenger operator Supervising the passengers on-board and assisting passengers in emergencies.
Vehicle operator Supervising the status of equipment on-board and handling the exceptional situation on-board remotely.
Maintenance operator Receiving information of the failure and assigning the maintenance task. are three chunks in the imaginal buffer, and thus W j in Pattern 2 is equal to 0.5. The parameter: mas is defined as 3.0 in the model. From different GOA emergency procedures, "condition", "object" and "action" could occur in the range from one to ten times. As a result, the aim of this cognitive model was to determine the human decision patterns of the different emergency procedures of GOAs with respect to the value of fan. The logical flow chart of the cognitive model is shown in Figure 3, and the main process is as follows: attend to the content of the emergency procedures in the external interface -> Retrieve the abstract content and encode the perceptual information in the imaginal buffer -> Retrieve the memory module with the sources in the imaginal module -> Match the procedural rules -> Generate output through the motor module. Typical processes of the traffic operator range from attending to the emergency procedures to making a judgement as to which GOA it belongs.
ACT-R 7.0 was adopted to accomplish the aforementioned process. The model includes 16 production rules and 32 chunks to realize the perceptual encoding and the corresponding determining parts. The perceptual encoding part has a function that could enable the model to read the key word from the screen and store it as imaginal information, and the corresponding determining part employs the production rules on how the model makes decisions for the emergency procedures. The main production rules are listed in Table 3 [27]: Production 1 was used to determine the location of the word such as "person" on the screen.
where: B i -base-level activation of chunk i; W j -attentional weights from source i; S ji -strength of association from source j to chunk i; fan j -number of chunks in declarative memory, in which j is the value of a slot plus one for chunk j being associated with itself; S -maximum associative strength.
Total activation A i also determines the time of its retrieval. When a retrieval request is made, the time it takes is given by Equation 3 [25]: where F is the latency factor; e is the natural constant. The chunk will be retrieved if its activation exceeds the retrieval threshold when a retrieval request is made. The probability of retrieval is defined in Equation 4: where τ is the retrieval threshold; s is the amount of noise in the system. Parameter S ji , which indicates the strength of association from source j to chunk i, is calculated by Equation 2. In the activation equation, parameter S is valued approximately 1.5 by Anderson and Reder [26], and we set the value as 1.6 to fit the fan data in our model. The W j is used to reflect the value of the imaginal activation parameter in our application; the W j in Pattern 1 is equal to 0.333 because there  For our proposed simulation model, an execution object based on Lisp has been designed to provide the interface for the logical model. In our case, a supervision interface was designed to simulate the detailed information about "condition", "object" and "action" in the emergency procedures. Figure 4 shows the visual window provided for the experiment of the cognitive model in which the necessary information is presented for attending to the model: when "condition"+ con "object"+ do "action". The width of the visual window is 1,000 pixels and the height of the visual window is 300 pixels.
In Figure 4 the "when" indicates the type of the emergency and its environment when it happens, the "con" denotes the dispatcher who should be involved in this emergency, and the "do" is followed by the actual tasks or the necessary actions which are implemented by the dispatcher.
In this simulation interface, the window produces the content with which the cognitive model can interact, and the cognitive model was informed to "attend" to the new refreshment, which is the change along with the information on the visual interface window. The detailed process for creating the interface is as follows: open the window -> clear the display -> present one of the emergency procedures -> wait for a response -> record the response -> analyse the results. Production 2 was used to request a shift of attention to the location of the word in production 1.

GUI OBJECT
In most cases, the cognitive model has to interact with the external real world to obtain the information inquired by the internal logical tasks. To accomplish the interactive behaviour, the abstract representation of the external world, which is called a device, is necessary for the cognitive model.

Note: g represents the goal buffer, r indicates the retrieve buffer, vl means visual location buffer, i indicates the imaginal buffer, v is the visual buffer, and m represents the motor buffer.
participants, the operators were the work staff at the Beijing metro with an average working experience of five years and an age interval between 30 and 40 years, and the students were the graduate students at the National Engineering Research Center of Rail Transportation Operation and Control System and their approximate age interval was between 24 and 26 years. Both male and female participants took part (30% female, 70% male). The emergency procedures were simplified to several knowledge tasks to facilitate the experiment process. Because some of the participants had not learned English well before, the procedures were not translated from Chinese into English and were described in Chinese. The knowledge tasks of the traffic operator in case of a train fire occurring between stations are shown in Tables 4-6.
From the interface, the cognitive model can see and manipulate the corresponding information and consider it as the input parameters of the logical model. The new information is considered to be perceptual information that can be encoded in the cognitive model. With the encoded input parameters, the model can match the procedure rules during the conflicting resolution process. Then, the matching procedure with the declarative memory is preferred to determine what should happen.

EXPERIMENT
To validate the predictive results of the cognitive model in the laboratory, we designed an experiment in which 29 persons (22 students and 7 operators) participated to judge which GOA the learned emergency procedures belonged to. Among the  Table 4 -Knowledge tasks of a traffic operator in case of a train fire occurring between stations (traditional GOA 2 system) 1.
If a fire alarm occurs, contact the driver to confirm whether there is a fire.

2.
If a fire alarm occurs, contact the driver to confirm whether the train can be moved.

3.
If an uncontrolled fire and unmovable train are confirmed, contact the driver to guide passengers into an evacuation section.

4.
If a controlled fire and unmovable train are confirmed, contact the driver to control the fire.

5.
If a controlled fire and unmovable train are confirmed, contact the driver to wait for rescue.

6.
If a fire and movable train are confirmed, contact the driver to move the train to the next station.

7.
If a fire is not confirmed, contact the driver to move the train to the station.

8.
If a fire is not confirmed, (traffic operator) cancel the fire alarm.

9.
If station evacuation occurs, contact the driver to broadcast on-board. 10. If station evacuation occurs, contact the driver to open the train door in the station.
11. If station evacuation occurs, contact station staff to guide passengers into station evacuation.
12. If station evacuation occurs, (traffic operator) hold the train at the next station. 4) Repeat steps 1 to 3 until the results of the test are all correct. The time for each relearning will be reduced by 10 seconds.

SIMULATION RESULTS AND DISCUSSIONS
To reveal the process of how the operators understand these emergency procedures of different GOAs, the simulation data were compared with the experimental data.
As shown in Figure 5, x-axis denotes the tested emergency procedure rule number, and y-axis represents the responsive time of the participant. The blue dots with broken lines denote the simulation data, and the red dots represent the actual data from the experiment. Table 7 shows the correlation coefficient of each procedure rule between the cognitive model and the experiment. From the table, most correlation coefficient values of procedure rule numbers are over 0.9, and only one procedure rule correlation coefficient value is between 0.8 and 0.9, which indicates that the results of the proposed cognitive model have Table 4 is learned by the participants to memorize the knowledge of the traditional system (GOA 2). After that, the knowledge tasks of higher levels of automated metro systems (GOA 3 and GOA 4) are further learned to form the respective memory, as shown in Tables 5 and 6. After that, the participants are required to judge which GOA system the knowledge task belongs to. Six sets of knowledge tasks are chosen to test the decision process of the participants. Each knowledge task in the same set has the same condition but different objects and actions in different GOAs. The time interval of participants from checking the knowledge tasks to outputting the levels of GOA is recorded.
In conclusion, the process of the cognitive experiment about knowledge tasks is shown as follows: 1) Learn Tables 4-6 within 1 minute for each table and the initial interval is 60 seconds. 2) Judge the test item (knowledge task in  to which GOA it belongs. 3) Inform the participants whether the answers are all correct.

2.
If a fire alarm occurs, contact the train attendant to confirm whether the train can be moved.

3.
If an uncontrolled fire and unmovable train are confirmed, contact the train attendant to guide passenger into an evacuation section.

4.
If an uncontrolled fire and unmovable train are confirmed, contact the train attendant to control the fire.

5.
If a controlled fire and unmovable train are confirmed, contact the train attendant to wait for rescue.

6.
If a fire and movable train are confirmed, (traffic operator) confirm the train should move to the next station.

7.
If a fire is not confirmed, (traffic operator) cancel the fire alarm.

8.
If station evacuation occurs, contact the train attendant to broadcast on-board.

9.
If station evacuation occurs, contact the station staff to guide passengers into station evacuation.
10. If station evacuation occurs, (traffic operator) hold the train at the next station.

1.
If a fire alarm occurs, contact the vehicle operator to confirm whether there is a fire.

2.
If a fire alarm occurs, contact the vehicle operator to confirm whether the train can be moved.

3.
If an uncontrolled fire and unmovable train are confirmed, contact the passenger operator to guide passengers into an evacuation section.

4.
If a controlled fire and unmovable train are confirmed, contact passengers to control the fire.

5.
If a controlled fire and unmovable train are confirmed, contact the station staff to guide passengers into an evacuation section.
6. If a fire and movable train are confirmed, (traffic operator) confirm the train should move to the next station.

7.
If a fire is not confirmed, (traffic operator) cancel the fire alarm.

8.
If station evacuation occurs, contact the passenger operator to broadcast on-board.

9.
If station evacuation occurs, contact the station staff to guide passengers into station evacuation.
high correlation with the results of the experiment. The simulation results are expected to have high correlation with the experimental study at an optimal fitting level. Based on the simulation process and the experiment of the participants, the decision pattern of each procedure rule with the fan value is identified to demonstrate the human cognitive performance of an emergency decision.  show the recognition latencies for six procedure rules of different GOAs from the simulation, and each procedure rule has its respective decision pattern for "condition", "object" and "action" fan values. For most knowledge tasks, the recognition delays of GOA 3 could be shorter than those of GOA 2 and GOA 4.
From Tables 8-10, the effect of fan is the factor that affects the recognition latencies, and at the same time, the decision pattern has a positive influence on the response time. Decision pattern 1 is the main   tions. The prediction results for the operator's cognitive capability indicate fewer new positions will lead to less recognition delays. Modelling operator behaviours in cognitive architectures could be considered as a beneficial approach for automated metro about how the evolution of GOAs affects the staff training. Potential applications include the exploration of the higher-level cognitive activities of operators and the evaluation of the human-machine interface in metro.
Further study will focus on a deliberate cognitive approach to explore the higher-level cognitive activities of operators in an automated metro system, and more research will be done on how feedback of the cognitive model influences the emergency procedures in emergency scenarios. cognitive approach for GOA 4 system, but for GOA 2 and GOA 3 systems, Pattern 2 rather than Pattern 1 plays the dominant role.
Among these participants, most of the operators prefer Pattern 1 for GOA 4 system and GOA 3 system; also, they use both Pattern 2 and Pattern 1 for GOA 2 system. Furthermore, their experimental data are shorter than the average data from student participants.
However, the variance of the response time between the simulation model and the experiment indicates that the parameters in ACT-R model need to be optimized. Particularly, some distinct different data are caused by individual and emotional factors, and it is difficult to simulate the disturbing cases in the ACT-R environment. In addition, the fixed areas are employed to attend to the knowledge of the ACT-R model, but the areas for procedural rules in the actual testing environment are variable.

CONCLUSION
The proposed model accounts for the visual awareness, hand output and memory-related decision-making to simulate the different levels of an operator's learning behaviour. The perceptual-motor process to simulate how the operators focus on emergency procedures is close to practical learning scenarios. Also, the data comparison of the simulation and the experimental study indicated that the quantitative analysis was nicely available in the cognitive model. A prediction for the cognitive capability of the learning emergency procedures can provide feedback for the optimal design of operational guidance and procedures. The recognition delays are related to the fan value of the basic knowledge in emergency procedures and the pattern by which the decision is made. The recognition delays of emergency procedures in GOA 4 are more than that of other lower GOAs under the same condi-