A FRAMEWORK FOR PERFORMANCE EVALUATION OF VANETS USING NS-3 SIMULATOR

During the last decade, the number of vehicles on roads has been rapidly growing. Therefore, the demands for communication on the move are also increasing and the attention from many researchers is focused on the Vehicular Ad hoc NETworks (VANETs) because of their importance for Intelligent Transportation Systems (ITSs). Due to the complexity and cost of practical evaluation of VANETs, the researchers often rely on network simulation in order to evaluate their work. In this paper, we have developed a Network Simulator 3 (NS-3) based framework for VANETs that provides network performance analysis based on the key performance indicators such as throughput, packet loss ratio, overhead, end-to-end delay, jitter, etc. Since VANETs are highly dynamic networks, many researchers have proposed different routing protocols in order to improve the network performance. In this paper we have compared several topology-based routing protocols, and proposed utilization of the commonly used Expected Transmission Count (ETX) metric to improve VANET performance.


INTRODUCTION
Modern life cannot be imagined without everyday usage of vehicles on the road. Intelligent Transportation Systems (ITSs), including their implementation in real life, are going through constant optimization in order to provide better performance. Since new technologies always follow modern trends, it is not surprising that many vehicle vendors offer facilities to both drivers and passengers that include safety applications, assistance to the drivers, emergency warning, etc. [1]. All these facilities could not be provided without the usage of wireless networks. Beside cellular networks, Wireless Ad hoc NETworks (WANETs) are used daily in the form of infrastructure and Mobile Ad hoc NETworks (MANETs). MANETs are networks which have completely mobile nodes and require very little or no infrastructure. Vehicular Ad hoc NETworks (VANETs) are a sub-class of MANETs in which mobile nodes are vehicles. VANETs have been gaining significant attention from the research community due to their increasing importance for the building of ITS.
Having all this in mind, it could be said that VANETs have some special properties. They need to support the increased number of wireless equipment that can be used in vehicles [2]. VANET architecture should be able to allow the connection between vehicles or between vehicles and fixed Road Side Units (RSUs). According to [3], there are four possible types of VANET communication, as depicted in Figure 1: -Vehicle to Vehicle (V2V) allows direct vehicular communication without any fixed infrastructure support; -Vehicle to Infrastructure (V2I) allows the vehicle to communicate with the RSU mainly for information and data gathering applications; -Hybrid architecture combines both V2I and V2V. If a vehicle cannot directly communicate with RSU then it uses V2V communication with some other vehicle that has direct connection to RSU. If there is direct connection, then it uses V2I communication; -Infrastructure to Infrastructure (I2I) is communication between two RSUs or between RSU and some external base station such as mobile base station.
Regardless of the type of VANET communication, its main goal is to avoid accidents by allowing vehicles in transit to send relevant traffic data to one and open-source, are available and can be employed to run the simulation modelling for VANETs. In this paper we are using Network Simulator 3 (NS-3) [5,6] in order to evaluate the performance of different routing protocols and metrics in V2V communication in a city scenario.
Although NS-3 is popular and a widely-used simulator that supports several routing protocols such as Ad hoc On-demand Distance Vector (AODV) [7], Dynamic Source Routing (DSR) [8], Destination Sequenced Distance Vector routing (DSDV) [9], Optimized Link State Routing (OLSR) [10], and others, it lacks a few features that are important for efficient simulations of VANETs. First, NS-3 does not support other metrics than hop count. In order to test their impact on performance of routing protocols in VANETs, we have developed a model for Expected Transmission Count (ETX) [11] metric within AODV protocol. The second problem of the NS-3 simulator is that there is no suitable method for efficient calculation of the basic Key Performance Indicators (KPIs) such as throughput, packet loss ratio, end-to-end (E2E) delay, jitter, received packets, routing packets, dropped packets, etc. Therefore, we have developed a framework for gathering simulation data, perform necessary statistical data processing and calculating KPIs, in order to efficiently analyse VANET networks. Based on the developed framework and ETX metric model implementation, we have done a series of simulations in order to test the performance of the routing protocols and metrics in VANETs.
This paper is organized as follows: the state of the art that provides information of mostly used simulators and routing protocols in VANETs is summarized in Section 2. The proposed NS-3 improvements for efficient simulation of VANETs are given in Section 3. Simulation results and discussions are given in Section 4, and the conclusion is given in the final section.

Simulators used for VANETs
Simulation is a popular choice for modelling the real-life activities especially in networks such as VANETs that are not easy to evaluate in practice [12]. Network simulators are often used to analyse the performance of network protocols under different network topologies and parameters. Although many different types of simulators, both commercial and another over an ad hoc network. This can be done in various ways, but the most frequent one is that the drivers receive a warning if there is a risk of accident or that the vehicle itself may take pre-emptive actions such as braking to slow down. In order to deliver these warnings and to forward packets it is necessary that vehicles in VANET can establish a high-quality route from source to destination. Therefore, routing protocols and appropriate choice of a routing metric play a very important role in VANETs.
In recent years many wireless network researchers have proposed different routing protocols and metrics in order to improve the network performance. However, the evaluation of proposed improvements is often not an easy task. When possible, measurements, experiments, and theoretical analysis should be included into research. Regardless of the network nature, and due to its large-scale heterogeneity and dynamics, it can be complicated, time-consuming and expensive to provide experimental evaluation of the new protocols. In turn, analysing VANET means to include several vehicles that have infrastructure to be part of the ad hoc network; practical experiments of VANETs are not easy and low-cost to perform. If researchers need a cost-effective and scalable mechanism to analyse the behaviour of the network or their own model under different conditions or with different parameters, network simulation is a very popular choice. Therefore, network simulations can help in achieving profound understanding of the network behaviour, as well as the verification of new solutions [4]. This is especially important for VANETs due to their complexity and difficulty of practical evaluation. Different types of simulators, both commercial  [3] VANET scenarios. However, this is not always the case. In order to solve this problem, the users have two common choices.
One option is to use external mobility generators in order to model the vehicle movement. These external tools often generate mobility trace files in a format that is suitable for importing into the network simulator. One of the most popular traffic engineering simulators of mobility is Simulation of Urban MObility (SUMO) [18]. SUMO is an open-source traffic simulator that provides route selection, traffic light control, and V2V communication. Its output cannot be directly used with a network simulator, but with additional software its output files can be converted into the format suitable for importing in OMNET++, NS-2 or NS-3. The MObility model generator for VEhicular networks (MOVE) [19] is an extension of SUMO for realistic traffic pattern generator in VANET simulations. Vanet-Mo-biSim [20] is a Java-based generator that supports both generating random maps and importing real geographical US-based maps. The output files are compatible with NS-2 or GloMoSim. In our work we have used one simple traffic generator BON-NMOTION [21] for generating vehicle mobility according to the Manhattan grid model.
Other option for modelling vehicle mobility is to use simulators that have integrated VANET mobility support within the network simulator itself. These integrated frameworks for executing the simulations without the need of running different software and resolving their inter-dependencies are given in [22][23][24][25]. VEINS (Vehicles in Network Simulation) [22] is an open-source VANET framework that uses SUMO as the mobility generator and OMNET++ open-source ones, can be used to run the simulation modelling for VANETs, in this paper we have focused on simulators that support the IEEE 802.11p standard for VANETs. The mostly used open-source simulators for VANETs are Objective Modular Network Testbed in C++ (OMNET++) [13], Global Mobile Information System Simulator (GloMoSim) [14], Network Simulator 2 (NS-2) [15], Network Simulator 3 (NS-3) [5,6], etc. Two most popular commercial network simulators are OPtimized Network Engineering Tools OPNET [16] and QualNet [17].
Based on literature, NS-2 is certainly one of the most used network simulators, but in the last couple of years OMNET++ and NS-3 simulator usage has grown rapidly. NS-3 is a single language simulator, written in C++, which introduces new simulation core with several advantages that promise to provide more reliable results than NS-2. Although it is a completely new simulator, it has inherited NS-2 popularity, so that it is not surprising that everyday number of NS-3 users is increasing and many researchers choose it for their research. An analysis of the suitability of using the NS-3 simulator in evaluating the performance of VANETs is given in Table 1. Considering the data in Table 1, it can be concluded that the NS-3 simulator is a good starting point for simulating VANETs, but lacks some general and VANET-specific elements to make it an effective VANET simulation platform. One of the aims of this research is to overcome these shortcomings.
In VANET, modelling vehicle mobility plays a very important role, so network simulators usually include several mobility models that can be used in Table 1

Pros Cons
General -open-source, which is necessary for research purposes; -only one programming language (C++) is used, which simplifies development and integration of new solutions; -good documentation is available online; -good community support; -direct code execution (DCE) support; -good support for custom-made extractors of simulation data.
-C++ is complicated for beginners; -there is no simple and consistent method for calculation of network key performance indicators (they must be custom-made by the users).
VANET specific -provides good integration with other simulation environments (mobility simulators); -includes proactive and reactive routing protocols; -includes Spectrum PHY model for wireless channels, which enables simulation of heterogeneous networks; -provides deterministic and probabilistic propagation models, and includes building models for city scenarios; -includes implementation of IEEE 802.11p standard; -includes implementation of WAVE standard.
-does not integrate mobility models for VANETs; -does not include VANET specific routing protocols; -lacks in applications with different QoS for data traffic generation.
to keep a record of updated network routes all the time. The nodes exchange topology information so that they all have the same view of the network. This helps to detect topology changes. Whenever a node needs to send a message, it just searches the routing table for the path to destination. However, maintaining an up-to-date topology in the routing tables causes a high control overhead. Two of the most popular proactive protocols are DSDV [9] and OLSR [10], but there are many others. For instance, performance comparison of DSDV and Dynamic MANET On-demand (DYMO) [27] protocols for VANETs in OMNET++ simulator is given in [28]. Reactive (on-demand) routing protocols generate route discovery only when it is needed. The routes to destination nodes are established only when the nodes have some data to send to an unknown destination. The source node then initiates a route discovery mechanism. Comparative to proactive protocols, the control overhead in reactive protocols is reduced; however, the route searching process that occurs before data packets can be forwarded may cause source node to suffer long delays. Two mostly used reactive protocols are AODV [7] and DSR [8]. A relative investigation of OLSR, AODV and DSR routing protocols in highway and city scenarios by varying the number of vehicles in both city and highway scenario is given in [29]. In the literature one can find improvements of AODV protocol specially focused on VANETs. For instance, Urban-AODV [30] protocol is designed for VANETs in urban environments, and another improvement of AODV protocol as the network simulator. iTETRIS [23] simulation platform for the large-scale evaluation uses SUMO and NS-3 as the network simulator with the help of iCS (iTETRIS Control System). The Trust Evaluation And Management (TEAM) framework, built using VEINS simulation environment which incorporates SUMO and OMNET++, is proposed in [24]. An extension of the SUMO and OMNET++ with realistic models of electric vehicle operations and services is proposed in [25].

Routing protocols for VANETs
Choosing adequate routing protocol is one of the major challenges in VANETs because of frequent changes in the network topology caused by fast movement of the vehicles. Routing protocols used in VANETs could be divided into five categories, shown in Figure 2 [26].
In order to perform packet forwarding, the topology-based routing protocols use link information that exists in the network. With topology-based protocols all type of packets could be sent, unicast, multicast and broadcast. The lesser resources and fewer bandwidth consumptions are needed than with other protocol types. On the other hand, these protocols provide more overhead because of the route discovery mechanism. Sometimes, protocol fails to discover a route because of the frequently moving nodes. They can be divided into three subgroups: proactive, reactive and hybrid.
Proactive routing protocols are protocols in which all nodes have routes before the packet needs to be sent. These are  Figure 2 -Classification of routing protocols for VANETs [26] the cluster then the cluster head creates a virtual network infrastructure which provides scalability. With these protocols good scalability can be provided for large networks but from the other side, they create large E2E delays and routing overhead. Clustering for Open Inter vehicular communication Network (COIN) [38] and Location Routing Algorithm with Cluster Based Flooding (LORA_CBF) [39] are the main examples of these routing techniques. Another example is Trust Based Authentication Technique (TBAT) proposed in [40] and evaluated using NS-2 simulator.
Broadcasting routing protocols flood packets over the entire VANET among all nodes inside a broadcast domain. Broadcast protocols are used for sharing safety-related information like climate, traffic, road conditions among vehicles, emergency warning, various announcements and delivering advertisements. These protocols provide good network reliability, but broadcasting can consume a large amount of network bandwidth and create many replicated packets in the destination nodes. Many broadcasting routing protocols have been developed, such as Distributed Vehicular broadCAST protocol (DV-CAST) [41] and DEnsity aware reliable broadCAsting protocol (DECA) [42].
Geo-cast routing protocols are position-based protocols that are used for sending multicast packets. The main idea is to deliver packet from one source node to a group of destination nodes that are within some specified geographical region called Zone Of Relevance (ZOR). If a vehicle is out of ZOR it will not be alerted in the case of some danger. When destination node needs to answer a message, communication is done with unicast packets using Zone Of Forwarding (ZOF). If a vehicle comes inside a new ZOF it has the responsibilities to forward data packet to other ZORs. With these protocols the network overhead is reduced and packet delivery is highly reliable. On the other hand, packet transmission is often greatly delayed. The various Geo-cast routing protocol representatives are Inter Vehicle Geo-cast (IVG) [43], Direction-based Geo-CAST Routing for Query Dissemination in VANET (DG-CASTOR) [44] and Distributed Robust Geo-cast (DRG) [45].
Beside all the mentioned protocols, in the available literature more solutions that improve performances of VANETs can be found [46][47][48]. In [46] the authors have proposed a Guaranteed Time Slot (GTS) allocation scheme to improve multiple-access performance in vehicular sensor networks which is evaluated using the Matlab and Opnet simulations.
for VANET is given in [31], which modifies the basic AODV routing parameters to provide better quality of service.
Hybrid protocols mix proactive and reactive methods. Firstly, all the routing information that is unknown is acquired by using proactive routing. Then, reactive routing mechanisms are used to maintain the routing information when the topology changes. Zone Routing Protocol (ZRP) [32] is a typical example of hybrid routing protocol. Each node has its own zone, and if some packet needs to be sent to some node that is in the same zone as the source node, the proactive protocol is used. However, if the destination node is outside the source node's zone, a reactive protocol is used. By this processing overhead is reduced.
In the position-based routing protocols, whenever the source node needs to communicate with the destination node, it uses both geographical position and network address. The position-based protocols do not require routing tables. Each node only needs to learn geographical information of its neighbours and the destination node, in order to determine its own next hop. Having in mind that most of the vehicles already have built-in positioning system, such as Global Position System (GPS), position-based protocols use this system to find the optimal route to destination. They are more suitable for networks with distributed nodes, and they have the lowest processing overhead. On the other hand, position-based routing needs GPS, so the problem can arise when GPS devices lose satellite signal, for example in tunnels. The main representatives of this category of protocols are Geographical Source Routing (GSR) [33], Greedy Perimeter State-less Routing (GPSR) [34] and Distance Routing Effect Algorithm for Mobility (DREAM) [35]. There are some improvements of the basic protocols in the literature, such as Maxduration-Minangle GPSR (MM-GPSR) routing protocol, created as an improvement of GPSR protocol for VANETs [36]. A new strategy named Path Aware GPSR (PA-GPSR) for VANETs, which extends routing tables in order to select the best path within NS-3 simulator is given in [37].
Cluster-based routing protocols are protocols in which a group of vehicles that have similar characteristics like speed, direction, etc., create a cluster. One node in the cluster is called cluster head and it is responsible for packet delivery to the rest of the nodes in cluster and for communication with other clusters. Packets between nodes in the same cluster are sent directly, but if the destination node is outside where p f represents the probability of successful packet transmission, and p r the probability of successful received ACK packet. The probabilities p f and p r are measured using dedicated Link Probe Packets (LPPs) which are broadcast every τ seconds [11].
Since the basic AODV protocol uses hop count as routing metrics, in order to implement AODV-ETX protocol few modifications of the basic AODV protocol had to be done. First, the additional LPP packets that are used to measure ETX are implemented. Secondly, since in the basic AODV protocol field for routing metric does not exist, in order to transfer ETX metric of some route through the network, this additional field must be created in the routing tables, Route Request (RREQ) and Route Reply (RREP) packets. Last modification is in the way in which network nodes handle RREQ packets. In the basic AODV protocol, when a node receives RREQ with already seen ID, it would not answer it. However, with AODV-ETX protocol, even if RREQ with already seen ID is received, the node will check if the ETX metric of the received RREQ is smaller than the previously known ETX metric, and if true, the node would answer that RREQ. This way, at the end of the route discovery procedure source node will have a route with the smallest ETX metric, and therefore the best quality route. A detailed description of this implementation can be found in [49], and the source code is publicly available in [50].

Framework for VANET performance evaluation
Regardless of the network type, one important part of each simulation is to evaluate the simulated scenario with reliable output statistics of basic network KPIs. Obviously, it is not possible for a network to achieve the best results for all KPIs, since many of them have opposite requirements. KPIs of a network must be realistic and the network should have acceptable results in all KPIs [51]. Currently, the collection of the simulation data in NS-3 can be realized using several methods.
-The first method is to use Flow Monitor [52].
This tool gives performance metrics on network layer instead of application layer and can be used only for unicast IP flows over TCP/UDP. If a user does not use TCP/UDP or uses broadcast packets, then Flow Monitor cannot be used NS-3 simulator is used in [47] to demonstrate the novel system architecture for VANETs that relies on the dynamic spectrum access framework. VANETs Quality of Service (QoS) based routing protocol based on multi-constrained ability to support ITS infotainment services are simulated using SUMO and NS-3 simulator in [48]. As previously stated, many researchers compared the performance of the most commonly used topology-based protocols. But very few of them included routing metric other than hop count in their protocols. Due to high mobility and frequent topology changes, the quality of the links in the VANETs frequently changes. Therefore, it is important to create reliable routing metrics in order to select the route from source to destination with the best KPIs. In our work we focused on NS-3 implementation of the most frequently used ETX [11] metric in order to study its impact on the performance of AODV protocol in VANETs.

NS-3 EXTENSIONS FOR VANET PERFORMANCE EVALUATION
NS-3 simulator is used by many recent research studies especially in developing and evaluating new network protocols. Although it is a very popular tool, NS-3 is a relatively new simulator so it is still lacking in some protocol models and features that are important for VANET simulations. First, NS-3 does not support routing metrics, so we have developed a model for ETX metric within AODV protocol. Also, NS-3 simulator does not include appropriate tools for the calculation of basic network KPIs such as throughput, packet loss ratio, E2E delay, jitter, etc. Hence, a researcher must develop and implement their own solution to KPI calculations. This leads to implementation differences and potential errors, and makes it difficult to compare the achieved results among different research groups. Therefore, we have developed a framework for gathering simulation data, performing necessary statistical data processing and calculating KPIs, in order to efficiently analyse the performances of VANET networks.

AODV-ETX implementation
According to [11], the Expected Transmission Count (ETX) metric for link l, is: Therefore, we proposed a framework for evaluating KPI values and store them into files. This is done through creation of an extension of NS-3 simulator with the aim to support efficient network performance analysis. This framework produces statistically processed results for basic KPIs on application layer, regardless of network configuration on lower layers and without additional post-processing requirements. The KPIs that the proposed framework includes are: throughput, packet loss ratio, number of sent and received packets, overhead (useful traffic ratio) and E2E delay statistics (minimum, maximum, average, median values, jitter and delay histogram). The proposed solution provides simplicity of installation and usage as Flow Monitor, but also enables application layer statistics, support for all routing protocols, statistics for broadcast packets and many more KPIs that can be used in VANET evaluation. The source code of the developed framework is publicly available [55].
The developed framework consists of two parts ( Figure 3): applications for network traffic generation and reception and, software for statistical processing and calculation of basic KPIs.
The first part includes a pair of test applications, one for generating (source) and the other for receiving (sink) user data traffic. The source application is based on OnOffApplication that is frequently used by NS-3 application for network traffic generation. The main modification includes specific packet header, which is used to provide unique packet identification, calculation of packet loss ratio and E2E delay. Packet header contains four fields: packet sequence number, time when the packet is sent, and identifiers of source node and source application. Applications can use Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) traffic, and either IPv4 or IPv6 addresses. Both applications contain trace sources, which are triggered whenever application packet is sent and received.
The second software part consists of trace sinks, connected to application trace sources. Whenever a trace source is triggered, the connected trace sink is called. This feature is used for the collection and statistical data processing from the packet headers. The data stored in packet header are sufficient to provide unique identification of the received packet, as well as for the calculation of all basic KPIs. For overhead calculations we have used already available trace sources provided in WiFi model. properly. One more issue regarding the Flow Monitor is that it does not work for all routing protocols, such as DSR.
-Users can get data from the simulator using the tracing systems: ASCII tracing and Packet CAPture (PCAP) tracing [5][6]. ASCII trace system produces relatively large text data files of predefined events of interest and requires post-processing to obtain the performance metrics, usually using some scripting language [53]. PCAP trace system relies on packet analysis and also requires post-processing with external software, such as WireShark [54]. -General trace mechanism uses trace sources and trace sinks to export data from the simulator. Trace sources are embedded in NS-3 models and provide source of relevant information for users of the simulator. Trace sinks are implemented by NS-3 users in order to calculate appropriate performance metrics. -Data Collection Framework (DCF) is a tool for statistical processing of simulation data. DCF consists of three elements: probes, collectors and aggregators. Probes capture data from trace sources embedded into NS-3 models and provide output in suitable form for the collectors. Collectors consume data from the probes in order to calculate performance metrics and forward these data to aggregators. The aggregators are used to store data from the collectors into files or to generate graphics. While all of these mechanisms can be used for network analysis, there is no unified way to obtain the basic KPIs consistently. Flow Monitor provides some of them, but it cannot handle all protocols and packet broadcasting. ASCII and PCAP tracings are slow, create large files, and require post-processing. Both methods are very inefficient if multiple simulation iterations are needed to obtain the statistically significant results. DCF tool provides good concepts for statistical analysis, but it lacks in probe and collector types and design flexibility for storing different network performance metrics. The users are therefore left to create their own KPIs calculators based on general tracing mechanisms and/or DCF, which is not easy even for experienced NS-3 users. This leads to inconsistent and unreliable solutions that differ among researchers, and therefore it is more difficult to compare the simulation results.
Simulations are frequently repeated many times in order to obtain statistically significant results. In this case, it is inconvenient and time-consuming to handle a large number of created files for each simulation run. Therefore, this framework also includes generation of one summary file, which summarises all KPIs from all simulation runs into one file. This approach enables a very efficient presentation of the simulation results and consistent evaluation of VANET networks.

SIMULATION SETUP AND RESULTS
The second goal of our research is to use the developed framework to analyse VANET performance and propose optimal topology-based routing protocol for V2V communication in urban environments. In order to analyse the impact of traffic load, four case scenarios are considered, when the network consists of 50, 100, 200 and 300 vehicles. Vehicles are distributed over the city area of 2,000×2,000 m 2 . The mobility scenario is generated with the Manhattan grid model using BONN-MOTION tool [21] with the following parameters. The simulation area is divided into 3x3 blocks, the distance after which a new speed and direction are generated (update distance) is set to 5 m, the probabilities of speed and turn change are both set to 0.5, mean speed is 12 m/s, standard deviation of speed is 0.6 m/s and probability that the vehicle would stop after the update distance is 0. The mobility model is loaded in NS-3 using NS-2 The simulation scenario often requires that more than one node generates traffic. Using the proposed framework multiple source and sink applications can be installed in the desired nodes. Therefore, it can be useful to classify the packets according to network traffic flows. The network traffic flow is uniquely defined with source node ID, source application ID, sink node ID, and sink application ID. Statistical processing of acquired data is done according to network traffic flow. All data are then averaged over all flows (AAF statistics) and stored into files. However, in order to support statistics for broadcast packets we have also included statistical processing based on averaging all the packets (AAP).
There are three types of output files: vector, scalar, and summary file. In this software release, all data are recorded in the text files using a comma separator (CSV files). One vector file is produced for each simulation run and it contains time series data for every received packet: reception time, flow ID, sequence number, and delay. One scalar file is also produced at the end of each simulation run and this file contains statistically processed data for every packet flow: throughput, packet loss ratio, E2E delay (minimum, maximum, median and average) and jitter. Results are available not only for every individual flow, but are also averaged for all flows and for all packets. At the end of this file a delay histogram of all packets is included. Generating of both of these files can be enabled or disabled by the user.  Figure 3 -Basic principle of operation of the proposed software framework with 50, 100, 200 and 300 vehicles. All four scenarios have shown similar results. It can be seen that not all KPIs are best for the same protocol, as expected. Throughput and PLR are best for OLSR protocol, but AODV protocol provides worse results. PLR for AODV protocol shows very high values of about 60 % for low vehicle density and slightly increases with the number of vehicles in the VANET. OLSR protocol shows constant and very low PLR of about 5%. DSR protocol performs a bit worse than OLSR with PLR below 10%. DSDV shows highly unstable results, with significant decrease in performance with increasing number of vehicles.
It is interesting to notice that advantages of proactive protocols in the terms of E2E delay and jitter cannot be applied to VANETs due to their frequent topology changes. UPR, median and maximum E2E delay as well as jitter of reactive protocols (AODV and DSR) have shown better results compared to both proactive protocols (OLSR and DSDV). The AODV protocol has the lowest median delay of 0.2-0.4 ms, while all other protocols show 2-3 times higher delays. Maximum delay is interesting for analysing the initial delay in a network, when a route has yet to be found. AODV and OLSR have shown trace files. On the Medium Access Control (MAC) sub-layer 802.11 p standard with 6 Mb/s throughput and 10 MHz bandwidth is used. The channels are modelled according to YansWiFiChannel with a two-ray ground propagation model. Packets are routed using four topology-based routing protocols available in NS-3 simulator AODV, DSR, DSDV and OLSR. UDP protocol is used on the transport layer. In both scenarios 10 nodes simultaneously generate Constant Bit Rate (CBR) traffic with fixed size packets of 64 Bytes and a bit rate of 2 kb/s. In order to obtain reliable results, the duration of each simulation is set to 600 s and the network initialization time is set to 10 s. The average and median values, as well as standard deviation of 10 uncorrelated measurement iterations are calculated and written in the summary output file, as described in the previous Section. Figure 4 shows the results of six basic KPIs: throughput, Packet Loss Ratio (PLR), overhead, median and maximum E2E delay, and jitter. Overhead is presented as a Useful Packet Ratio (UPR) that includes MAC/PHY and the routing overhead. The results are shown for all four network scenarios   Since AODV protocol has shown the best results in terms of E2E delay, but the worst in terms of throughput and PLR, we have analysed the possibility to improve AODV performance by introducing ETX metric in AODV protocol. As previously stated, NS-3 does not support the ETX metrics, so we have developed a model for ETX with AODV routing protocol in NS-3. Besides the Manhattan model, in this case, we also provide the simulation results for random waypoint model to be used as a reference point for comparison of AODV and AODV-ETX protocols ( Figure 5). It should be noted that parameters for speed and pause time used for the Manhattan model remained the same for random waypoint model as well.
It can be seen from Figure 5 that using the ETX metric indeed improves the performance of AODV protocol in terms of throughput and PLR, since it chooses the routes of higher quality. For sparse networks, PLR is decreased from about 60% to 8% for the Manhattan model, and from about 50% to 5% for the random waypoint model. This result is comparable to DSDV and OLSR protocols. Network reliability is significantly higher, while median E2E delay is slightly increased. However, since the ETX similar results of about 3 s for initial delay, but as the number of vehicles increases, AODV seems to show a slightly higher upward trend. With the increasing number of vehicles, the overhead for proactive protocols increases (UPR decreases), while with reactive protocols the overhead stays the same or even decreases. For example, AODV shows 5% less overhead than OLSR for sparse networks, and even more than 11% for dense networks. Therefore, reactive protocols have shown better performance due to their on-demand characteristics to determine a route. When comparing AODV and DSR, having in mind that control packets in DSR are larger than in AODV, it is not surprising that AODV has better UPR and E2E delay.
Based on the above, it could be concluded that in terms of overhead and delay, AODV protocol has shown best performances, while in terms of throughput and PLR, OLSR is the best. Therefore, depending on the traffic type and network density different protocol should be used. If reliable packet delivery or low density network is used, OLSR protocol would be suitable, but for delay of critical packets in dense networks, AODV protocol should be used. Most VANETs commonly operate in three distinct environments: highways, rural areas and city areas. The second goal of our research was to propose an efficient routing protocol for V2V communication in urban environments. In order to evaluate our proposal, the framework is used to conduct a series of simulations of VANET within different routing protocols. In simulations, we used the Manhattan model to illustrate the conditions of an urban environment. In addition, it is possible to use other mobility models or obtained from external microscopic vehicular traffic simulators by loading the corresponding trace files.
Simulation results have shown that AODV protocol has the greatest potential for usage in V2V communication in dense VANETs in the city, since it has 2-3 times lower delay than DSR, DSDV and OLSR protocols, and at least 5% lower overhead. But it has low reliability for packet delivery with PLR that exceeds 60%. Although other environments were not tested, it can be expected that the selection of the most suitable protocol for highways needs further research, especially considering very high speeds and connections to infrastructure nodes.
In order to improve AODV protocol performance, we have proposed the implementation of ETX metric in AODV protocol. Simulation results show that using ETX metric can improve the reliability for packet delivery of AODV protocol in V2V communications. Packet loss ratio is significantly reduced from 60% to only 8% for sparse network, but for dense network overhead is highly increased and the effect of enhancement via the ETX metric is reduced.
Focus of our future research will be to further improve both ETX metric and AODV protocol, making them more suitable for VANETs, and with the aim to decrease routing overhead in dense networks, while keeping better throughput and packet loss ratio. One of possible solutions is to reduce size of ETX field that is added to all types of control packets in AODV protocol. Further, ETX metric could be calculated only based on reverse probability, and therefore, the size of LPP packet will be reduced as well. As AODV protocol is concerned, one of the improvements can be to reduce number of RREQ packets generation based on network density. Nodes can, with some probability, refrain from rebroadcasting RREQ packets and thus reduce the overhead. If the network is dense, fewer RREQ metric uses additional periodic LPP control packets it shows much higher overhead than with the basic AODV protocol. Also, with a greater number of vehicles, the effect of choosing better routes is reduced, due to large overhead. Therefore, although ETX metrics shows significant improvement of AODV protocol, it is necessary to introduce further modifications in order to reduce overhead for dense networks.

CONCLUSION
The NS-3 is a very good tool for wireless network simulations that includes the basic elements for simulating VANETs such as: topology-based routing protocols, models of wireless channels that enable simulations of heterogeneous networks, a large selection of propagation models and, implementations of IEEE 802.11p and WAVE standards. The main drawback of NS-3 is the lack of a simple and consistent method for calculating network KPIs.
One of the goals of our research was to provide a simple framework for testing the performance of VANETs. Since in NS-3 there is no suitable solution for obtaining important KPIs for VANETs, the users have no other option than to create their own KPIs calculators, which leads to inconsistent and unreliable results that often differ among researchers. Therefore, we have proposed a framework that provides standardised and easy-to-use environment in which researchers can obtain fair comparisons of their results. Although the proposed framework is tested in a V2V communication in VANET city scenario, it can be also used with various other network topologies and scenarios. For instance, V2I simulations are easily implemented using stationary nodes as RSUs. However, additional source and sink applications must be installed in the RSU nodes to simulate communication with the vehicles. In our opinion, the proposed framework is a useful contribution to researchers who want to use NS-3 as a tool for evaluating new protocols and metrics. This work is an initial step towards creating a comprehensive solution for VANETs performance analysis. Future versions of the framework will include lacking routing protocols for VANETs for testing both V2V and V2I solutions. Support is also needed for various applications (such as safety, efficiency, infotainment applications), which have different QoS requirements, so that new solutions targeting specific VANETs applications can be tested.
should be generated, and vice versa. This way the routing overhead of AODV protocol will be reduced.