A METHOD FOR AUTOMATIC IMAGE RECTIFICATION AND STITCHING FOR VEHICLE YAW MARKS TRAJECTORY ESTIMATION

The aim of this study has been to propose a new method for automatic rectification and stitching of the images taken on the accident site. The proposed method does not require any measurements to be performed on the accident site and thus it is frsjebalaee of measurement errors. The experimental investigation was performed in order to compare the vehicle trajectory estimation according to the yaw marks in the stitched image and the trajectory, reconstructed using the GPS data. The overall mean error of the trajectory reconstruction, produced by the method proposed in this paper was 0.086 m. It was only 0.18% comparing to the whole trajectory length.


INTRODUCTION
The traffic in horizontal road curves is always a source of higher risk, which is being reduced by the visibility improvements, the reduction of the maximum allowed driving speed, etc. [1,2].In case of the sideslip of the vehicle, it is always important to find the causes of the accident.The causes of the sideslip usually lie in the speed of the vehicle and the road pavement.When the tire is at an angle to the path of travel, the lateral forces occur at the contact point to road pavement.As a result, the tire leaves yaw marks on the road pavement when sideslip angle exceeds its maximum value at the currently acting normal force.Road roughness could have a great influence on the basic vehicle dynamics [3] as well as on the properties of the left tire marks.
There are several methods proposed for prediction and real-time control of the vehicle trajectory [4].The dynamic parameters evaluation models are adjusted to operate vehicle moving conditions in real time by using data from acceleration and yaw rate sensors [5,6].However, in case of a traffic accident the reconstructions are mostly based on the references of the post evidences [7].
The basic physics of Coulomb friction and circular movement underlie the formula of critical speed calculation, and various disputes arise due to the inaccuracy of measured parameters, which leads to false speed calculations during the accident reconstruction procedure [8].In order to estimate the critical speed of the vehicle, the two parameters, namely the radius of the tire mark curvature and the road friction coefficient should be found.The accuracy of the measurements performed in order to find the tire mark curvature and the road friction predetermines the results of the accident reconstruction procedure [9].
Road accident reconstruction is usually made by taking a set of pictures of the road pavement and the yaw marks.The received images should be properly rectified and matched to reconstruct the whole trajectory of the sideslip.It is crucial that the trajectory of tire marks is detected accurately.Moreover, the precise position of the trajectory in respect to the road elements is information important for forensic engineers as well.This is one of the reasons why the image processing methods are being included into traffic accident investigations [10,11,12,13].
PC-Rect, PhotoModeler or alternative image rectification software is usually applied to perform the projective image transform and matching in order to The novelty of the proposed method relies on the image rectification algorithm, specialized for the images with yaw marks on the road pavement.The projection matrix coefficients, initially estimated according to the automatically matched feature points, are additionally updated using the ellipsis matching on the marker and pavement pattern matching in successive images.Unlike the currently used image rectification software, the proposed method does not require any measurements on the accident site and does not require manual marking of the feature points in successive images.
The experimental research was performed to compare the vehicle sideslip trajectory, estimated by applying the proposed method and the trajectory estimated using GPS measurements during the experiment.

MATERIALS AND METHODS
Two different physical markers are proposed to be used while taking the pictures of tire marks (see Figure 1).The markers are square, have the same size and colour but differ with the number of black filled circles printed on the marker, one and two, respectively.
The two proposed markers are different in order to avoid possible image reordering mistakes during the image sorting.Markers play an important role in compensating the perspective distortions in the image.Considering that the image cannot be taken parallel to the road (to cover a larger area of the accident field), the proposed markers are easily detectable in the image by the use of automatic image analysis algorithms and are suitable for feature points extraction and mapping in consequent images.
Three main stages describe the proposed method of sideslip trajectory estimation from images: Individual Image Analysis, Image Alignment and Stitching, Trajectory Extraction.Each stage is additionally divided into two, three and two steps, respectively (see receive the image of the whole trajectory with visible yaw marks on the pavement [14,15].A set of additional measurements have to be performed on the site of an accident in order to have the actual measures of the distances between the objects visible in the image taken on the accident site.On a highly loaded roads or highways, the time for tire marks analysis and precise measurement of the environment is limited.Moreover, manual measurement often leads to some errors.
The 20-30 images are usually needed for the reconstruction of the complete visible sideslip trajectory.In order to stitch these images into one image of the continuous trajectory, the images should be rectified according to the estimated feature points.In optimal case, the pictures made by a camera should have the same scale factor and should be taken by positioning the camera parallel to the road.Otherwise, the geometrical rectification of each image should be performed in order to align all images into a single image for trajectory estimation.The feature points should be manually matched for each pair of images by using of PC-Rect or alternative software.
The aim of this paper is to propose a method for sideslip trajectory estimation from a set of images without the need to perform any measurements on the accident site and an algorithm for automatic matching and stitching of these images.
The authors propose to use two types of physical markers during image capture and image processing pipeline to perform a rectification of initial images.In order to take the whole trajectory of the sideslip, a set of pictures should be made to collect visible marks left by tires on the pavement.Practically it is not possible to ensure the constant position of the camera with respect to the pavement.Therefore, the tire straight skid mark segmentation and slope angle detection based on line searching image-processing technique is applied to solve problems associated with manual measurement [16,17].The first step in the analysis of the tire mark images requires feature point extraction algorithm in order to stitch the current image with the next one (a consequent image).Usually the image stitching requires a set of matched points.The user usually selects these points manually.In automatic stitching algorithms, a set of feature points is extracted and additional matching algorithms are applied.However, the number of extracted feature points vary from image to image and the point matching algorithm performance highly depends on the image contents.Since the physical markers are used in our proposed method, the algorithm does not depend on the image content (e.g.pavement texture details) and number of extracted and matched feature points remains constant.Each marker is a rectangle for which the corner coordinates Image rectification is performed in the second step of the individual image analysis part.The rectification of each image in the analysis set is performed according to the marker image by estimating the camera projection matrix: c Hc where H is the homogeneous camera projection matrix, c marker are the coordinates of the marker in the image, c template are the coordinates of the marker in the template image.
Because of the projection distortions, the circles in the marker appear as ellipsis in the image.Therefore, the contours of the ellipsis x marker = {x x1 , x y1 , x z1 , … , x xM , X yM , X zM } are extracted in the rectified image and used as the additional features to minimize the rectification error by applying the iterative Levenberg-Marquardt optimization.As a cost function, we selected the difference between major and minor axes of the ellipsis, fitted on the contour of the marker circle in the rectified image.During the optimization, the values of the H matrix are changed in order to minimize the cost function.The goal is to achieve the circles (instead of ellipses) in every image with the same radius (keeping the same aspect ratio of the markers in all images).

b) Image Alignment and Stitching
The goal of the second stage in the proposed method is to align the consequent image pairs and finally get the single stitched image with the whole sideslip trajectory available for analysis.Three processing steps are performed in this stage: coarse image alignment, precise image alignment and image stitching.The first step uses the corner coordinates of the physical markers (1.1 step in Figure 2).A camera projection matrix is estimated by using at least four pairs of matched points in the consequent images.The received camera projection matrix is used as the basis for step 2.2 of the method.
The precision of the marker corner coordinates estimation is limited.Additionally, we introduce a precise image alignment procedure used to update the initially estimated coefficients h of the camera projection matrix H.During this procedure, the patterns of road pavement are used to get the additional feature points for matching between two images.For the pattern matching, a set of feature points, based on SURF features, are detected for each image pair.Next, the feature matching is performed in order to receive a set of matching feature points.The outliers from the matched point set are removed by application of the M-estimator SAmple Consensus (MSAC) algorithm, which is a variant of the classical Random Sample Consensus (RANSAC) algorithm.
The coefficients of the projection matrix are updated in order to get better stitching according to the marker corners and the texture feature points.

c) Trajectory Extraction
The third stage of the proposed method is used to manually set the control points on the visible yaw marks in the image.In order to estimate the sideslip trajectory, the pixel coordinates are transformed into metres according to real scale of picture and initially measured dimensions of the physical marker.An expert should manually set the control points for the trajectory in each image after rectification and stitching.An experimental investigation showed several possible reasons why the automatic detection of control points on the visible yaw marks should not be used.There could be additional yaw marks still present on the pavement that were left by other vehicles (not those left by the accident party).In addition, the pavement may have different texture with different intensity of left yaw marks which the automated image analysis methods are not able to detect or classify into the same trajectory.In addition, the yaw marks left by two or more tires could overlap in the image and automatic detection of the edge of yaw mark becomes a very complicated task even for an expert.
In general, in order to estimate the trajectory using the method proposed in this paper, the following is needed: two different markers of known size, images taken by placing the markers on the pavement in the accident field and manual setting of the control points in the stitched trajectory image.
An experimental investigation was performed using the front wheel drive car (Toyota Avensis) upon the dry asphalt road conditions.The hard step steering wheel input manoeuver was performed and various driving speeds were used in each test to get different sideslip trajectories.No braking and further steering actions were taken during the vehicle slip.
To reconstruct the vehicle sideslip trajectory, a set of images were taken by the use of a regular camera (Samsung ES90; 14 megapixel; focal length 9 mm; ISO-80; T=1/291 sec.).To take pictures of the yaw marks for each trajectory, a set of images was taken by placing physical markers on the pavement in the following order: a marker with one black circle was placed on the right side of the camera view and the marker with two black circles -on the left side (see Figure 1).Next, the first marker was moved to the left, without moving the second marker on the pavement, until the second marker appeared on the right side of the camera view (see Figure 2).For the third image, the second marker (with two black circles) was moved to the left and the procedure was repeated until the yaw marks were not visible anymore.
The method proposed in this paper was applied for all collected pictures received after five performed experiments.The resulting image for one set of pictures is shown in Figure 3.
In order to compare the precision of trajectory estimation by the proposed automatic field of accident reconstruction method, the Race Technology Global Positioning System (GPS) module with data logger sampled at 20 Hz was used.GPS data are usually used to validate the dynamics of the vehicle [18].These GPS-based measurements of the sideslip trajectory were used in our experiment in order to perform the comparative analysis of the trajectory estimation using the proposed vision-based method and the trajectory estimation using GPS.The visual comparison of the trajectory, estimated by the proposed method with the trajectory measured by GPS module is given in Figure 4.
The solid curve received after approximation of the points measured by GPS (Figure 4) denotes the vehicle movement trajectory, which was estimated by the use of the GPS module.The dashed curve shows the trajectory, which was obtained by the use of our proposed vision-based sideslip trajectory reconstruction.The reconstruction was performed by approximation of the manually marked control points on a stitched image with yaw marks.In addition, we estimated the distances of the sideslip at discrete step, marked with the black vertical line markers.These measurements were performed at every metre in the reconstructed trajectory.
The error of the trajectory reconstruction was measured as the mean absolute error of the distances between the trajectory estimated using GPS and the

Reconstructed trajectory
where N is the number of data points (positions) p of GPS recorded trajectory; p are the (x, y) coordinates of GPS recorded positions; r is the point on the reconstructed trajectory (x, y), nearest to p position; d is the distance between trajectories used for comparison.Taking into account the fact that the trajectory of the wheel {p w1, p w2 , …, p wN } and the GPS module {p GPS1 , p GPS2 , …, p GPSN } may be slightly different, an additional component p u was added to the total uncertainty of the trajectory matching.Then the wheel position according to GPS module was expressed as follows: ) where p u is the value for transforming the coordinates of the vehicle to the coordinates of the wheel.

According to the scheme shown in
where a GPS,w, b GPS,w are the distances between the mounted GPS module and the wheel in vehicle lateral and longitudinal direction, respectively; } is the vehicle yaw angle, measured by the gyroscope sensor, mounted next to GPS module.The influence of the vehicle body roll and pitch angles (cornering with braking) on uncertainty estimation was not taken into account because of the small values of these oscillations.

RESULTS AND DISCUSSION
Five experimental tests were performed on the different dry asphalt roads.After each sideslip test, up to 30 images of the yaw marks visible on the pavement were taken putting the two different markers proposed in our method.The automatic rectification and stitching according to our proposed algorithm was applied for each image set.The trajectory was estimated by applying the approximation of the control points, manually marked by an expert on yaw marks, visible on the automatically stitched image.After the performed vision-based trajectory estimation for each test, the resulting trajectory was compared with GPS data and the trajectory reconstruction error was estimated.
The error of the trajectory estimation was calculated by using the assumption that the GPS-based estimation of the trajectory was correct.The mean absolute error of trajectory estimation at discrete trajectory points was calculated for each test.The summary of the results are given in Table 1.The highest mismatch was obtained at the edges (at the beginning and at the end) of the estimated trajectory.
One of the causes of the obtained trajectory mismatch was the placement of the GPS module during the experiment setup.In ideal case, the GPS module should be placed on the wheel that draws the yaw marks on the pavement.Unfortunately, there was no possibility to place the GPS module on the wheel.
The overall mean error of the trajectory reconstruction, produced by our method was 0.086 m.It is only 0.18% comparing to the whole trajectory length.The presented method enables to stitch separate images automatically and operate with all yaw mark trajectory for further analysis what was not offered by software of traffic accident analysis [19].The rectification and stitching of 20 images using PC-Rect, PhotoModeler or alternative image rectification software would require setting manually at least four points in each image (at least 80 points in total), while the method proposed in this paper performs rectification automatically.
Due to limited resolution of the images taken on the accident site and the image pre-processing applied in the camera (e.g. an embedded smoothing filter for image sensor noise removal), the exact manual pointing of the rectangular object corner in the image is practically not possible.The method proposed in this paper uses a unique combination of the corner detection by searching for the intersection of lines, matched on a contour of the marker, and the ellipsis contour matching to circle in order to improve the image rectification results.This specific method can be applied only if the proposed markers are used.However, the use of the markers eliminates the uncertainty in feature point matching, which is usually received when the images with pavement texture are matched.
One of the advantages of the presented method compared to methods usually used in forensic image analysis are the minimized requirements for image taking.No special requirements are necessary, except sufficient image quality, position of photographer, camera direction, distance to tire mark or taking pictures in vertical position [20].
Lambourn et al. proposed to use a laser-scanning device in order to identify the influence of the vehicle active safety systems on tire marks and speed calculation [21].Despite the expensiveness of such equipment, additional reflectors placed along the tire marks are necessary to determine the trajectory of the tire marks.Furthermore, after the determination of the radius from the arc of 10 m length, the vehicle speed calculation error reached up to 10%.Therefore, the alternative method of radius determination is required.
In summary, the method proposed in this paper comprises the fixation of an accident scene, the constitution of digital scene and accurate determination of the radius from the tire marks.However, the image processing pipeline of the presented method may be improved for better tire marks identification by increasing its contrast comparing to the neighbouring pavement, e.g. using the method proposed by Wang et al. [22].The wider application of the method proposed in this paper, e.g.determination of other evidences from accident scene [23] can be targeted as well.

CONCLUSION
This study was performed in order to present a vision-based vehicle sideslip trajectory estimation for traffic accident reconstruction.The proposed method requires only taking a set of pictures on the accident site by using two specific markers with known dimensions.The actual measurements on the accident site are not needed any more for image rectification if the proposed method is used and there is no risk of having measurement errors on the site.Two markers were introduced for coarse alignment and stitching of images taken on the accident site.Stages of Individual Image Analysis, Image Alignment and Stitching, Trajectory Extraction were introduced in the proposed method for full reconstruction of vehicle trajectory based on the yaw marks.The received vehicle slip trajectory will be used in the following research works for better estimation of critical speed calculation in traffic forensic activity.
In order to estimate the accuracy of the vision-based vehicle trajectory estimation the GPS based trajectory estimation was used.The results of the performed experimental investigation proves the ability to reconstruct the curve of the vehicle yaw marks by using a set of images taken on the road and using only two markers for each image.
It should be noted that the vision-based trajectory estimation is sensitive to the quality of the images, taken on the accident site.The images should be of high resolution (14 megapixel or higher) and taken in the daylight in order to reduce the image noise caused by an image sensor in the low light environment.
The average of the trajectory estimation mean error for five performed experiments was 0.086 m.It is only 0.18% on the average comparing to the trajectory length of the sideslip.
Since the GPS module was placed not on the wheel that draws the yaw marks on the pavement during the sideslip, but rather in the middle of the vehicle instead, the future investigation should be done in order to compare the vision-based trajectory estimation results with the actual trajectory.
In addition, the improvements of presented image rectification and stitching method should be done by reducing the required number of images taken on the accident site but maintaining the achieved accuracy.

Figure 1 -
Figure 1 -A representative pair of images used in the traffic accident reconstruction

Figure 3 -
Figure 3 -Visualization of reconstructed scene showing detected tire marks

Figure 5 ,
the coordinates of GPS recorded position transferred to wheel position are:

Figure 4 -Figure 5 -
Figure 4 -An illustration of the trajectory estimation results (the solid line shows the results recorded by GPS module, and the dashed line shows the trajectory estimation results using the method proposed in this paper) V.Žuraulis, etal.: A Method for Automatic Image Rectification and Stitching for Vehicle Yaw Marks Trajectory Estimation Main stages of the proposed trajectory estimation method based on tire mark image analysis V. Žuraulis, et al.: A Method for Automatic Image Rectification and Stitching for Vehicle Yaw Marks Trajectory Estimation

Table 1 -
Results of the experimental investigation