Robust Pedestrian Detection Framework using Harris Corner Detector and Kalman Filter

Robust Pedestrian Detection Framework using Harris Corner Detector and Kalman Filter

Manisha Domale

Electronics Department Vishwakarma Institute of Technology Pune, India

Dr. Vijay Gaikwad

Electronics Department Vishwakarma Institute of Technology Pune, India

Abstract Detecting and tracking objects are among the most prevalent and challenging tasks that a surveillance system. Under the business intelligence notion, an object is a human, a queue of people, a crowd. In this paper we introduce the reader to main trends and provide taxonomy of popular methods to give an insight to underlying ideas as well as to show their limitations in the hopes of facilitating integration of object detection and tracking for more effective business oriented video analytics.

Conditioned on these results, we then address object interactions, tracking, and prediction during a second step. The approach is through an experiment evaluated on many long and tough video sequences from busy inner-city locations. Our results show that the projected integration makes it doable to deliver robust tracking performance in scenes of realistic complexness.

Keywords Multiple Pedestrian Tracking, Harries Detector, Kalman Filtering

1. INTRODUCTION

   Detection and tracking moving objects have many applications in the field of machine vision like video compression, monitoring systems, industrial control, and gesture-based computer interaction. Yilmaz et al. evaluated and classified moving object tracking methods [1]. According to their classification, tracking methods are divided into three categories: point-based tracking, kernel-based tracking, and Silhouette tracking. The point-based technique is further divided into two groups: deterministic and statistical. The kernel-based technique is additionally divided into two groups, that are pattern matching and classifier-use. The last one, Silhouette technique, uses the shape of objects and evaluate object contour methods.

   According to Yilmaz et al., moving object tracking methods in various areas are faced with issues such as overlapping moving objects, change in brightness, little background motion, lack of motion stability in background and camera moving. to fix any of these problems, we should seek appropriate solutions [1].

   According to Wu-Chih Hu et.al Moving object detection is relatively difficult for video captured by a moving camera since camera motion and object motion are mixed. During this technique, the feature points within the frames are found and then classified as belonging to foreground or background features. Next, moving object regions are obtained exploitation Associate in Nursing integration theme

   supported foreground feature points and foreground regions, which are obtained using an image difference scheme. [2] Slim et al. [3] projected a method for tracking pedestrians with a mobile camera using a color histogram. The problem of pedestrians overlapping was eliminated with considering bar chart of peoples head in the overlapping region. Lee et al. [3] introduced a technique for pursuit a mobile automaton by another mobile robot; in their method, for tracking the target robot by the tracker, they set up a angle of view of mobile tracking camera for tracking desired position and target robot by means of position information and motion information regarding each robots. However, in their method, tracking could face problems like barriers in front of the robot tracker camera, overlapping target robot and its disappearance. Yokoyama [4] used contour-based object technique for tracking and feature a gradient method for detecting and tracking objects supported optical flow and edge. Zhan et al. [5] used background difference method for moving the object supported background update model.

   In this paper, a robust and fast yet simple technique for detecting and tracking moving objects is given. This technique is based on following mobility edge through fixed edges. The results show that the proposed technique, any to its efficiency, will overcome challenges such as brightness variations and background changes over time.

2. FEATURE POINT CLASSIFICATION

   The proposed method has three main parts, namely the classification of feature points, the moving object detection, and the moving object tracking. An algorithm flowchart is shown in fig.1. To detect pedestrians in the wide area, video sequences Harries corner detection and are used as appearance based methods and Gaussian mixture model (GMM) was used to extract moving pedestrians. Like our approach, we combined the Harris corner detectors at the decision level to give a new detector. These algorithms are discussed in details in the following subsections.

   1. GAUSSIAN MIXTURE MODEL

      An adaptive online background mixture model that can robustly deal with lighting changes, repetitive motions, clutter, introducing or removing objects from the scene and slowly moving objects[17]. Their motivation was that a unimodal background model couldn't handle image acquisition noise, light change and multiple surfaces for a particular pixel at the same time [18].

      Moving object detection

      Input Video frames

      Foreground feature point Background feature points

      Find feature point

      foreground feature

      Feature point matching

      Update scheme

      Foreground feature point

      Calculate information of object region

      Feature point classification

      Moving object tracking

      Kalman filter for moving object detetion

      Output Result

      Fig.1 Flowchart of proposed method

      Thus, we used a mixture of Gaussian distributions to represent every pixel in the model. Due to its promising features, we implemented and integrated this model in our visual surveillance system. in this method, the values of an single pixel (e. g. Scalars for gray values or vectors for color images) over time is considered as a pixel process and the recent history of each pixel, , is modeled by a mixture of K Gaussian distributions. The probability of observing a current pixel value then becomes:

      k

      P( Xt) wi,t *( Xt, i,t, i,t)

   2. HARRIES CORNER DETECTOR

      The Harris corner tracker is based on an assumption that corners are associated with maxima of the local autocorrelation function. It is less sensitive to noise within the image than most other algorithms, because the computations are based entirely on first derivatives [14]. The algorithm is popular due to its high reliability in finding junctions and its sensible temporal stability, creating it a lovely corner detector for pursuit [16]. It has to be noted that as a result of these algorithms rely on spatial derivatives, image smoothing is commonly required to enhance their performance. While up

      i1

      (1)

      the detection dependableness, it's been shown that smoothing

      where wi,t is an estimate of the weight of the ith Gaussian (Gi,t) in the mixture at time t, i,t is the mean value of Gi,t and i,t is the variance matrix of Gi,t is a Gaussian probability density function. The procedure for detecting foreground pixels is as follows. At the starting of the system, the K Gaussian distributions for a pixel are initialized with predefined mean, high variance and low previous weight. Once a new pixel is observed within the image sequence, to determine its type, its RGB vector is observed against the K Gaussians, until a match is found.

      could end in poor localization accuracy. Shifting the window in any direction should large change in appearance; Harris corner detector offers a mathematical approach for decding that case holds: flat, edge or corner; Let gradient vectors as a set of (dx,dy) points with a middle of mass outlined as being at (0,0); work an ellipse to that set of points via scatter matrix. For small shift [u,v] we have a bilinear approximation:

      v

      E(u, v) [u, v]M u

      (2)

      1. (b)

   Fig.2. Results obtained using Harries Corner Detector (a) Enter exit frame 133 (b) library1 frame 90

   Where M is 2*2 matrix computed from image derivatives:

   Measure of corner response:

   R det M k(traceM )2

   (5)

   I 2 I I

   M w(x, y)

   x x y

   I I I 2

   x, y

   x y y

   (3)

   det M 12

   Change of intensity for the shift [u,v]:

   E(u, v) w(x, y)[I (x u, y v) I (x, y)]2

   x, y

   (4)

   traceM 1 2

   (k is empirically determined constant: k-0.04-0.06)

   1. (b)

   Fig.3. Results obtained for Oneshop frame 53(a) Background subtraction (b) Person detection

3. MULTIPLE PERSON DETCTION AND TRACKING

   The difference between a typical Kalman filter and the enhanced Kalman filter are highlighted by occlusion rate calculation which making the tracking during occlusion is capable and the involvement of occlusion scene determination. [18].

   Besides that, object recognition capability is added to the enhanced Kalman filter so that the tracking target can be

   The Kalman filter has two distinctive features. One is that its mathematical model is described regarding state-space concepts. Another is that the solution is computed recursively. Usually, the Kalman filter is described by system state model and measurement model. The state-space model is described as a system state model and analysis model as shown in (6) and (7) respectively.

   recognized by others. This feature is running all the time to assure that the targeted object will be acknowledged all the time.

   s(t) O(t 1)s(t 1) w(t)

   z(t) H (t)s(t) v(t)

   (6)

   (7)

   1. (b)

      Fig. 4. Results obtained for Enterexit frame 100(a) Foreground detection (b) Person detection

      (a) (b)

      Fig.5. Results obtained for Library 2 frame 320 (a) Foreground detection (b) Person tracking

      Where O(t -1) and H(t) are the state transition matrix and measurement matrix . The w(t) and v(t) are white Gaussian noise with zero means.

      Kalman filter has two phases: prediction step and correction step. The prediction step is cause for projecting forward the current state, obtaining a prior estimate of the state s (t).The work of the correction step is for the feedback. It incorporates an actual measurement into the previous estimate to obtain an improved posterior estimate s+ (t), which is written as shown in (8).

      Step 3: Calculate the gap Di between the tracked person and each moving person. Calculate the difference of the Minimum

      Bounding box ri between the tracked object and each moving person, find the smallest Di and ri, and this object isn't tracked, then its location is that the new location of the tracked object, go to Step five.

      Step 4: Apply prediction tracking to the moving object using the Kalman filter, find the location and so go to Step five.

      S (t) S (t) k(t)[z(t) H (t)S (t)]

      (8)

      Step 5: Predict the location of each moving person using the

      The algorithm of the proposed tracking scheme of multiple person detections is given below.

      Step 1: find the width and height of every tracked person. If there's no tracked person, then go to Step five.

      Step 2: tracked person is searched for among the search region SR. Four conditions are checked and dealt with.

      Search region is defined as:

      (9)

      Where W, H are the width and height of minimum bounding box of moving person.

      1. If there are multiple monitored people within the search area SR, then go to Step three.

      2. If there's no tracked person within the search space SR, then head to Step four.

      3. If there's just one moving person and it's not half-tracked. Within the search region SR, then find width, and height and so go to Step five.

      4. If there is just one moving person and it's tracked in the search region SR, then there are multiple overlapping objects. Go to Step five.

         Kalman filter to find the width, and height of every tracked object. If the retention time of the tracked object is larger than the given threshold TR, the data and Kalman filter of the tracked object are removed. Go to Step one. In the planned algorithm for multiple moving object tracking, the given threshold of the retention time TR = 0:1s is set from experience. If a new moving object is added, it's assigned as a candidate tracked object. If the time of incidence of the candidate tracked object is larger than 0:5s, the Kalman filter is used for the prediction trailing. The proposed algorithm will track overlapping objects and objects whose scale changes (object is moving toward or away from the camera).

4. EXPERIMENTAL RESULTS Experiments are done on a computer with an Intel Core i5

   2.3 GHz CPU and 4 GB of RAM. The algorithms were implemented in Matlab13.

   1. Performance evaluation of self made dataset

      In order to illustrate the performance of person detection and tracking, the true detection rate TR and the false detection

      rate FR were adopted as defined in Eqs. (1) And (2), respectively, where N is the total number of moving objects, TP is the total number of true detection objects, and FN is the total number of false detection objects. Ts The results of the performance explain for the self-made dataset are shown in

      N

      TR TP *100%

      (9)

      table 2. Experimental results show that the proposed technique performs better than other method. The proposed method has good performance in terms of true detection rate and false detection rate. So, the proposed technique greatly increases the performance of person tracking.

      TABLE III.

      Sequence	Enter exit		One shop		TUD stand
      Techniques	R	P	R	P	R	P
      ACF[9]	0.58	0.76	0.32	0.48	–	–
      DPM[9]	0.74	0.89	0.63	0.90	0.77 [10]	0.97 [10]
      HOG/ SVM[9]	0.62	0.77	0.32	0.48	–	–
      KSP[7]	–	–	–	–	0.63	0.79
      Online CRF[8]	–	–	–	–	0.85	0.86
      GMCP[11]	–	–	–	–	0.81	0.95
      Proposed Method	0.93	0.84	1	0.52	0.93	0.84

      TP FN *100%

      TP FN

      (10)

      TABLE I.

      Sequence	Resolution	Frames
      Library 1	1280 x 720	822
      Library 2	1280 x 720	880
      Library 3	1280 x 720	1475

      Sequence	N	TP	FN	TR%	FR%
      Library 1	822	1166	0	90.24	0
      Library 2	880	1041	0	86.96	0
      Library 3	1475	1359	70	70.41	4.89

      Fig.7. Description of self made dataset TABLE II.

      Fig.7. Performance evaluation of self made dataset

   2. Performance evaluation of public dataset

   A public dataset that includes three video sequences (Enter exit, one shop, TUD stadtmitte) was used to evaluate the performance of proposed method.

   In order to illustrate the performance of public dataset, Precision (P) and Recall (R) were adopted as defined in Eqs. 1112, respectively, where TP is the total number of true positive pixels, FP is the total number of false positive pixels, and FN is the total number of false negative pixels.

   Fig.8 Performance evaluation and comparison of public dataset

5. CONCLUSION

The method of multiple pedestrian detection in video

P TP

TP FP

R TP

TP FN

(11)

(12)

sequences is proposed in this paper. Feature points in each

and every frame are detected and then classified as foreground and background features. We have used harries corner detector for

Extracting features from frames. Foreground feature points

This paper proposed a novel method capable of detecting and tracking people in cluttered real-world scenes with many people and changing backgrounds. Kalman filter improve people-detection by people-tracklet detection in image sequences. Those tracklets are then used to enable people tracking in complex scenes with many people and long-term occlusions. In the future we will extend the proposed approach using a 3D limb model to allow people-detection from arbitrary viewpoints and across multiple cameras.

then used to detect the pedestrian in the frame. These foreground feature points are then updated for every frame. Then foreground is extracted by background subtraction algorithm in combination with harries feature point detector to have better results. Moving pedestrian is obtained from above method is then tracked by Kalman Filter which will track the pedestrian with minimum bounding box.

We have tested our method on various available public datasets. Experimental results show that the proposed method gives better result. Therefore the proposed method is useful for surveillance and other pedestrian related applications.

1. (b)

(c) (d)

Fig.6. Bar chart (%) of performance evaluation of public dataset (a)Enter Exit, (b) One shop, (c)TUD Stadtmitte, (d)Performance evaluation

REFERENCES

1. A. Yilmaz, O. Javed and M. Shah Object tracking: a survey, ACM Computing Surveys, vol. 38, no. 4, pp. 1-45, 2006.

2. J.S. Lim and W.H. Kim, Detection and tracking multiple pedestrians from a moving camera, International Symposium on Visual Computing, pp. 527-534, 2005.

3. C. Lee, Vision tracking of a moving robot from a second moving robot using both relative and absolute position referencing methods, 37th Annual Conference on IEEE Industrial Electronics Society, pp. 325-330, 2011.

4. M. Yokoyama, A contour-based moving object detection and tracking, Joint IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, pp. 271-276, 2005.

5. R. Zhang Object tracking and detecting based on adaptive background subtraction, International Workshop on Information and Electronics Engineering, pp.1351-1355, 2012.

6. C. Stauffer and W.E.L. Grimson, Learning patterns of activity using real time tracking, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 747-757, 2000.

7. J. Berclaz, F. Fleuret, E. T¨ uretken, and P. Fua. Multiple objects tracking using k-shortest paths optimization. IEEE Trans. PAMI, 2011.

8. B. Yang and R. Nevatia. Multi-target tracking by online learning a crf model of appearance and motion patterns. InIJCV, 2014.

9. Elie Moussy,et.al A Comparative View on Exemplar 'Tracking- by-Detection' Approaches Approaches. IEEE International Conference on Advanced Video- and Signal-based Surveillance (AVSS), Aug 2015, Karlsruhe, Germany.

10. Sahar Rahmatian, et.al Online multiple people tracking-by- detection in crowded scenes Journal of Advances in Computer Engineering and Technology, 1(2) 2015.

11. Amir Roshan Zamir,et.al GMCP-Tracker: Global Multi-object Tracking Using Generalized Minimum Clique Graphs UCF Computer Vision Lab, Orlando, FL 32816, USA.

12. Aziz Karamiani et.al Detecting and Tracking Moving Objects in Video Sequences using Moving Edge Features Scientific Co- operations International Workshops on Electrical and Computer Engineering Subfields 22-23 August 2014, Koc University, ISTANBUL/TURKEY.

13. Kenji Okuma et.al A Boosted Particle Filter:Multitarget Detection and Tracking University of British Columbia, Vancouver B.C V6T 1Z4, CANADA.

14. C. Harris and M. Stephens, A combined corner and edge detector, in Proc. 4th Alvey Vision Conf., Manchester, U.K., 1988, pp. 147151.

15. Wongun Choi et.al Detecting and Tracking People using an RGB-D Camera via Multiple Detector Fusion Willow Garage, Menlo Park, CA, USA.

16. Peter I. Rockett IEEE transactions on image processing, vol. 12, no. 12, december 2003 Performance Assessment of Feature Detection Algorithms: A Methodology andCase Study on Corner Detectors.

17. T. Bouwmans, F. El Baf, B. Background Modeling using Mixture of Gaussians for Foreground Detection РA Survey Vachon Laboratoire MIA, Universit̩ de La Rochelle, Avenue M. Cr̩peau, 17000 La Rochelle, France.

18. C Stauffer, W Grimson, Adaptive background mixture models for real – time tracking. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1999, 2(6) 248 252.

19. Tracking Human Position and Lower Body Parts Using Kalman and Particle Filters Constrained by Human Biomechanics Jesus Mart´nez del Rinc´on, Dimitrios Makris, Member, IEEE, Carlos Orrite, Member, IEEE and Jean-Christophe Nebel, Senior Member, IEEE.