{"id":22,"date":"2023-05-04T15:51:43","date_gmt":"2023-05-04T15:51:43","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/f23team7\/?page_id=22"},"modified":"2023-05-06T22:12:22","modified_gmt":"2023-05-06T22:12:22","slug":"spring-2023","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/f23team7\/spring-2023\/","title":{"rendered":"Spring 2023"},"content":{"rendered":"\n

Overview<\/h2>\n\n\n\n

We reviewed relevant literature and decided to implement tracking-by-detection method in top-down view<\/strong> in order to efficiently aggregate multiple views. We trained two multi-view detectors (MVDet, MVDeTr<\/strong>) on MMPTrack dataset<\/strong>, and implemented SORT<\/strong> as our baseline tracking algorithm. Detected and tracked bounding boxes in top-down view are reprojected onto camera planes for final output.<\/p>\n\n\n\n

Presentation slides<\/a>\u00a0and\u00a0poster<\/a>\u00a0are publicly available.<\/p>\n\n\n\n

Literature Review<\/h2>\n\n\n\n

Paper 1: Real-Time <\/strong>Multiple People Tracking <\/strong>with Deeply Learned Candidate Selection <\/strong>and Person Re-Identification<\/strong><\/h3>\n\n\n\n

Previously mentioned tracking-by-detection framework has two main challenges: one is unreliable detection, and another one is occlusion which results in ambiguities in data association. Then the challenge is to associating unreliable detection results with existing tracks in real-time, or in other words only using current and past frames.<\/p>\n\n\n\n

Authors propose three novel ideas to tackle this challenge. First, they suggest that detection and tracks can complement each other<\/strong> in different scenarios. Detection results of high confidence can prevent tracking drifts while prediction of tracks can handle noisy detection by occlusion. So, the authors propose to collect candidates from outputs of both detection and tracking <\/strong>while default framework used to collect only from detection results.<\/p>\n\n\n\n

In order to optimally select from a considerable amount of candidates in real-time, the authors present a novel scoring function<\/strong> that fuses an object classifier and a tracklet confidence<\/strong>. <\/p>\n\n\n

\n
\"\"
Chen, Long, et al. (2018) Real-time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-Identification
<\/figcaption><\/figure>\n<\/div>\n\n\n

Given an image frame, score maps of the entire image<\/strong> are predicted using a fully convolutional neural network with an encoder-decoder architecture. At the end there is a position-sensitive RoI pooling layer to explicitly encode spatial information<\/strong> into the score maps.<\/p>\n\n\n\n

This region-based fully convolutional neural network shares most computations on the entire image<\/strong>, which makes it much more efficient compared to classification on image patches (cropped from heavily overlapped candidate regions).<\/p>\n\n\n\n

For computing tracklet confidence<\/strong>, we estimate new location of existing track using Kalman filter<\/strong>. These predictions are adopted to handle detection failures caused by occlusion in crowded scenes and various visual properties of objects. Hence, tracklet confidence is designed to measure the accuracy of the filter using temporal information<\/strong>.<\/p>\n\n\n\n

Lastly, the authors adopt deeply learned appearance representation<\/strong> to improve the identification ability of the tracker. A deep neural network with convolutional backbone from GoogLeNet followed by part-aligned fully connected layers extracts feature vectors from RGB images.<\/p>\n\n\n\n

Tracks are hierarchically associated<\/strong> with different candidates of different features by first using appearance representation<\/strong>, and then based on IOUs<\/strong>, and finally new tracks are initialized for remaining detection results. This enables to avoid tracking other unwanted objects and backgrounds. For the hierarchical data association, we only need to extract ReID features for candidates from detections once per frame.<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n
\"\"
Chen, Long, et al. (2018) Real-time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-Identification<\/figcaption><\/figure>\n\n\n\n

Proposed method is much more time efficient by sharing computations on the entire image. Also, when evaluated on the same MOT16 dataset, proposed method achieves state-of-the-arts performance<\/strong> while being 5-20 times faster<\/strong> than most existing methods.<\/p>\n\n\n\n

Paper 2: Multi-view People Tracking <\/strong>via <\/strong> Hierarchical Trajectory Composition<\/strong><\/h3>\n\n\n\n

Different cues, like appearance and location, have been studied for cross-camera data association at that time (2016). However, the author notices that existing approaches usually rely on the effectiveness of particular cues, which is an over-strong assumption. As shown in the case below, the validity of different cues varies at different time and places.<\/p>\n\n\n\n

\"\"
Xu, Yuanlu, et al. Multi-view people tracking via hierarchical trajectory composition.<\/figcaption><\/figure>\n\n\n\n

The proposed method learns an optimal strategy for utilizing multiple cues. Specifically, it’s a hierarchical composition model<\/strong> with 3 composition criteria: appearance coherence<\/strong>, geometry proximity<\/strong>, and motion consistency<\/strong>. In the mode, graph nodes are tracklets<\/strong>, which refer to a sequence of states (appearance feature, location, timestamp, visibility). The composition of nodes will grow tracklets and finally form a complete trajectory<\/strong>, and the criterion will measure likelihood of such composition. <\/p>\n\n\n

\n
\"\"
Xu, Yuanlu, et al. Multi-view people tracking via hierarchical trajectory composition.<\/figcaption><\/figure>\n<\/div>\n\n\n

At inference time, the model will do an iterative greedy construction<\/strong> which reduces computational cost: First, enumerate to find two tracklets with max probability<\/strong> to be merged<\/strong> into a new one. Then re-estimate states<\/strong> for the new node. This process will repeat until max composition probability goes under a threshold, and each subtree connected to the root should be an object trajectory.<\/p>\n\n\n\n

Paper 3:  Multi-view Detection <\/strong>with Feature Perspective Transformation<\/strong><\/h3>\n\n\n\n

What Paper 1 and Paper 2 have in common is that they are based on object detection at each camera view and seek to aggregate information across cameras later. The downside of this kind of method is the detection accuracy will suffer from severe occlusion. Recent methods like Paper 3 are based on a different idea. The authors propose an end-to-end model for direct multi-view detection and produce results as an occupancy map in the top-down view.<\/p>\n\n\n\n

This kind of methods inspire us to build a tracker based on a multi-view detector. So a detailed description of a similar model can be found in our method<\/a> page. This early cross-camera association<\/strong> strategy gives us a solid detection result (~97% recall and ~99% precision) on the MMPTRACK dataset.<\/p>\n\n\n\n

\n\n\n\n

Baseline Results<\/h2>\n\n\n\n

We trained multi-view detector (MVDet, MVDeTr) on MMPTrack dataset for 10 epochs on a single NVIDIA GeForce GTX 1070. We used SORT algorithms as our baseline tracking algorithm. Tracked bounding boxes are projected from world grid to each camera views by setting a fixed z coordinate of 1.6m which translates to bounding box height.<\/p>\n\n\n\n

\n
\n
\"\"
MVDeTr<\/strong> performs better than MVDet.<\/figcaption><\/figure>\n<\/div><\/div>\n\n\n\n
\n
\"\"
Our method has multi-object tracking accuracy that is on par with algorithms of highest ranks in MMPTrack Challenge, but has frequent ID switches<\/strong> and low ratio of correctly identified detections.<\/figcaption><\/figure>\n<\/div><\/div>\n<\/div>\n\n\n\n

MVDet + SORT Visualization<\/h3>\n\n\n\n