{"id":85,"date":"2022-04-29T19:56:24","date_gmt":"2022-04-29T19:56:24","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/?page_id=85"},"modified":"2022-12-21T01:06:12","modified_gmt":"2022-12-21T01:06:12","slug":"experiments-results","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2022team11\/experiments-results\/","title":{"rendered":"Experiments & Results"},"content":{"rendered":"\n

Menu<\/h2>\n\n\n\n

Fall 2022<\/a><\/h2>\n\n\n\n

Spring 2022<\/a><\/h2>\n\n\n\n

Fall 2022<\/a><\/h1>\n\n\n\n

To review, the challenges we face with Synthetic Dataset generation are as follows:<\/p>\n\n\n\n

  1. It is very time-consuming to build a dataset. <\/li>
  2. Multiview setting requires a lot of hacking to the game.<\/li>
  3. Domain transfer would be a problem<\/li><\/ol>\n\n\n\n

    With these challenges in mind, we pivoted to the TBD Pedestrian dataset by CMU, which is multi-view and provides camera matrices and 2D trajectory labels for every pedestrian in all 3 camera views. The dataset is time synchronized so we know the frame number, pedestrian ID, and position coordinates of every person across all views. <\/p>\n\n\n\n

    \"\"
    Proposed pipeline <\/figcaption><\/figure>\n\n\n\n

    The pipeline we designed for 3D trajectory prediction is as follows:<\/p>\n\n\n\n

    1. Perform 2D pose estimation with ViTPose on all 3 camera views<\/li>
    2. Use a Greedy approach to match the bounding boxes of each pedestrian across at least 2 camera views. <\/li>
    3. Create a dataset of 2D pose sequences for each pedestrian across 2 camera views.<\/li>
    4. Perform triangulation using the camera matrices and the 2D pose sequences to obtain the 3D pose ground truth for each pedestrian<\/li>
    5. Perform trajectory forecasting to predict the 3D trajectories of each pedestrian<\/li><\/ol>\n\n\n\n

      (I) 2D Pose Estimation with ViTPose<\/strong><\/h2>\n\n\n\n

      Results of ViTPose on TBD Dataset<\/h2>\n\n\n\n
      \"\"
      Results on View 1<\/figcaption><\/figure>\n\n\n\n
      \"\"
      Results on View 2<\/figcaption><\/figure>\n\n\n\n
      \"\"
      Results on View 3<\/figcaption><\/figure>\n\n\n\n

      (II) Bounding Box Matching<\/strong><\/h2>\n\n\n\n

      Now that we have obtained the bounding boxes and 2D pose key points for each pedestrian, we need to match them across camera views. We need 2D pose sequences for each pedestrian across camera views so that we can triangulate them to obtain 3D pose key points. Since 2 views are sufficient for triangulation, we perform bounding box matching for View 1 and View 2. <\/p>\n\n\n\n

      To match pedestrians across views, we use the following greedy approach:<\/p>\n\n\n\n

      1. for each pedestrian with a trajectory label and pedestrian ID, for the first frame of that pedestrian ID, we find the closest bounding box to the trajectory point. <\/li>
      2. for subsequent frames, we track that bounding box, by finding the bounding box with the highest IOU with the previous frame<\/li><\/ol>\n\n\n\n
        \"\"
        Matching 2D trajectory label to bounding box in View 1<\/figcaption><\/figure>\n\n\n\n
        \"\"
        Matching 2D trajectory label to bounding box in View 2<\/figcaption><\/figure>\n\n\n\n

        We now have a sequence of 2D pose trajectories for each pedestrian in both views, on which we can now perform triangulation to obtain 3D pose information.<\/p>\n\n\n\n

        (III) Triangulation<\/strong><\/h2>\n\n\n\n

        We use 2D pose sequences from view 1 and view 2, and the camera matrices for triangulation to obtain 3D ground truth pose sequences for each pedestrian.<\/p>\n\n\n\n

        \"\"
        Approach for Triangulation<\/figcaption><\/figure>\n\n\n\n
        \"\"
        Perform triangulation to obtain 3D pose ground truth information<\/figcaption><\/figure>\n\n\n\n

        However, the triangulated 3D joints are noisy due to the downstream errors from both pose estimation and bounding box matching. Hence to obtain a smoother trajectory with less jitter we apply smoothening on skeleton gravity, limb length, and limb angles.<\/p>\n\n\n\n

        \n
        \"\"
        Before smoothing<\/figcaption><\/figure>\n\n\n\n
        \"\"
        After smoothing<\/figcaption><\/figure>\n<\/figure>\n\n\n\n

        (IV) Trajectory Forecasting<\/strong><\/h2>\n\n\n\n

        We take 2 approaches to trajectory forecasting:<\/p>\n\n\n\n

        1. Predicting 3D point trajectory <\/li><\/ol>\n\n\n\n
          \"\"
          Pipeline for prediction 3D point trajectory<\/figcaption><\/figure>\n\n\n\n

          Result<\/strong><\/h2>\n\n\n\n
          \n
          \"\"
          Input 3D sequence<\/figcaption><\/figure>\n\n\n\n
          \"\"
          Output 3D point trajectory<\/figcaption><\/figure>\n<\/figure>\n\n\n\n

          2. Predicting 3D skeleton and trajectory<\/p>\n\n\n\n

          \"\"
          Pipeline for prediction 3D skeleton trajectory<\/figcaption><\/figure>\n\n\n\n

          Result<\/strong><\/h2>\n\n\n\n
          \n
          \"\"
          Input 3D sequence<\/figcaption><\/figure>\n\n\n\n
          \"\"
          Output 3D skeleton and trajectory prediction<\/figcaption><\/figure>\n<\/figure>\n\n\n\n

          Spring 2022<\/a><\/h1>\n\n\n\n

          2D Pose Estimation<\/h2>\n\n\n\n

          We experimented extensively with Openpose in both real-world and simulation settings. For the real-world data, we experimented on the Shibuya crossing live video – a traffic intersection that streams a live video feed 24\/7. For the simulation setting, we set up the JTA Dataset Mods [2] to hook on to the GTA 5 game and collect the dataset. <\/p>\n\n\n\n

          Experiments and Results on Shibuya videos<\/span><\/strong><\/p>\n\n\n\n

          When we naively apply openpose on the test video we get the below result:<\/p>\n\n\n\n

          Directly applying Openpose on test video<\/figcaption><\/figure>\n\n\n\n
          \"\"<\/figure>\n\n\n\n

          But since we know there are people mainly in the bottom left, and top right portion of the frame when people are waiting and in the middle when people are crossing the intersection, we can crop the video to these sections of the video and run Openpose. The results are as follows:<\/p>\n\n\n\n

          \"\"
          One solution: Crop and resize<\/figcaption><\/figure>\n\n\n\n
          Bottom left crop<\/figcaption><\/figure>\n\n\n\n