How to Visualize 2d Continuously Moving Object Python

In this tutorial, we will learn Object tracking using OpenCV. A tracking API that was introduced in OpenCV 3.0. We will learn how and when to use the 8 different trackers available in OpenCV 4.2 — BOOSTING, MIL, KCF, TLD, MEDIANFLOW, GOTURN, MOSSE, and CSRT. We will also learn the general theory behind modern tracking algorithms.

This problem has been perfectly solved by my friend Boris Babenko as shown in this flawless real-time face tracker below! Jokes aside, the animation demonstrates what we want from an ideal object tracker — speed, accuracy, and robustness to occlusion.

This post has been tested on OpenCV 4.2.

Demo of Object tracking using OpenCV

If you do not have the time to read the entire post, just watch this video and learn the usage in this section. But if you really want to learn about object tracking, read on.

What is Object Tracking?

Simply put, locating an object in successive frames of a video is called tracking.

The definition sounds straight forward but in computer vision and machine learning, tracking is a very broad term that encompasses conceptually similar but technically different ideas. For example, all the following different but related ideas are generally studied under Object Tracking

1. Dense Optical flow: These algorithms help estimate the motion vector of every pixel in a video frame.
2. Sparse optical flow: These algorithms, like the Kanade-Lucas-Tomashi (KLT) feature tracker, track the location of a few feature points in an image.
3. Kalman Filtering: A very popular signal processing algorithm used to predict the location of a moving object based on prior motion information. One of the early applications of this algorithm was missile guidance! Also as mentioned here, "the on-board computer that guided the descent of the Apollo 11 lunar module to the moon had a Kalman filter".
4. Meanshift and Camshift: These are algorithms for locating the maxima of a density function. They are also used for tracking.
5. Single object trackers: In this class of trackers, the first frame is marked using a rectangle to indicate the location of the object we want to track. The object is then tracked in subsequent frames using the tracking algorithm. In most real-life applications, these trackers are used in conjunction with an object detector.
6. Multiple object track finding algorithms: In cases when we have a fast object detector, it makes sense to detect multiple objects in each frame and then run a track finding algorithm that identifies which rectangle in one frame corresponds to a rectangle in the next frame.

Tracking vs Detection

If you have ever played with OpenCV face detection, you know that it works in real-time and you can easily detect the face in every frame. So, why do you need tracking in the first place? Let's explore the different reasons you may want to track objects in a video and not just do repeated detections.

1. Tracking is faster than Detection: Usually tracking algorithms are faster than detection algorithms. The reason is simple. When you are tracking an object that was detected in the previous frame, you know a lot about the appearance of the object. You also know the location in the previous frame and the direction and speed of its motion. So in the next frame, you can use all this information to predict the location of the object in the next frame and do a small search around the expected location of the object to accurately locate the object. A good tracking algorithm will use all information it has about the object up to that point while a detection algorithm always starts from scratch. Therefore, while designing an efficient system usually an object detection is run on every n^th frame while the tracking algorithm is employed in the n-1 frames in between. Why don't we simply detect the object in the first frame and track it subsequently? It is true that tracking benefits from the extra information it has, but you can also lose track of an object when they go behind an obstacle for an extended period of time or if they move so fast that the tracking algorithm cannot catch up. It is also common for tracking algorithms to accumulate errors and the bounding box tracking the object slowly drifts away from the object it is tracking. To fix these problems with tracking algorithms, a detection algorithm is run every so often. Detection algorithms are trained on a large number of examples of the object. They, therefore, have more knowledge about the general class of the object. On the other hand, tracking algorithms know more about the specific instance of the class they are tracking.
2. Tracking can help when detection fails: If you are running a face detector on a video and the person's face gets occluded by an object, the face detector will most likely fail. A good tracking algorithm, on the other hand, will handle some level of occlusion. In the video below, you can see Dr. Boris Babenko, the author of the MIL tracker, demonstrate how the MIL tracker works under occlusion.

3. Tracking preserves identity: The output of object detection is an array of rectangles that contain the object. However, there is no identity attached to the object. For example, in the video below, a detector that detects red dots will output rectangles corresponding to all the dots it has detected in a frame. In the next frame, it will output another array of rectangles. In the first frame, a particular dot might be represented by the rectangle at location 10 in the array, and in the second frame, it could be at location 17. While using detection on a frame we have no idea which rectangle corresponds to which object. On the other hand, tracking provides a way to literally connect the dots!

Object tracking using OpenCV 4 – the Tracking API

OpenCV 4 comes with a tracking API that contains implementations of many single object tracking algorithms. There are 8 different trackers available in OpenCV 4.2 — BOOSTING, MIL, KCF, TLD, MEDIANFLOW, GOTURN, MOSSE, and CSRT.

Note: OpenCV 3.2 has implementations of these 6 trackers — BOOSTING, MIL, TLD, MEDIANFLOW, MOSSE, and GOTURN. OpenCV 3.1 has implementations of these 5 trackers — BOOSTING, MIL, KCF, TLD, MEDIANFLOW. OpenCV 3.0 has implementations of the following 4 trackers — BOOSTING, MIL, TLD, MEDIANFLOW.

Update: In OpenCV 3.3, the tracking API has changed. The code checks for the version and then uses the corresponding API.

Before we provide a brief description of the algorithms, let us see the setup and usage. In the commented code below we first set up the tracker by choosing a tracker type — BOOSTING, MIL, KCF, TLD, MEDIANFLOW, GOTURN, MOSSE, or CSRT. We then open a video and grab a frame. We define a bounding box containing the object for the first frame and initialize the tracker with the first frame and the bounding box. Finally, we read frames from the video and just update the tracker in a loop to obtain a new bounding box for the current frame. Results are subsequently displayed.

Download Code To easily follow along this tutorial, please download code by clicking on the button below. It's FREE!

Object tracking using OpenCV – C++ Code

#include <opencv2/opencv.hpp> #include <opencv2/tracking.hpp> #include <opencv2/core/ocl.hpp>  using namespace cv; using namespace std;  // Convert to string #define SSTR( x ) static_cast< std::ostringstream & >( \ ( std::ostringstream() << std::dec << x ) ).str()  int main(int argc, char **argv) {     // List of tracker types in OpenCV 3.4.1     string trackerTypes[8] = {"BOOSTING", "MIL", "KCF", "TLD","MEDIANFLOW", "GOTURN", "MOSSE", "CSRT"};     // vector <string> trackerTypes(types, std::end(types));      // Create a tracker     string trackerType = trackerTypes[2];      Ptr<Tracker> tracker;      #if (CV_MINOR_VERSION < 3)     {         tracker = Tracker::create(trackerType);     }     #else     {         if (trackerType == "BOOSTING")             tracker = TrackerBoosting::create();         if (trackerType == "MIL")             tracker = TrackerMIL::create();         if (trackerType == "KCF")             tracker = TrackerKCF::create();         if (trackerType == "TLD")             tracker = TrackerTLD::create();         if (trackerType == "MEDIANFLOW")             tracker = TrackerMedianFlow::create();         if (trackerType == "GOTURN")             tracker = TrackerGOTURN::create();         if (trackerType == "MOSSE")             tracker = TrackerMOSSE::create();         if (trackerType == "CSRT")             tracker = TrackerCSRT::create();     }     

Share this post