Tracking fish in complex backgrounds using machine learning

After labelling more than thousand images of fish in the videos with complex backgrounds, uneven lighting and bubbles, I trained a faster-RCNN model using the object detection API and then tested the frozen model on video recordings using OpenCV. The results? Pretty solid detection of both fish in most frames of the recording. However, the aerator in the tank and some background objects are also detected on occasion as fish.

To identify and track individual fish across frames in the video, I implemented the SORT algorithm (Simple and Online Real-Time Tracking) and modified it to limit the number of possible detected objects to two. The individual trajectories can survive mild occlusions (like the one shown here) using the current algorithm.

From the trajectories, we can visualize space use or exploration in the tank of each fish. We can also measure sociability as the distance between the two fish, the average speed and the similarity of direction of movement of the two fish as an indicator of coordination. Many of these features are available in the movekit repository developed by Eren Cakmak, Arjun Majumdar, and Jolle Jolles at the University of Konstanz.