Possible issue in GlobalMotionCompensation.cpp

[arnaud-nt2i]

@viplix3 Hi!
Thanks again for your work.

I have been playing with GMC algos and there seeme to be a pb in
HomographyMatrix OpenCV_VideoStab_GMC::apply(const cv::Mat &frame_raw, const std::vector<Detection> &detections) {}
https://github.com/viplix3/BoTSORT-cpp/blob/4185b005e34d49c6c9e5cdf2dbde70ed11f70a0f/botsort/src/GlobalMotionCompensation.cpp#L571C21-L571C21

according to the PR that added Masking to opencv/videostab/src/global_motion.cpp : https://github.com/opencv/opencv_contrib/pull/2052 :
The mask video must contain black (0 values) for parts, which are ignored during stabilization.

Or, the code is doing the opposite :

cv::Mat mask = cv::Mat::zeros(frame.size(), CV_8U);
mask(rect) = 255; and then
_keypoint_motion_estimator->setFrameMask(mask);

So I am wondering if Keypoints for BasedMotionEstimator are only taken on detection and not on background ...
By the way, there should be a clamp before mask(rect) = 255; : clampRectInplace(rect, width, height);
with a function added in utils.h :

inline void clampRectInplace(cv::Rect &roi, int maxW, int maxH)
{
roi.x = std::max(0, std::min(roi.x, maxW));
roi.y = std::max(0, std::min(roi.y, maxH));
roi.width = std::max(0, std::min(roi.width, maxW - roi.x));
roi.height = std::max(0, std::min(roi.height, maxH - roi.y));
}

[arnaud-nt2i]

@viplix3
another element, https://docs.opencv.org/3.4/d0/d13/classcv_1_1Feature2D.html#a5968e9bc8497a8eb845272b9442559f3

According to official open cv doc, for cv::GFTTDetector:
"mask Mask specifying where to look for keypoints (optional). It must be a 8-bit integer matrix with non-zero values in the region of interest."

So it seems in your code, you are setting every detections as ROI (255) so key points will only be looked on potentially moving targets and not on background ( to correctly estimate camera motion)

I think your code in HomographyMatrix ORB_GMC::apply() is the good way to deal with the mask

Bye !

[viplix3]

Hi @arnaud-nt2i

Thanks for showing interest in the work.

As you've pointed out correctly, the detection regions should be masked off so they don't contribute to GMC calculations, however, it also depends on how the output of the GMC algorithm is used.

For the ORB GMC algorithm I am doing the correct thing, i.e. I set the detection region to 0 as shown here.

Doing this essentially masks off the detections so keypoints are estimated only on the background region and hence only the background is used for the estimation of the homography matrix b/w consecutive frames as explained by you.

Thanks for pointing it out, I will update the code for the rest of the GMC algorithms accordingly.

[pk429]

Hi @arnaud-nt2i

Thanks for showing interest in the work.

As you've pointed out correctly, the detection regions should be masked off so they don't contribute to GMC calculations, however, it also depends on how the output of the GMC algorithm is used.

For the ORB GMC algorithm I am doing the correct thing, i.e. I set the detection region to 0 as shown here.

Doing this essentially masks off the detections so keypoints are estimated only on the background region and hence only the background is used for the estimation of the homography matrix b/w consecutive frames as explained by you.

Thanks for pointing it out, I will update the code for the rest of the GMC algorithms accordingly.

Hi！I have some questions for it. In the OpenCV_VideoStab_GMC module, the foreground is set to white (255), while the background is set to black (0). Based on test results, the video stabilization effect of OpenCV's VideoStab is superior to that of ORB. Why?