Issue with NMM Merging Mechanism for Dense Rice Grain Images

[SaiJeevanPuchakayala]

Search before asking

• I have searched the Supervision issues and found no similar feature requests.

Question

Issue with the stitching mechanism of detected images using Non-Maximum Merging (NMM) in the supervision library. The merging mechanism does not seem to be working correctly on dense rice grain images. The code slices a high-resolution image into smaller 512x512 images for detection and attempts to merge them back to the original 2048x2048 image. However, the merging results are not accurate for densely packed objects.

Steps to Reproduce:

import torch
from sahi.predict import get_sliced_prediction
from sahi import AutoDetectionModel
import cv2
import matplotlib.pyplot as plt
import numpy as np

# Callback function for inference
def callback(image: np.ndarray) -> sv.Detections:
    with torch.no_grad():  # Disable gradient calculation
        result = model(image, device='cuda')[0]  # Specify GPU if available
    return sv.Detections.from_ultralytics(result)

# Function to process an image and display the result
def process_image(image_path: str):
    try:
        image = cv2.imread(image_path)
        image = cv2.resize(image, (2048, 2048))
        if image is None:
            raise FileNotFoundError(f"Image at path {image_path} could not be loaded.")

        # Perform inference
        slicer = sv.InferenceSlicer(
            slice_wh=(512, 512),
            overlap_ratio_wh=(0.1, 0.1),
            thread_workers=10,
            iou_threshold=0.1,
            overlap_filter_strategy=sv.OverlapFilter.NON_MAX_MERGE,
            callback=callback)

        detections = slicer(image)

        # Filter detections based on confidence
        detections = detections[detections.confidence > 0.3]

        mask_annotator = sv.MaskAnnotator()
        annotated_image = mask_annotator.annotate(scene=image.copy(), detections=detections)

        # Draw contours around the masks and fit ellipses
        for i in range(detections.mask.shape[0]):
            mask = detections.mask[i].astype(np.uint8)
            contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
            cv2.drawContours(annotated_image, contours, -1, (0, 255, 0), 2)

        plt.figure(figsize=(20, 20))
        plt.imshow(cv2.cvtColor(annotated_image, cv2.COLOR_BGR2RGB))
        plt.axis('off')
        plt.show()

        # Print the number of detections
        print(f"Number of detections: {len(detections.confidence)}")

    except Exception as e:
        print(f"An error occurred: {str(e)}")
        raise

# Example usage
image_path = '/content/drive/MyDrive/Mixed Grains/test image .jpg'
process_image(image_path)

And here’s what I’m seeing: instead of nicely merged detections, I get overlapping and misaligned masks. The NMM just doesn't seem to handle these dense objects well.

Sample Image: The image used is a high-resolution dense rice grain image.

Expected Behavior: The sliced 512x512 images should be correctly merged back to the original 2048x2048 image with accurate detections.

Observed Behavior: The merging mechanism of NMM does not seem to work as expected on dense rice grain images. The detections are not accurately merged, leading to overlapping and misaligned masks.

What I’ve Tried:

• Image Size: Adjusted the image size and slice size.
• Overlap Ratio: Played around with different overlap ratios.
• Confidence Threshold: Tweaked the confidence threshold for detections.

Additional

No response

[LinasKo]

Hi @SaiJeevanPuchakayala 👋

Unfortunately, this is a tricky case. Tweaking with the parameters is the right approach. You may also try OverlapFilter.NON_MAX_SUPPRESSION and OverlapFilter.NONE.

Otherwise, you'll need a better-trained detection or instance segmentation model.

[SaiJeevanPuchakayala]

Hi @LinasKo 👋

I played a lot with tweaking parameters and also applied NMS and NMM, but NMM is the right filter for this work. Still, I'm not able to get the expected results. But I doubt if there is something wrong going on with the training images (I attached a few training images below). The doubt I have is when the actual high-resolution image is sliced, even the annotation and the object (rice grain) are also getting sliced, and these are the images being used for training the model. Then, the trained model is used to make the inference, as shown earlier in the thread above.

[LinasKo]

We'll look into it. Something tells me it could be related to how we determine which detections to merge when segmentation models are used.

Edit: it might take some time, we're stretched thin at the moment

[SaiJeevanPuchakayala]

We'll look into it. Something tells me it could be related to how we determine which detections to merge when segmentation models are used.

Edit: it might take some time, we're stretched thin at the moment

Hey @LinasKo, were you able to find a fix for the above issue with merging the detections?

[SkalskiP]

Hi, @SaiJeevanPuchakayala 👋🏻 @LinasKo is on vacation right now.