How to remove the background of an image using Watershed on android?

Question

I’m developing an app that classifies images of wounds. For this, I need to highlight the region of interest of the background to facilitate the work of the classification algorithm.

The user uses the cellular touch to inform the Watershed algorithm where it will segment the area of the wound. With this, I pass the coordinates and the image to be segmented.

However, I am having difficulty using Watershed together with the user demarcated region because I have no experience with opencv+android.

In the code below I pass the image to the set of coordinates for the backgroud extraction function, only that I still could not complete by not knowing at what point I pass the coordinates gives area for the Watershed to segment only that region.

Main screen

Background extraction method

typealias Coordinates = Pair<Point, Point>
private fun extractForegroundFromBackground(coordinates: Coordinates, currentPhotoPath: String): String {
    // TODO: Provide complex object that has both path and extension


    val width: Int  = bitmap.getWidth()
    val height: Int = bitmap.getHeight()
    val rect = Rect(coordinates.first, coordinates.second)//Coordinates ROI
    val rgba = Mat()
    val edges = Mat()
    val hierarchy = Mat()
    val gray_mat = Mat()
    val threeChannel = Mat()
    Utils.bitmapToMat(bitmap, gray_mat)
    //Conversion colors
    cvtColor(gray_mat, rgba, COLOR_RGBA2RGB)
    cvtColor(rgba, threeChannel, COLOR_RGB2GRAY)
    threshold(threeChannel, threeChannel, 100.0, 255.0, THRESH_OTSU)
    //Find contours
    Canny(threeChannel, edges, 10.0, 100.0)
    val contours: List<MatOfPoint> = ArrayList()
    findContours(edges, contours, hierarchy, RETR_EXTERNAL, CHAIN_APPROX_SIMPLE)
    val markers = Mat(rgba.size(), CvType.CV_32S, Scalar(0.0))
    //Draw contours
    for (i in contours.indices) {
        drawContours(markers, contours, i, Scalar(i + 1.0), -1, 8, hierarchy, INTER_MAX, Point())
    }
    val marker_tempo = Mat()
    markers.convertTo(marker_tempo, CvType.CV_32S)
    //Start the WaterShed Segmentation
    watershed(rgba, markers)
    //Generated output
    marker_tempo.convertTo(markers, CvType.CV_8U)
    result_Bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.RGB_565)
    Utils.matToBitmap(edges, result_Bitmap)
    image.setImageBitmap(result_Bitmap)

    return currentPhotoPath
}

Exit:

I need to return the image with the black background and only the region of the highlighted wound, as shown below.

Answer 1

private fun extractForegroundFromBackground(coordinates: Coordinates, currentPhotoPath: String): String {
    // TODO: Provide complex object that has both path and extension

    // Matrices that OpenCV will be using internally
    val bgModel = Mat()
    val fgModel = Mat()

    val srcImage = Imgcodecs.imread(currentPhotoPath)
    val iterations = 5

    // Mask image where we specify which areas are background, foreground or probable background/foreground
    val firstMask = Mat()

    val source = Mat(1, 1, CvType.CV_8U, Scalar(Imgproc.GC_PR_FGD.toDouble()))
    val rect = Rect(coordinates.first, coordinates.second)

    // Run the grab cut algorithm with a rectangle (for subsequent iterations with touch-up strokes,
    // flag should be Imgproc.GC_INIT_WITH_MASK)
    Imgproc.grabCut(srcImage, firstMask, rect, bgModel, fgModel, iterations, Imgproc.GC_INIT_WITH_RECT)

    // Create a matrix of 0s and 1s, indicating whether individual pixels are equal
    // or different between "firstMask" and "source" objects
    // Result is stored back to "firstMask"
    Core.compare(firstMask, source, firstMask, Core.CMP_EQ)

    // Create a matrix to represent the foreground, filled with white color
    val foreground = Mat(srcImage.size(), CvType.CV_8UC3, Scalar(255.0, 255.0, 255.0))

    // Copy the foreground matrix to the first mask
    srcImage.copyTo(foreground, firstMask)

    // Create a red color
    val color = Scalar(255.0, 0.0, 0.0, 255.0)
    // Draw a rectangle using the coordinates of the bounding box that surrounds the foreground
    Imgproc.rectangle(srcImage, coordinates.first, coordinates.second, color)

    // Create a new matrix to represent the background, filled with white color
    val background = Mat(srcImage.size(), CvType.CV_8UC3, Scalar(0.0, 0.0, 0.0))

    val mask = Mat(foreground.size(), CvType.CV_8UC1, Scalar(255.0, 255.0, 255.0))
    // Convert the foreground's color space from BGR to gray scale
    Imgproc.cvtColor(foreground, mask, Imgproc.COLOR_BGR2GRAY)

    // Separate out regions of the mask by comparing the pixel intensity with respect to a threshold value
    Imgproc.threshold(mask, mask, 254.0, 255.0, Imgproc.THRESH_BINARY_INV)

    // Create a matrix to hold the final image
    val dst = Mat()
    // copy the background matrix onto the matrix that represents the final result
    background.copyTo(dst)

    val vals = Mat(1, 1, CvType.CV_8UC3, Scalar(0.0))
    // Replace all 0 values in the background matrix given the foreground mask
    background.setTo(vals, mask)

    // Add the sum of the background and foreground matrices by applying the mask
    Core.add(background, foreground, dst, mask)

    // Save the final image to storage
    Imgcodecs.imwrite(currentPhotoPath + "_tmp.png", dst)

    // Clean up used resources
    firstMask.release()
    source.release()
    bgModel.release()
    fgModel.release()
    vals.release()
    dst.release()

    return currentPhotoPath
}