In this article, we guide you in a step-by-step tutorial to use OpenCV and Streamlit to detect pathologies in ultrasound images. Within an interactive Streamlit application that takes the image and mask generated from a model as input and gives us the bounded box, mask outline, and heatmap. With all code functions needed.

 

Author: Gowthami Wudaru

 

Detecting pathologies

We have participated in Detecting Pathologies Through Computer Vision in Ultrasound Omdena challenge to build an Ultrasound solution that can detect the type and location of different pathologies. The solution works with 2D images and also can process a video stream.

Identify the presence of a specific pathology on the ultrasound image and provide the location of the pathology with bounding box coordinates and mask. Ultrasound is a relatively inexpensive and portable modality of diagnosis of life-threatening diseases and for use in point of care. This will assist to deliver impactful and feasible medical solutions to countries where there are significant resource challenges.

Visualizations help highlight the tumor location and give us the intensity of the tumor at each point in AOI (Area of Interest).

The app takes the image and mask generated from a model as input and gives us the bounded box, mask outline, and heatmap.

 

Work with Streamlit

Streamlit is an open-source framework for creating interactive apps in a short time entirely in python. The main advantage is its compatibility with other libraries like Matplotlib, OpenCV, NumPy, pandas, and many more. Installation is very easy!

pip install streamlit

 

OpenCV, Computer Vision most popular library

OpenCV is a Computer Vision library. We use it in our app to get the contours, draw them on images, and generate heatmaps using color maps. We can install OpenCV using pip.

pip install opencv-python-headless

 

Let’s define the steps needed to build this app with code

 

The APP CODE

We first import the dependencies.

import streamlit as st
from PIL import Image
import matplotlib.pyplot as plt
import pandas as pd
import cv2
import numpy as np
from matplotlib.patches import Rectangle

 

Read Data

We define a function to read images as inputs. We use file_uploader from streamlit to accept only png, jpg, jpeg file formats for images and Pillow’s Image to open image files.

def read_image(name):
  image = st.file_uploader("Upload an "+ name, type=["png", "jpg", "jpeg"])
  if image:
    im = Image.open(image)
    im.filename = image.name
    return im

 

Show Visualizations

The show image function takes an image, mask, and finds the contours in the mask i.e white object boundary from the black background. CHAIN_APPROX_NONE stores all the boundary points. RETR_EXTERNAL returns only extreme outer flags. All child contours are left behind.

Mask contours in computer vision - Source: Omdena

Source: Omdena Mask

Then, the image and contours are sent to _bbAndMask and _heatmap functions to get a bounded box, mask outline, and heatmap.

def show_image(image, mask):
  mask = cv2.cvtColor(np.array(mask), cv2.COLOR_BGR2GRAY)
  cnts,_= cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
  _bbAndMask(image, cnts)
  _heatmap(image, cnts)

 

The bounded box and mask outline are shown in a single figure with two axes.

Source: Omdena Bounded box and mask outline

Source: Omdena Bounded box and mask outline

We use Matplotlib for generating the visualizations and st.pyplot shows the visualizations in the Streamlit app.

def _bbAndMask(image, cnts):
  fig, (ax1, ax2) = plt.subplots(1, 2)
  ax1.axis('off')
  ax2.axis('off')
  _bbox(image, cnts, ax1)
  _maskOutline(image, cnts, ax2)
  st.pyplot(fig)

 

Bounded Box

We get the bounded box coordinates using OpenCV’s boundingRect and add Rectangle as a patch to the axis.

def _bbox(image, cnts, ax):
  ax.imshow(image)
  for c in cnts:
    area = cv2.contourArea(c)
    if area < 10:
      continue
    [x, y, w, h] = cv2.boundingRect(c)
    ax.add_patch(Rectangle((x, y), w, h, color = "red", fill = False))

 

Mask Outline

The mask outline is drawn in the image using _drawMask function with fill = False i.e the contour is not filled and the outline is shown.

def _maskOutline(image, cnts, ax):
  img = _drawMask(image, cnts, False)
  ax.imshow(img)

We generate the heatmap of the entire image by applying COLORMAP_JET and store it in heatmap_img. We draw contours using drawContours from OpenCV and draw outlines or fill the contour area in the image with zeros (markers). Then, we create a mask for the non-zero area in markers and fill the actual image in the masked area with heatmap_img. The third argument -1 in drawContours specifies to draw all the contours and the fifth argument “t” specifies the thickness, -1 filling the entire contour.

def _drawMask(image, cnts, fill=True):
  image = np.array(image)
  markers = np.zeros((image.shape[0], image.shape[1]))
  heatmap_img = cv2.applyColorMap(image, cv2.COLORMAP_JET)
  t = 2
  if fill:
    t = -1
  cv2.drawContours(markers, cnts, -1, (255, 0, 0), t)
  mask = markers>0
  image[mask,:] = heatmap_img[mask,:]
  return image

 

Heatmap

The _drawMask is used in heatmap generating with fill = True (default). We use an interactive widget i.e slider from 0 to 1 with 0.1 increments to show the heatmap at different intensities. We are overlaying the image and heatmap with different intensities by specifying alpha in imshow.

def _heatmap(image, cnts):
  fig2 = plt.figure()
  plt.axis('off')
  hm = st.slider("slider for heatmap", min_value=0.0, max_value=1.0, step=0.1, value=0.5)
  img = _drawMask(image, cnts)
  plt.imshow(img, alpha=hm)
  plt.imshow(image, alpha=1-hm)
  plt.title("heatmap")
  st.pyplot(fig2)

Source: Omdena Heat maps with different opacity

Source: Omdena Heat maps with different opacity

 

We then call read_image and show_image from main.

def main():
  st.set_page_config(page_title='Omdena Envisionit', page_icon=None, layout='centered', initial_sidebar_state='auto')
  st.title('Detecting Pathologies Through Computer Vision in Ultrasound')
  image = read_image('image')
  mask = read_image('mask')
  if image and mask:
    show_image(image, mask)

 

We call the main function when the script/module is run.

if __name__ == "__main__":
  main()

 

Running the APP

We store the code in a file (app.py) and run the Streamlit app using

streamlit run app.py

streamlit run app.py

 

You can now view your Streamlit app in your browser.

Local URL: http://localhost:8501
Network URL: http://XXX.XXX.XXX.XXX:8501

Source: Omdena - Pathology dashboard Landing Page

Source: Omdena – Pathology dashboard Landing Page

 

Conclusion

We successfully created a Streamlit app that takes the inputs — image and mask by browsing files in the computer, and shows the bounded box and mask outline on the image and a heat map with an interactive slider to get a sense of the tumor intensity.

Source: Omdena - Pathology Streamlit App in Action

Source: Omdena – Pathology Streamlit App in Action

Develop Your Career and Make a Real-World Impact

Innovation

The world´s only place for truly collaborative AI projects to apply your skills on real-world data with changemakers from around the world.

Apply & grow your skills in our real-world projects

Upcoming AI Projects

AI Teams

Make an impact in our upcoming projects in Natural Language Processing, Computer Vision, Machine Learning, Remote Sensing, and more.

Check out our projects!

Stay in touch via our newsletter.

Be notified (a few times a month) about top-notch articles, new real-world projects, and events with our community of changemakers.

Sign up here