Visualizing Pathologies in Ultrasound Images Using OpenCV and Streamlit

April 4, 2022

In this article, we guide you in a step-by-step tutorial to use OpenCV and Streamlit to detect pathologies in ultrasound images. Within an interactive Streamlit application that takes the image and mask generated from a model as input and gives us the bounded box, mask outline, and heatmap. With all code functions needed.

Detecting Pathologies in Ultrasound Images using OpenCV and Streamlit

We have participated in Detecting Pathologies Through Computer Vision in Ultrasound Omdena challenge to build an Ultrasound solution that can detect the type and location of different pathologies. The solution works with 2D images and also can process a video stream.

Identify the presence of a specific pathology on the ultrasound image and provide the location of the pathology with bounding box coordinates and mask. Ultrasound is a relatively inexpensive and portable modality of diagnosis of life-threatening diseases and for use in point of care. This will assist to deliver impactful and feasible medical solutions to countries where there are significant resource challenges.

Visualizations help highlight the tumor location and give us the intensity of the tumor at each point in AOI (Area of Interest).

The app takes the image and mask generated from a model as input and gives us the bounded box, mask outline, and heatmap.

Work with Streamlit

Streamlit is an open-source framework for creating interactive apps in a short time entirely in Python. The main advantage is its compatibility with other libraries like Matplotlib, OpenCV, NumPy, pandas, and many more. Installation is very easy!

pip install streamlit

Work with OpenCV

OpenCV is Computer Vision’s most popular library. We use it in our app to get the contours, draw them on images, and generate heatmaps using color maps. We can install OpenCV using pip.

pip install opencv-python-headless

Let’s define the steps needed to build this app with code

The APP CODE

We first import the dependencies.

import streamlit as st
from PIL import Image
import matplotlib.pyplot as plt
import pandas as pd
import cv2
import numpy as np
from matplotlib.patches import Rectangle

Read Data

We define a function to read images as inputs. We use file_uploader from Streamlit to accept only png, jpg, jpeg file formats for images and Pillow’s Image to open image files.

def read_image(name):
  image = st.file_uploader("Upload an "+ name, type=["png", "jpg", "jpeg"])
  if image:
    im = Image.open(image)
    im.filename = image.name
    return im

Show Visualizations

The show image function takes an image, mask, and finds the contours in the mask i.e white object boundary from the black background. CHAIN_APPROX_NONE stores all the boundary points. RETR_EXTERNAL returns only extreme outer flags. All child contours are left behind.

Mask contours in computer vision - Source: Omdena

Then, the image and contours are sent to _bbAndMask and _heatmap functions to get a bounded box, mask outline, and heatmap.

def show_image(image, mask):
  mask = cv2.cvtColor(np.array(mask), cv2.COLOR_BGR2GRAY)
  cnts,_= cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
  _bbAndMask(image, cnts)
  _heatmap(image, cnts)

The bounded box and mask outline are shown in a single figure with two axes.

Source: Omdena Bounded box and mask outline

We use Matplotlib for generating the visualizations and st.pyplot shows the visualizations in the Streamlit app.

def _bbAndMask(image, cnts):
  fig, (ax1, ax2) = plt.subplots(1, 2)
  ax1.axis('off')
  ax2.axis('off')
  _bbox(image, cnts, ax1)
  _maskOutline(image, cnts, ax2)
  st.pyplot(fig)

Bounded Box

We get the bounded box coordinates using OpenCV’s boundingRect and add Rectangle as a patch to the axis.

def _bbox(image, cnts, ax):
  ax.imshow(image)
  for c in cnts:
    area = cv2.contourArea(c)
    if area < 10:
      continue
    [x, y, w, h] = cv2.boundingRect(c)
    ax.add_patch(Rectangle((x, y), w, h, color = "red", fill = False))

Mask Outline

The mask outline is drawn in the image using _drawMask function with fill = False i.e the contour is not filled and the outline is shown.

def _maskOutline(image, cnts, ax):
  img = _drawMask(image, cnts, False)
  ax.imshow(img)

We generate the heatmap of the entire image by applying COLORMAP_JET and store it in heatmap_img. We draw contours using drawContours from OpenCV and draw outlines or fill the contour area in the image with zeros (markers). Then, we create a mask for the non-zero area in markers and fill the actual image in the masked area with heatmap_img. The third argument -1 in drawContours specifies to draw all the contours and the fifth argument “t” specifies the thickness, -1 filling the entire contour.

def _drawMask(image, cnts, fill=True):
  image = np.array(image)
  markers = np.zeros((image.shape[0], image.shape[1]))
  heatmap_img = cv2.applyColorMap(image, cv2.COLORMAP_JET)
  t = 2
  if fill:
    t = -1
  cv2.drawContours(markers, cnts, -1, (255, 0, 0), t)
  mask = markers>0
  image[mask,:] = heatmap_img[mask,:]
  return image

Heatmap

The _drawMask is used in heatmap generating with fill = True (default). We use an interactive widget i.e slider from 0 to 1 with 0.1 increments to show the heatmap at different intensities. We are overlaying the image and heatmap with different intensities by specifying alpha in imshow.

def _heatmap(image, cnts):
  fig2 = plt.figure()
  plt.axis('off')
  hm = st.slider("slider for heatmap", min_value=0.0, max_value=1.0, step=0.1, value=0.5)
  img = _drawMask(image, cnts)
  plt.imshow(img, alpha=hm)
  plt.imshow(image, alpha=1-hm)
  plt.title("heatmap")
  st.pyplot(fig2)

Source: Omdena Heat maps with different opacity

We then call read_image and show_image from main.

def main():
  st.set_page_config(page_title='Omdena Envisionit', page_icon=None, layout='centered', initial_sidebar_state='auto')
  st.title('Detecting Pathologies Through Computer Vision in Ultrasound')
  image = read_image('image')
  mask = read_image('mask')
  if image and mask:
    show_image(image, mask)

We call the main function when the script/module is run.

if __name__ == "__main__":
  main()

Running the APP

We store the code in a file (app.py) and run the Streamlit app using:

streamlit run app.py

You can now view your Streamlit app in your browser.

Local URL: http://localhost:8501
Network URL: http://XXX.XXX.XXX.XXX:8501

Source: Omdena - Pathology dashboard Landing Page

Conclusion

We successfully created a Streamlit app that takes the inputs — image, and mask by browsing files on the computer, and shows the bounded box and mask outline on the image and a heat map with an interactive slider to get a sense of the tumor intensity.