In this article, we guide you in a step-by-step tutorial to use OpenCV and Streamlit to detect pathologies in ultrasound images. Within an interactive Streamlit application that takes the image and mask generated from a model as input and gives us the bounded box, mask outline, and heatmap. With all code functions needed.
Author: Gowthami Wudaru
We have participated in Detecting Pathologies Through Computer Vision in Ultrasound Omdena challenge to build an Ultrasound solution that can detect the type and location of different pathologies. The solution works with 2D images and also can process a video stream.
Identify the presence of a specific pathology on the ultrasound image and provide the location of the pathology with bounding box coordinates and mask. Ultrasound is a relatively inexpensive and portable modality of diagnosis of life-threatening diseases and for use in point of care. This will assist to deliver impactful and feasible medical solutions to countries where there are significant resource challenges.
Visualizations help highlight the tumor location and give us the intensity of the tumor at each point in AOI (Area of Interest).
The app takes the image and mask generated from a model as input and gives us the bounded box, mask outline, and heatmap.
Work with Streamlit
Streamlit is an open-source framework for creating interactive apps in a short time entirely in python. The main advantage is its compatibility with other libraries like Matplotlib, OpenCV, NumPy, pandas, and many more. Installation is very easy!
pip install streamlit
OpenCV, Computer Vision most popular library
OpenCV is a Computer Vision library. We use it in our app to get the contours, draw them on images, and generate heatmaps using color maps. We can install OpenCV using pip.
pip install opencv-python-headless
Let’s define the steps needed to build this app with code
The APP CODE
We first import the dependencies.
import streamlit as st from PIL import Image import matplotlib.pyplot as plt import pandas as pd import cv2 import numpy as np from matplotlib.patches import Rectangle
We define a function to read images as inputs. We use file_uploader from streamlit to accept only png, jpg, jpeg file formats for images and Pillow’s Image to open image files.
def read_image(name): image = st.file_uploader("Upload an "+ name, type=["png", "jpg", "jpeg"]) if image: im = Image.open(image) im.filename = image.name return im
The show image function takes an image, mask, and finds the contours in the mask i.e white object boundary from the black background. CHAIN_APPROX_NONE stores all the boundary points. RETR_EXTERNAL returns only extreme outer flags. All child contours are left behind.
Then, the image and contours are sent to _bbAndMask and _heatmap functions to get a bounded box, mask outline, and heatmap.
def show_image(image, mask): mask = cv2.cvtColor(np.array(mask), cv2.COLOR_BGR2GRAY) cnts,_= cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE) _bbAndMask(image, cnts) _heatmap(image, cnts)
The bounded box and mask outline are shown in a single figure with two axes.
We use Matplotlib for generating the visualizations and st.pyplot shows the visualizations in the Streamlit app.
def _bbAndMask(image, cnts): fig, (ax1, ax2) = plt.subplots(1, 2) ax1.axis('off') ax2.axis('off') _bbox(image, cnts, ax1) _maskOutline(image, cnts, ax2) st.pyplot(fig)
We get the bounded box coordinates using OpenCV’s boundingRect and add Rectangle as a patch to the axis.
def _bbox(image, cnts, ax): ax.imshow(image) for c in cnts: area = cv2.contourArea(c) if area < 10: continue [x, y, w, h] = cv2.boundingRect(c) ax.add_patch(Rectangle((x, y), w, h, color = "red", fill = False))
The mask outline is drawn in the image using _drawMask function with fill = False i.e the contour is not filled and the outline is shown.
def _maskOutline(image, cnts, ax): img = _drawMask(image, cnts, False) ax.imshow(img)
We generate the heatmap of the entire image by applying COLORMAP_JET and store it in heatmap_img. We draw contours using drawContours from OpenCV and draw outlines or fill the contour area in the image with zeros (markers). Then, we create a mask for the non-zero area in markers and fill the actual image in the masked area with heatmap_img. The third argument -1 in drawContours specifies to draw all the contours and the fifth argument “t” specifies the thickness, -1 filling the entire contour.
def _drawMask(image, cnts, fill=True): image = np.array(image) markers = np.zeros((image.shape, image.shape)) heatmap_img = cv2.applyColorMap(image, cv2.COLORMAP_JET) t = 2 if fill: t = -1 cv2.drawContours(markers, cnts, -1, (255, 0, 0), t) mask = markers>0 image[mask,:] = heatmap_img[mask,:] return image
The _drawMask is used in heatmap generating with fill = True (default). We use an interactive widget i.e slider from 0 to 1 with 0.1 increments to show the heatmap at different intensities. We are overlaying the image and heatmap with different intensities by specifying alpha in imshow.
def _heatmap(image, cnts): fig2 = plt.figure() plt.axis('off') hm = st.slider("slider for heatmap", min_value=0.0, max_value=1.0, step=0.1, value=0.5) img = _drawMask(image, cnts) plt.imshow(img, alpha=hm) plt.imshow(image, alpha=1-hm) plt.title("heatmap") st.pyplot(fig2)
We then call read_image and show_image from main.
def main(): st.set_page_config(page_title='Omdena Envisionit', page_icon=None, layout='centered', initial_sidebar_state='auto') st.title('Detecting Pathologies Through Computer Vision in Ultrasound') image = read_image('image') mask = read_image('mask') if image and mask: show_image(image, mask)
We call the main function when the script/module is run.
if __name__ == "__main__": main()
Running the APP
We store the code in a file (app.py) and run the Streamlit app using
streamlit run app.py
streamlit run app.py
You can now view your Streamlit app in your browser.
We successfully created a Streamlit app that takes the inputs — image and mask by browsing files in the computer, and shows the bounded box and mask outline on the image and a heat map with an interactive slider to get a sense of the tumor intensity.