Developing a Multimodal Model for Chest Disease Detection Using Radiology Images and Text Data

Local Chapter Berlin, Germany Local Chapter , Silicon Valley, USA Chapter

Coordinated byGermany ,

Status: Completed

Project Duration: 23 Jul 2023 - 10 Sep 2023

Open Source resources available from this project

Project background.

In this project, collaborators will develop a multimodal model using chest X-rays and electronic health records (EHR)/clinical data to detect pneumonia and tuberculosis. The goal is to leverage both visual information from X-rays and textual information from EHR/clinical data to improve the accuracy of disease detection. Today Multi-Modal data is commonly collected and used in diagnosing diseases; this technique is relatively new and will give participants a gist of this methodology.

The problem.

Chest diseases are prevalent in countries, and this methodology will help us use two modalities to diagnose accurately and provide richer contextual information leading to better patient outcomes.

Project goals.

Learn about Multimodal models. How to build them. Become more familiar with NLP and or Computer Vision. This project touches on both topics and more.

Project plan.

  • Week 1

    Project Setup and Data Exploration

  • Week 2

    NLP and Computer Vision Basic to advance

  • Week 3

    CNN for Computer vision

  • Week 4

    Natural Language Processing (NLP) for EHR/Clinical Data

  • Week 5

    Multimodal basics

  • Week 7

    More Multimodal learning and Project wrap up

Learning outcomes.

NLP, Computer Vision, Multimodal Learning, Leadership & public speaking skills, teamwork and collaboration

Share project on: