Developing a Multimodal Model for Chest Disease Detection Using Radiology Images and Text Data

Local Chapter Berlin, Germany Local Chapter , Silicon Valley, USA Chapter

Coordinated by ,

Status: Completed

Project Duration: 23 Jul 2023 - 10 Sep 2023

Open Source resources available from this project

Project background.

In this project, collaborators will develop a multimodal model using chest X-rays and electronic health records (EHR)/clinical data to detect pneumonia and tuberculosis. The goal is to leverage both visual information from X-rays and textual information from EHR/clinical data to improve the accuracy of disease detection. Today Multi-Modal data is commonly collected and used in diagnosing diseases; this technique is relatively new and will give participants a gist of this methodology.

Project plan.

  • Week 1

    Project Setup and Data Exploration

  • Week 2

    NLP and Computer Vision Basic to advance

  • Week 3

    CNN for Computer vision

  • Week 4

    Natural Language Processing (NLP) for EHR/Clinical Data

  • Week 5

    Multimodal basics

  • Week 7

    More Multimodal learning and Project wrap up

Learning outcomes.

NLP, Computer Vision, Multimodal Learning, Leadership & public speaking skills, teamwork and collaboration

Share project on: