Projects in Medical Imaging with TensorFlow 2.X and Keras
# Deep Learning # Medical Imaging
# Image Classification # Image Segmentation # Object Detection
Project Motivation:
Since I have been studying a lot in deep learning (still a lot to catch up), I will like to apply all the techniques to my projects. My goal is to implement the models and tools that I learned over the past few years and build AI models of my interests. In this article, I will be introducing three projects relevant to Medical Imaging.
Project 1 — Pneumonia Image Classification:
Image Classification:
The goal of an image classification model is to identify objects seen in pictures. For instance, for a given picture of a cat, can AI guess the content is a cat?
How Classifying Pneumonia Helps Doctors:
In general, having an auto diagnosis system reduces the report time for the patients. Additionally, doctors can review the results produced by AI and decide whether to second AI’s opinion. In other words, AI diagnosis becomes a supportive role. Although this project relates to Pneumonia X-ray image classification, same model and same concept can be transplanted to other medical imaging projects. For instance, “Classifying Different Blood Cells”.
Project Summary:
I want to build a CNN model that can properly classify patients into either 0 for not having pneumonia or 1 for having pneumonia. Taking a step further, instead of only producing predictions, I will analyze my model in two ways.
1. Creating a mechanism to inspect and ensure our model makes rational predictions
2. Evaluating results not on average accuracy but with recall and precision
Data Source:
RSNA Pneumonia Detection Challenge
(https://www.kaggle.com/c/rsna-pneumonia-detection-challenge)
Project Steps:
Step-1:
Extract labels from the given csv file and images from dicom files.
— Use Python Augmentor Package to create augmented images for 1s, since positive cases are very few and I am dealing with imbalanced dataset.
( Note: It is important to not use augmentation that will distort the images excessively)
Step-2:
Create csv files with columns that specified (image_path, label), which
prevent us from loading all images into RAM and cause computation issues.
Step-3:
Download and setup Transfer Learning pipeline and modify the model output layer.
Evaluate results with
a) Confusion Matrix
b) Classification Reports
c) GradCAM with tf-explain
Results:
Below are the results after 8 epochs of training. I have set up an EarlyStopping callback that monitored on ‘val_loss’ to prevent overfitting but I will argue that the model can reach a better result after a couple more epochs.
The reason is that the training set is quite large and my batch_size is small which creates potential bias for every update in weights. In my case the weights tend to skew to 0s which makes training unstable.
For Future Study and Improvements:
There are definitely rooms for improvements. Opportunities that I have in mind include:
- Select different models, or search for published state-of-the-art models
- Set up appropriate class weights or build custom loss function that give more attention to 1s (positives)
- Clean up the data and ensure that the dataset does not include any distorted or tilted images. (Some images are not horizontally aligned)
Finally, here is the link to the project where I hosted on GitHub.
I look forward to seeing comments or suggestions from you!
Project 2 — Heart Location Detection:
Object Detection:
The goal of an object detection model is to local objects seen in pictures. For instance, for a given picture of a cat, can AI locate and create bounding box surrounding the cat?
How Locating Organs Helps Doctors:
Let’s assume that previously, doctors would have to count the white blood and red blood cells in person. Then by the white/red cells ratio, the doctor would then file a health report for his/her patient.
“Counting Red Cells / White Cells personally is time-consuming and is prone to errors”, therefore, having an auto-detection system reduces the report time. The object detection model allows swift calculations of object counts in images. Additionally, the object detection model can also be used in robotic surgery in the future. Robots will recognize where an organ is but in reality, perhaps this is still difficult since surgery contains lots of uncertainty.
Project Summary:
My goal is to build an Object Detection model that can identify the location of a heart from X-Ray images. The performance of the model will be evaluated with the Intersection of Union (IoU) metric with a threshold of 0.5. For more information on IoU, I highly recommend you check out PyImageSearch’s article listed below.
Data Source:
RSNA Pneumonia Detection Challenge
(https://www.kaggle.com/c/rsna-pneumonia-detection-challenge)
Project Steps:
Step-1:
Extract bounding boxes labels from the given csv file and images from dicom files.
Step-2:
Use Imgaug to create augmented images with associated new bounding boxes. In addition to the augmentation pipeline, I challenge myself to create a more efficient data consumption pipeline by saving labels and images into
TF-Records.
Note: Saving Data into TF-Record can not only save your storage space up to 70%~80% but also it is said to make training faster in the TensorFlow Eco-system.
Step-3:
Reading data from from disk and parsing data with tf.io functions.
Step-4:
Using the same ResNet50 model from the previous Pneumonia Classification Project. A small adjustment was made in the prediction layer where the output neurons became 4.
Results:
Below are the results after 63 epochs of training. I have set up an EarlyStopping callback that monitored on ‘val_loss’ to prevent overfitting. My results are shown as below:
Number of predictions where iou > threshold(0.5): 29
Number of predictions where iou < threshold(0.5): 3Note: the blue bounding boxes are the ground truth, while the red bounding boxes are the predicted one. However, some blue bounding boxes are distorted (I will try to figure out what happened in the future….)
For Future Study and Improvements:
There are definitely rooms for improvements. Opportunities that I have in mind include:
- Select different models, or search for published state-of-the-art models
- Changing a loss function to improve overall IoU score
- If RAM allowed, increasing the batch size for a more accurate update in gradients
- Use TensorFlow Object Detection API (Results are posted on GitHub)
Finally, here is the link to the project where I hosted on GitHub.
I look forward to seeing comments or suggestions from you!
Project 3 — Left Atrium MRI Scans Segmentation:
Image Segmentation:
The goal of an image segmentation model is to local objects seen in pictures and take a step further by coloring the areas. For instance, for a given picture of a cat, can AI locate and create color mask on the cat?
How Segmenting Organs Helps Doctors:
How big is a patient’s tumor? Where is the tumor exactly? These are the questions that can be answered using a segmentation model. Unlike object detection model, a segmentation model goes down to the pixel-level and thus gives a better overall results in locating objects.
Project Summary:
My Goal is to create two image segmentation models that can identify the left atrium segmentation from MRI scans. These models are Fully Convolutional Network (FCN-8) and UNET. The accuracy of the model will be measured using the binary_crossentropy loss and both predicted results will be compiled to GIFs.
Side Note:
Model structures are learned from Coursera Course, Advanced Techniques TensorFlow
Data Source:
Source: http://medicaldecathlon.com/
License: https://creativecommons.org/licenses/by-sa/4.0/
Project Steps:
Step-1:
Extract mri and segmentation labels using nibabel.
Step-2:
Covert extracted numpy arrays to TF-Records, a data format that is believed to be the best for training when using TensorFlow Keras
Additionally, this format saves a lot of storage space on your PC :D! (70~80% storage savings)
Note: Saving Data into TF-Record can not only save your storage space up to 70%~80% but also it is said to make training faster in the TensorFlow Eco-system.
Step-3:
Build models, FCN-8 and UNET with
Final Output Layer: “Sigmoid”
Loss: “Binary CrossEntropy”
Optimizer: “Adam”
Side Note: These structures can also be used for multi-class segmentation problem but changes need to apply accordingly
Results:
Ground Truth
FCN-8 and UNET Results
