Deep Learning Enabled Hybrid CT-MRI with Highly Sparse Sensory Data

Among the many medical imaging modalities, CT and MRI scans are utilized most often for imaging bone and soft tissue respectively. As such, physicians often require both images to fully diagnose patients and determine treatment plans. To better harness these advantages, this work has developed a deep-learning model for creating hybrid CT-MRI images from highly sparse sensory data. The key component of this technology utilizes the assumption that each hybrid, CT, and MRI image can be reconstructed from a shared latent feature space. Therefore, the training set only requires a mini-batch of CT and MRI images each which is fed into the cross-modal auto encoder framework. Preliminary testing has shown that analytic reconstruction of just the CT or MRI images show large streak artifacts due to under-sampled data, but the hybrid image has high similarity with the true images. The degree of fusion between CT and MRI can also be tuned using a fusion hyperparameter allowing physicians to create the most useful image.

Stage of Research