Retinal disease classification from bimodal OCT and OCTA using a CNN-ViT hybrid architecture

Retinal diseases are the leading cause of vision impairment and blindness worldwide. Early and accurate diagnosis is critical for effective treatment, and recent advances in imaging technologies such as Optical Coherence Tomography (OCT) and OCT Angiography (OCTA), have enabled detailed visualization of the retinal structure and vasculature. By leveraging these modalities, this study proposes an advanced deep learning architecture called MultiModalNet for automated multi-class retinal disease classification. MultiModalNet employs a dual-branch design, where OCTA projection maps are processed through a ResNet101 encoder, and cross-sectional slices from the OCT volume (B-scans) are analyzed using a Vision Transformer (ViT-Large). The extracted features from both branches were fused and passed through the fully connected layers for the final classification. Evaluated on the 3-class OCTA-500 dataset, which includes Age-related Macular Degeneration (AMD), Diabetic Retinopathy (DR), and Normal cases, the proposed model achieved state-of-the-art classification accuracy of 94.59 percent, significantly o utperforming single-modality baselines. This result highlights the effectiveness of integrating vascular and structural information to improve the diagnostic performance. The findings suggest that hybrid multi-modal deep learning approaches can play a transformative role in computer-aided ophthalmology, enhancing both clinical decision-making and screening workflows.