Multimodal AI-enhanced ship detection for mapping fishing vessels and informing on suspicious activities

Ship detection using remote sensing and data from tracking devices like Automatic Identification System (AIS) play a critical role in maritime surveillance, supporting security, fisheries management, and efforts to combat illegal activities. However, challenges such as varying ship sizes, complex backgrounds, and intentional deactivation of AIS hinder accurate mapping. This study proposes a novel multimodal framework that integrates Sentinel-1 Synthetic Aperture Radar, Sentinel-2 and higher resolution optical imagery. It features an enhanced deep learning-based ship detection model combined with an AIS matchmaking algorithm to detect and cross-reference potentially suspicious maritime activities. The detection model is based on an enhanced You Only Look Once architecture, optimized for identifying small vessels in cluttered and noisy image backgrounds. The model achieves superior performance, surpassing state-of-the-art accuracy on multiple public datasets while reducing training time by 12% compared to baseline models. To ensure transparency within the pipeline, Eigen-CAM explainability techniques were employed, while CO2 emissions were minimized during training using CodeCarbon, aligning the process with environmentally sustainable practices. Finally, the effectiveness of the pipeline was validated in a case study, successfully identifying potential ‘dark vessels’ and highlighting their possible involvement in suspicious activities.