Unmanned Aerial Vehicles (UAVs) integrated with lightweight visual cameras hold significant promise in renewable energy asset inspection and monitoring. This study presents an AI-assisted soiling detection methodology for solar photovoltaic (PV) panels using UAV-captured RGB images. The proposed scheme introduces an autonomous end-to-end soiling detection model for common types of soiling in solar panel installations, including bird droppings and dust. Detecting soiling, particularly bird droppings, is critical due to their pronounced negative impact on power generation, primarily through hotspot formation and their resistance to natural cleaning processes such as rain. A dataset containing aerial RGB images of PV panels with dusting and bird droppings is collected as a prerequisite. This study addresses the unique challenges posed by the small size and indistinct features of bird droppings in aerial imagery. Adopting the YOLOv5s object detection model as our baseline, we have implemented custom modifications with just two detection heads, each tailored for the dust and bird droppings classes. The proposed model significantly enhances the detection accuracy of bird droppings while maintaining comparable performance for the dust class. Our modified YOLOv5 model achieves a 13\% improvement in mean average precision (mAP50) for the bird droppings class, alongside a reduction in model size, which facilitates future edge computing applications