1. Introduction and Research Background
The global energy landscape is undergoing a significant transformation, with renewable energy sources playing an increasingly critical role in addressing climate change and reducing dependence on fossil fuels. Among these, solar energy has emerged as one of the most promising and rapidly expanding sectors. According to the International Energy Agency (IEA), the global photovoltaic (PV) installed capacity reached a cumulative total of approximately 1,629 GW by the end of 2023, with China contributing nearly 40% of this capacity. The continuous growth of the solar industry, however, brings forth significant operational challenges, particularly in the maintenance and inspection of solar panels.
The efficiency of solar panels is directly impacted by the accumulation of foreign substances on their surfaces. Dust, sand, bird droppings, fallen leaves, and other debris can significantly reduce the amount of sunlight reaching the photovoltaic cells, leading to substantial losses in power generation. Research has demonstrated that even light dust accumulation can reduce power output by several percent, while severe contamination can cause losses exceeding 20%. More critically, the uneven heating caused by surface contaminants can lead to the “hot spot” effect, which may cause permanent damage to the solar panels, including cell cracking, solder joint failure, and in extreme cases, fire hazards. The challenge of maintaining clean solar panels is exacerbated by the large-scale deployment of PV power stations, often located in remote, dusty, or harsh environmental conditions, making manual inspection labor-intensive, time-consuming, and costly.
Traditional methods of PV panel inspection rely on either physical inspection by trained personnel or the analysis of electrical characteristics such as voltage-current (I-V) curves. While the electrical approach can detect anomalies, it cannot precisely localize the type and position of foreign objects without additional instrumentation. The visual inspection approach, particularly when combined with unmanned aerial vehicles (UAVs), offers a scalable and efficient alternative. However, the application of deep learning for automated foreign object detection on solar panels is still maturing, facing challenges such as complex backgrounds, small object sizes, imbalanced datasets, and the need for real-time processing on edge devices. This thesis addresses these challenges by developing a comprehensive intelligent inspection framework that combines UAV-based data acquisition, advanced generative image augmentation, a high-precision detection network, and a lightweight model optimized for real-world deployment.
2. UAV-Based Inspection System Design
The foundation of an effective intelligent inspection system for solar panels is a robust data acquisition platform. In this work, I designed a comprehensive UAV-based inspection system specifically tailored for photovoltaic plants. The system architecture was designed with several key requirements in mind: high-resolution image acquisition, extended flight endurance, autonomous flight capabilities, and efficient data transmission.
2.1 Hardware System Architecture
The UAV inspection system is categorized into three primary components: the UAV flight platform, the payload system, and the ground control station. The flight platform includes the airframe, motors, electronic speed controllers (ESCs), flight controller, and onboard processor. The payload system comprises the gimbal-stabilized camera, figure transmission modules, and positioning sensors. The ground control station consists of a remote controller, a high-performance personal computer, and communication links for real-time data exchange.
For this study, I selected the DJI Matrice 350 RTK multi-rotor UAV as the flight platform. This UAV was chosen based on its exceptional flight performance, robust build quality, and extensive sensor suite. Table 1 summarizes the key specifications of the selected platform.
| Parameter | Value |
|---|---|
| Aircraft Weight (with batteries) | 6.47 kg |
| Motor Wheelbase | 895 mm |
| Max Flight Time | 45 minutes |
| Max Flight Altitude | 5000 m |
| Max Takeoff Weight | 9.2 kg |
| Max Flight Speed | 23 m/s |
| GPS Hovering Accuracy | ±0.1 m (horizontal), ±0.1 m (vertical) |
| Operating Temperature | -20°C to 50°C |
The imaging payload selected for this research is the Zenmuse P1 camera, known for its high-resolution sensor, integrated three-axis gimbal stabilization, and compatibility with multiple lens options. The camera specifications, detailed in Table 2, were critical for capturing the fine details necessary to detect small foreign objects on solar panels.
| Parameter | Value |
|---|---|
| Dimensions | 198 × 166 × 129 mm |
| Sensor | 45 MP effective pixels |
| System Power Consumption | 20 W |
| Aperture Range | f/2.8 – f/16 |
| Operating Temperature | -20°C to 50°C |
| Flight Accuracy | Planimetric: 3 cm; Elevation: 5 cm |
2.2 Global Path Planning and Inspection Process
To maximize the efficiency of the inspection mission, I designed a global path planning strategy tailored to the layout of the solar panel arrays. Considering the distribution of photovoltaic panels in the target PV station, the “S” shaped path was selected over the “rectangular spiral” path. The “S” shaped route minimizes redundant coverage and ensures the UAV captures images with consistent overlap, which is essential for constructing uniform datasets. The complete inspection process is as follows: the UAV begins at a designated point, follows the pre-planned “S” pattern over the PV array, captures images with the camera oriented perpendicular to the panels, and transmits the data in real-time to the ground station for immediate processing or future model training.
3. Dataset Construction and Data Augmentation
A high-quality dataset is the most critical factor in training a robust object detection model. In this research, I constructed a custom dataset of foreign objects on solar panels using the UAV inspection system described above. Data collection was conducted at a rooftop PV platform and a commercial PV plant under varying lighting and weather conditions to ensure diversity.
3.1 Dataset Compilation and Annotation
After several rounds of data screening, the final dataset comprised 900 images containing three primary classes of contaminants: leaves, dust/grime, and bird droppings. These images were annotated using the LabelImg tool, which generates VOC-format XML files. The dataset was strategically partitioned into training (80%), validation (10%), and test (10%) sets. The data flow diagram in Figure 2 illustrated how the data was processed, starting from raw UAV images, through annotation and preprocessing, to the final structured dataset used for model training. The initial dataset was relatively small, which is a common problem in specialized detection tasks. To address this limitation, I implemented a two-pronged data augmentation strategy combining traditional data augmentation techniques with a novel deep learning-based generative approach.
3.2 Traditional Data Augmentation
Traditional augmentation methods were divided into geometric and photometric transformations to increase the diversity of the training data. These methods are computationally efficient and help improve the model’s invariance to certain transformations. However, they do not add new structural information to the dataset. An example of the traditional data augmentation output demonstrated the effects of random translation, rotation, and flipping operations. It became clear that while these methods are useful, they are insufficient for maximizing model generalization.
3.3 Improved DCGAN for Data Augmentation
To synthesize new, realistic samples and enhance the native diversity of the dataset, I turned to Generative Adversarial Networks (GANs). Specifically, I improved the Deep Convolutional GAN (DCGAN) architecture.
The core of the GAN framework involves a min-max game between a generator (G) and a discriminator (D). The objective function is:
$$\min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{data}(x)}[\log D(x)] + \mathbb{E}_{z \sim p_z(z)}[\log(1 – D(G(z)))]$$
The vanilla DCGAN improved the original GAN by introducing convolutional layers, removing fully connected layers, and incorporating batch normalization. However, to further enhance the quality of generated images, I proposed three key improvements to the DCGAN architecture:
- Self-Attention Mechanism in the Generator: This mechanism allows the network to capture global dependencies within an image. The self-attention feature map is calculated as:
$$\beta_{j,i} = \frac{\exp(f(x_i)^T g(x_j))}{\sum_{i=1}^N \exp(f(x_i)^T g(x_j))}$$
where \(f\) and \(g\) are linear transformations of the input feature map. This enables the generator to produce images with more coherent structures rather than just local textures. - SE (Squeeze-and-Excitation) Attention Blocks: I incorporated SE blocks into both the generator and discriminator. These blocks adaptively recalibrate channel-wise feature responses, allowing the network to focus on the most informative features. The excitation operation is defined as:
$$s = \sigma(W_2 \delta(W_1 z))$$
where \(z\) is the result of global average pooling, \(\delta\) is the ReLU function, and \(\sigma\) is the sigmoid function. - SELU Activation Function: The Scaled Exponential Linear Unit (SELU) was used to replace the Leaky-ReLU in the discriminator. The SELU activation function is characterized by:
$$\text{SELU}(x) = \lambda \begin{cases} x & \text{if } x > 0 \\ \alpha e^x – \alpha & \text{if } x \leq 0 \end{cases}$$
where \(\lambda \approx 1.0507\) and \(\alpha \approx 1.6732\). This function helps to normalize the activations within the network, improving training stability and reducing the risk of gradient explosion.
The architecture of the improved generator and discriminator was detailed in the original work, with the generator incorporating dense layers, transposed convolutions, self-attention, and SE blocks, while the discriminator used a series of convolutions with SELU activations, SE blocks, and a final dense layer for classification.
3.4 Experimental Validation of Augmentation
I performed experiments using the constructed dataset to evaluate the performance of the GANs. The experimental environment consisted of an Intel i5-13490KF CPU, an NVIDIA RTX 4060 GPU, and the Windows 11 operating system. The models were trained for 8000 iterations with a learning rate of 0.001 and a batch size of 32.
To quantitatively assess the quality of the generated images, I used several metrics, including the Inception Score (IS), Fréchet Inception Distance (FID), Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index (SSIM). The FID is a particularly robust metric defined as:
$$FID = ||\mu_r – \mu_g||^2 + Tr(\Sigma_r + \Sigma_g – 2(\Sigma_r \Sigma_g)^{1/2})$$
where \(\mu\) and \(\Sigma\) are the mean and covariance of the real (r) and generated (g) feature vectors.
The results of my experiments were compelling. The improved DCGAN achieved a significantly better FID score of 276.1, compared to 322.05 for the standard DCGAN and 405.54 for the original GAN. Table 3 shows a detailed quantitative comparison of the generated image quality.
| Model | IS | FID | SSIM | PSNR |
|---|---|---|---|---|
| GAN | 1.12 | 405.54 | 0.181 | 15.6 |
| DCGAN | 1.30 | 322.05 | 0.204 | 25.4 |
| Improved DCGAN | 1.76 | 276.1 | 0.3837 | 36.4 |
The visual quality of the generated images also improved substantially. The improved DCGAN could generate diverse leaf images with varied colors and shapes, and bird droppings with more distinct characteristics, significantly improving the data diversity of foreign objects on solar panels. By combining traditional augmentation and the improved DCGAN, the final dataset was expanded from 900 to 6,546 images. The expanded dataset subsequently led to a notable enhancement in the baseline model’s performance, increasing the mAP from 83.3% to 89.6% and Recall from 82.3% to 87.4%.
4. High-Accuracy Foreign Object Detection Algorithm
With a robust dataset in place, the next critical phase was the development of a high-accuracy object detection algorithm capable of identifying foreign materials on solar panels in complex backgrounds. The images captured by UAVs often feature reflective surfaces, varying lighting conditions, and small-sized objects, all of which complicate the detection task.
4.1 Baseline Model: YOLOv8s
I selected YOLOv8s as the baseline model due to its excellent balance of speed and accuracy. YOLOv8 builds upon the strengths of previous YOLO versions by introducing a redesigned backbone (C2f module replacing C3), a decoupled detection head, and anchor-free detection mechanisms, which streamline the detection pipeline and improve performance. However, my initial experiments on the solar panel dataset revealed that the baseline YOLOv8s still suffered from missed detections and suboptimal precision, particularly for small objects in the presence of complex noise.
4.2 Architectural Enhancements: The BiFB-YOLO Model
To address the shortcomings of the baseline model, I proposed a novel detection network called BiFB-YOLO. This model integrates three major improvements over the standard YOLOv8s:
1. BiFPN-Concat Feature Fusion: The original PANet (Path Aggregation Network) is replaced with a weighted bidirectional feature pyramid network (BiFPN) integrated into the Concat operation. This BiFPN-Concat module introduces learnable weights to different input features, allowing the network to prioritize more informative features during the fusion of multi-scale data. Unlike its predecessors, this mechanism effectively fuses deep semantic information with shallow spatial details. The fusion process greatly enhances the model’s ability to detect foreign objects of varying sizes on solar panels. This strategy considers the fact that different feature maps contribute unequally to the final detection.
2. Global Attention Mechanism (GAM): To further enhance feature extraction, I inserted a GAM attention module into the backbone network. GAM is designed to enhance interaction across dimensions, preserving information to strengthen the “global” cross-dimension effects. It consists of a channel attention sub-module and a spatial attention sub-module. The channel attention sub-module involves a 3D arrangement of the input tensor, which is then processed by a multi-layer perceptron (MLP) to capture channel dependencies. The spatial sub-module uses a series of 7×7 convolutions to focus on crucial positional information. This helps the model suppress irrelevant background information (e.g., building edges, ground textures) and focus on the contaminants on the solar panels.
3. SIoU Loss Function: The standard CIoU loss was replaced with the SIoU (Smooth Intersection over Union) loss function. CIoU considers the overlap area, central point distance, and aspect ratio. However, SIoU introduces a more comprehensive approach by incorporating an angle penalty term, distance loss, shape loss, and IoU loss. The key advantage of SIoU is its consideration of the angle between the bounding boxes, which accelerates convergence and helps the model refine the location of the detected foreign objects with higher precision. The SIoU loss function \(L_{SIoU}\) is defined as:
$$L_{SIoU} = 1 – IoU + \frac{\Delta + \Omega}{2}$$
where \(\Delta\) is the distance loss (incorporating the angle cost), and \(\Omega\) is the shape loss.
These enhancements synergistically improved the model’s performance. The BiFPN-Concat provided richer multi-scale features, GAM ensured that the network focused on the most relevant features, and SIoU refined the localization. The collective impact of these modifications on detection accuracy was substantial.
4.3 Experimental Analysis and Results
I conducted a series of extensive experiments to evaluate the effectiveness of the BiFB-YOLO model. The experiments were conducted on the constructed dataset using a consistent set of hyperparameters (input size 640×640, SGD optimizer, initial learning rate of 0.01, momentum of 0.937, weight decay of 0.0005, and 200 epochs).
Ablation Study: The ablation study, presented in Table 4, systematically verified the contribution of each module.
| Model | BiFPN-Concat | GAM | SIoU | Precision (%) | Recall (%) | mAP@0.5 (%) |
|---|---|---|---|---|---|---|
| 1 | × | × | × | 89.2 | 87.4 | 89.6 |
| 2 | √ | × | × | 89.7 | 87.8 | 90.2 |
| 3 | × | √ | × | 93.5 | 89.4 | 95.7 |
| 4 | √ | √ | × | 94.3 | 90.6 | 96.5 |
| 5 | √ | √ | √ | 95.0 | 92.9 | 96.2 |
The results demonstrated that each component contributes positively to the final model. The most significant single improvement came from adding the GAM attention mechanism, which increased the mAP by 6.8% over the baseline. The final model, BiFB-YOLO, achieved a Precision of 95.0%, a Recall of 92.9%, and an mAP@0.5 of 96.2%.
Comparison with other detectors: I compared BiFB-YOLO against other popular detection frameworks, and the results are shown in Table 5. It outperformed all other models, including SSD, Faster R-CNN, and several YOLO variants, demonstrating its superiority for the foreign object detection task on solar panels.
| Model | Precision (%) | Recall (%) | mAP@0.5 (%) | mAP@0.5:0.95 (%) |
|---|---|---|---|---|
| SSD | 83.2 | 81.6 | 84.5 | 46.8 |
| Faster R-CNN | 81.6 | 82.3 | 82.9 | 47.3 |
| YOLOv3 | 81.9 | 80.3 | 84.1 | 47.2 |
| YOLOv5 | 89.2 | 86.3 | 87.4 | 52.0 |
| YOLOv7 | 88.7 | 87.2 | 88.9 | 51.8 |
| YOLOv8 | 89.2 | 87.4 | 89.6 | 59.3 |
| BiFB-YOLO | 95.0 | 92.9 | 96.2 | 67.1 |
Visual inspection of the detection results confirmed that BiFB-YOLO effectively eliminated the missed detections present in the baseline YOLOv8s and displayed higher confidence scores.
5. Lightweight Algorithm for Real-Time Deployment
The high accuracy of the BiFB-YOLO model came at the cost of increased model complexity and computational load, making it challenging to deploy on embedded platforms such as the onboard computer of a UAV. To address this, I proposed a lightweight variant, MoBiFB-YOLO, designed to maintain high detection accuracy while significantly reducing the model’s memory footprint and computational requirements, thereby enabling real-time inspection of solar panels.
5.1 Lightweight Backbone: MoECA
The core of this lightweight design is the development of a new efficient convolutional block, which I named MoECA, based on MobileNetV3. To address the limitations of the SE attention in MobileNetV3, I incorporated the Efficient Channel Attention (ECA) mechanism, which is known for its effectiveness in modeling cross-channel interactions without dimensionality reduction. The ECA mechanism achieves this through a 1-D convolution layer of kernel size \(k\), which is adaptively determined. The attention weight computation is:
$$w = \sigma(C1D_k(y))$$
where \(y\) is the input from a global average pooling operation, \(C1D\) denotes the 1D convolution, and \(\sigma\) is the sigmoid function. This lightweight ECA module is more efficient than the original SE module, as it avoids the complexity of multiple fully connected layers. The resulting MoECA block successfully balances computational efficiency and feature extraction capability.
5.2 Lightweight Detection Head: PConv-Head
The second key improvement involves optimizing the decoupled detection head. YOLOv8s uses a decoupled head with multiple branches. I introduced a partially shared convolution strategy, where early feature extraction layers are shared between the classification and regression tasks, before branching into separate paths. Furthermore, I replaced the standard convolutions in the head with PConv (Partial Convolution) to exploit spatial redundancy and reduce computation. PConv applies standard convolutions only to a subset of the input channels, leaving the rest unchanged. This strategy drastically reduces FLOPs and memory access while retaining competitive performance. The FLOPs of a PConv is:
$$FLOPs = h \times w \times c_p^2 \times k^2$$
where \(h\), \(w\) are the height and width of the feature map, and \(c_p\) is the number of channels being convolved.
5.3 Experimental Results and Analysis
I performed experiments to validate the performance of the proposed lightweight model. The comparison of different lightweight backbones integrated into the BiFB-YOLO framework is shown in Table 6.
| Model | Params (M) | FLOPs (G) | FPS (Hz) | mAP@0.5 (%) |
|---|---|---|---|---|
| BiFB-YOLO (Original) | 11.8 | 28.3 | 76.2 | 96.2 |
| + GhostNet | 8.2 | 20.6 | 89.3 | 93.1 |
| + ShuffleNetv2 | 9.7 | 23.2 | 82.4 | 90.6 |
| + MobileNetv3 | 6.3 | 13.0 | 93.6 | 92.9 |
| + MoECA (Ours) | 6.2 | 12.5 | 93.2 | 93.2 |
The ablation study for the lightweight improvements, shown in Table 7, demonstrates the effectiveness of each component. By applying both the MoECA backbone and the PConv-Head, the model achieved a good trade-off between speed and accuracy.
| Model | MoECA | PConv-Head | mAP@0.5 (%) | Params (M) | FLOPs (G) | FPS (Hz) |
|---|---|---|---|---|---|---|
| 1 | × | × | 96.2 | 11.8 | 28.3 | 76.2 |
| 2 | √ | × | 93.2 | 6.2 | 12.5 | 93.2 |
| 3 | × | √ | 94.0 | 8.8 | 16.2 | 105.1 |
| 4 | √ | √ | 95.1 | 3.8 | 7.0 | 103.1 |
The final MoBiFB-YOLO model reduced the number of parameters by 67.8% (from 11.8M to 3.8M) and FLOPs by 75.3% (from 28.3G to 7.0G) compared to the high-precision BiFB-YOLO model, while incurring only a marginal 1.1% drop in mAP (from 96.2% to 95.1%). The detection speed also improved significantly. A comparison with other state-of-the-art lightweight models confirmed that MoBiFB-YOLO offered the best overall balance of accuracy and efficiency for classifying foreign objects on solar panels.
6. Software System for Foreign Object Detection
To make the detection algorithm accessible and practical for inspection personnel, I designed and implemented a comprehensive software system based on the PyQt5 framework. This system integrates the trained MoBiFB-YOLO model, providing a user-friendly interface for uploading data, adjusting parameters, and viewing results.
6.1 System Architecture
The system architecture consists of four primary modules:
- User Authentication Module: This module handles user registration and login, ensuring system security and personalizing the user experience. The login and registration interfaces were designed to be intuitive and secure.
- Data Upload Module: This module supports the input of images and video files from local directories. It is also designed to be extensible, allowing for future integration of live camera feeds. The system can process both individual UAV-captured images and full inspection videos.
- Parameter Configuration Module: This module allows users to adjust critical detection parameters such as confidence threshold and IoU threshold to fine-tune the detection performance based on the specific operational requirements. It also provides a mechanism to load the appropriate model weights.
- Result Display Module: This module presents the detection results, including annotated images with bounding boxes and labels, the count of different foreign objects, and their distribution. The provided display shows the detection of ‘bird droppings’ and ‘leaves’ on solar panels.
6.2 System Workflow
The workflow is designed to be straightforward: a user logs in, selects an input source (image or video), loads the trained model weights, sets the detection parameters, and initiates the detection process. The system then runs the inference and displays the results. This integrated platform effectively streamlines the entire inspection process, from raw data to actionable intelligence.
7. Conclusion and Future Work
In this thesis, I have presented a comprehensive framework for the intelligent inspection of foreign objects on the surface of solar panels. The research addressed several critical challenges in the field, including efficient data acquisition, image data scarcity, detection accuracy, and model deployability. The key contributions of my research can be summarized as follows:
First, I designed and validated a complete UAV-based inspection system hardware platform, ensuring high-quality data acquisition and efficient mission execution. The integration of a global path planning strategy facilitated complete coverage of the PV plant.
Second, I developed an improved DCGAN data augmentation method that generates high-quality synthetic images of foreign objects on solar panels. The enhancements, including self-attention and SE mechanisms, effectively improved the diversity and quality of the training dataset, which in turn significantly boosted the performance of the detection models.
Third, I proposed a high-precision detection algorithm, BiFB-YOLO, which successfully improved the detection of foreign objects by integrating a weighted bi-directional feature pyramid, global attention mechanisms, and a refined loss function. The model demonstrated a significant improvement over the standard YOLOv8s, achieving an mAP of 96.2%.
Fourth, to address the constraints of real-time deployment, I developed a lightweight algorithm, MoBiFB-YOLO. By introducing the MoECA efficient convolutional network and a PConv-based lightweight detection head, the model achieved a substantial reduction in both parameter count and computational cost, with minimal impact on detection accuracy. The final model is well-suited for running on resource-constrained edge computing devices commonly found on UAVs.
Finally, I implemented a user-friendly software system that integrates the lightweight detection model, providing a practical platform for PV plant operators to monitor the cleanliness of solar panels efficiently.
For future research, several avenues can be pursued. I plan to deploy the developed detection model directly onto the UAV’s onboard computer for fully autonomous real-time inspection. Further refinement of the GAN-based augmentation could focus on generating more realistic and diverse defect types. Additionally, expanding the detection scope to include other PV panel defects, such as cracks, delamination, and the use of thermal infrared spectral data, would improve the robustness and practicality of the method. I believe that the work presented here can make a strong contribution to the sustainable operation and maintenance of solar photovoltaic systems.