Photovoltaic power generation has become a central pillar in the global transition toward clean energy. With the rapid expansion of solar power plants, especially in remote and harsh environments, the reliability of solar panels under prolonged outdoor exposure has become a critical concern. One of the most common and hazardous failure modes is the hot-spot effect, which is triggered by partial shading, dust accumulation, bird droppings, or internal cell damage. These hot spots not only reduce the energy yield but also create long-term safety risks due to localized overheating. Traditional manual inspection methods are labor-intensive, time-consuming, and often unable to deliver the required precision. In response to these challenges, I have dedicated my research to developing robust and efficient deep-learning-based detection methods for thermal defects in infrared images of solar panels. This work focuses on improving detection accuracy, reducing false positives, and balancing model complexity with inference speed. The following sections describe the dataset creation, the proposed lightweight YOLO-based algorithms, a segmentation-first detection framework, and a novel detector-enhanced approach. Throughout the study, I have leveraged convolutional neural networks, attention mechanisms, semantic segmentation, and transformer-inspired architectures to achieve high-performance hot-spot identification.
1. Dataset Construction and Preprocessing
One of the primary obstacles in researching hot-spot detection in infrared imagery is the scarcity of publicly available datasets. Most existing datasets are captured under electroluminescence or visible light, which are not directly applicable to outdoor photovoltaic inspection tasks. To address this, I built a dedicated infrared image acquisition platform using a thermal camera (YSCP01-07) and a set of solar panels with both intact and damaged cells. The acquisition process was carried out under natural sunlight conditions to preserve the real-world thermal signatures. A total of 672 raw images were collected, and after removing invalid or redundant frames, 482 images containing solar panels with at least one visible hot spot were retained for further processing.
The physical mechanism behind hot-spot formation is tightly linked to the electrical characteristics of a solar cell. Under normal operation, the photogenerated current \(I_{sa}\) is balanced by the diode dark current \(I_{b}\), the shunt current \(I_{dh}\), and the output current \(I\). This relationship is expressed as:
$$I_{sa}=I_{b}+I_{dh}+I \quad (1)$$
However, when a cell is partially shaded or soiled, the photogenerated current in that region drops significantly. Consequently, the sum of \(I_{b}\), \(I_{dh}\), and \(I\) exceeds \(I_{sa}\), causing the shaded cell to become reverse-biased. This reverse-bias condition leads to a rapid increase in local temperature, producing a hot spot. Over time, sustained hot-spot operation can irreversibly damage the solar panel, reduce its output power, and even induce fire hazards.
1.1 Annotation and Labeling
For the detection task, I used the LabelImg tool to manually annotate every hot spot and the associated illumination effect region with oriented rectangular bounding boxes. Each image was stored in the YOLO format, with label files containing normalized coordinates. For the segmentation task, I employed the Labelme tool to achieve pixel-wise polygon annotations of the solar panel areas. This dual-annotation strategy enabled me to train both detection and segmentation models on the same source data. The resulting dataset was then randomly split into training, validation, and test sets with ratios of 60%, 20%, and 20% for detection experiments, and 90% training / 10% validation for segmentation experiments.
1.2 Data Augmentation
Because the original dataset is relatively small, I applied several image augmentation techniques to improve generalization and prevent overfitting. The preprocessing pipeline included geometric transformations (horizontal/vertical flips, random scaling, rotation, and random clipping), color-space adjustments (brightness, contrast, saturation, hue), and the addition of Gaussian noise. In addition to these basic operations, I evaluated two advanced augmentation strategies: Mosaic and Mixup.
Mosaic augmentation combines four randomly selected images into one composite training sample. This technique enriches the contextual diversity of the images and effectively expands the number of objects per training instance. Mixup augmentation blends two randomly selected images with a weighted average, and the corresponding labels are also linearly combined. My comparative study showed that Mosaic augmentation yielded better performance for the hot-spot detection task than Mixup, especially in terms of bounding-box alignment and label consistency. Therefore, I adopted Mosaic as the default augmentation during all subsequent training phases.
2. Lightweight YOLO v5 for Hot-Spot Detection
To achieve rapid and accurate hot-spot recognition, I selected the YOLO v5 architecture as the base detector due to its well-balanced trade-off between speed and accuracy. The original YOLO v5 model consists of three main components: a CSPDarknet53 backbone for feature extraction, a neck network with FPN+PAN for multi-scale feature fusion, and a detection head that produces bounding boxes and class probabilities. However, the default YOLO v5 model has a considerable number of parameters, which can be problematic for deployment on embedded devices or drones with limited computational resources. Moreover, the standard network sometimes fails to detect small or low-contrast hot spots in cluttered thermal images.
To overcome these limitations, I proposed a lightweight YOLO v5 architecture. The key modifications are as follows:
- Replacing the backbone with an optimized ShuffleNet-v2: The original CSPDarknet53 was substituted with a trimmed version of ShuffleNet-v2, which is specifically designed for mobile devices. ShuffleNet-v2 uses channel split, depthwise convolutions, and channel shuffle to reduce computational cost while preserving high-level semantic features. I removed the second pointwise convolution after the depthwise convolution in the basic block, which simplified the structure and, surprisingly, improved accuracy on my hot-spot dataset.
- Integration of CBAM attention: I inserted Convolutional Block Attention Modules (CBAM) at selected positions in the neck network. CBAM sequentially applies channel attention and spatial attention to refine the feature maps. This allowed the network to focus on the most informative channels and spatial locations that correspond to hot spots, thereby suppressing irrelevant background information.
The channel attention computation is defined as:
$$M_{c}(F)=\sigma\left(MLP(AvgPool(F))+MLP(MaxPool(F))\right)$$
where \(F\) is the input feature map, \(AvgPool\) and \(MaxPool\) are global average and max pooling operations, \(MLP\) denotes a shared multilayer perceptron, and \(\sigma\) is the sigmoid activation. The spatial attention map is computed as:
$$M_{s}(F’)=\sigma\left(f^{7\times 7}\left([AvgPool(F’); MaxPool(F’)]\right)\right)$$
where \(f^{7\times 7}\) represents a convolution with a \(7\times 7\) kernel.
The overall improved network architecture is illustrated in the conceptual diagram in my original research. The backbone now consists of trimmed ShuffleNet-v2 blocks, followed by a series of CBAM-enhanced convolutions and detection heads.
2.1 Ablation Experiments
To evaluate the contribution of each improvement, I conducted a set of ablation experiments. All models were trained with the identical dataset and hyperparameters. The results are summarized in Table 1.
| Model | P (%) | R (%) | Parameters (M) | Inference time (ms) |
|---|---|---|---|---|
| YOLO v5 | 89.1 | 97.7 | 7.02 | 147.06 |
| YOLO v5 + ShuffleNet | 79.5 | 96.5 | 5.20 | 119.05 |
| YOLO v5 + CBAM | 83.1 | 98.3 | 39.90 | 93.46 |
| Proposed Lightweight YOLO v5 | 93.5 | 95.9 | 5.30 | 92.59 |
Table 1 clearly shows that the proposed lightweight model achieves the highest precision (93.5%) while maintaining a small parameter count of only 5.3 million and the lowest inference time among the tested variants. The recall value is slightly lower than that of the baseline, but the improvement in precision indicates a substantial reduction in false positives. The training loss curves also confirmed that the proposed model converged faster and more stably than the other configurations.
2.2 Comparison with Mainstream Detectors
To further validate the effectiveness of the proposed method, I compared it against several widely used object detectors, including YOLO v5S, v5M, v5L, v5X, YOLO v3, and YOLO v3-spp. The quantitative results are presented in Table 2.
| Model | P (%) | R (%) | mAP50 (%) | Parameters (M) |
|---|---|---|---|---|
| YOLO v5S | 89.1 | 97.7 | 98.1 | 7.09 |
| YOLO v5M | 71.1 | 98.1 | 97.8 | 20.97 |
| YOLO v5L | 83.1 | 98.3 | 98.4 | 46.27 |
| YOLO v5X | 84.6 | 98.3 | 98.6 | 86.38 |
| YOLO v3 | 81.6 | 98.1 | 97.9 | 61.67 |
| YOLO v3-spp | 78.8 | 98.8 | 97.9 | 62.71 |
| Proposed | 93.5 | 95.9 | 98.3 | 5.37 |
From Table 2, the proposed model achieves the highest precision and the smallest parameter size among all competing algorithms. Although the recall is slightly below that of some larger models, the overall mAP remains competitive at 98.3%, and the lightweight nature makes it highly suitable for real-time UAV-based inspection. Qualitative detection results demonstrated that the proposed method can simultaneously identify hot spots caused by different factors and is more robust than other models in complex scenes.
3. Deeplab-YOLO: A Segmentation-First Framework
Despite the success of the lightweight YOLO v5 model, a critical issue remains: in complex backgrounds, heat-absorbing objects such as rocks, soil, or building materials can appear with thermal signatures very similar to those of hot spots. This leads to false detections and missed detections. To solve this problem, I adopted a two-stage strategy that first segments the solar panel region from the infrared image, and then performs hot-spot detection only within the segmented panel area. This approach effectively suppresses background interference and significantly improves the reliability of hot-spot recognition.
3.1 Improved Deeplabv3+ for Solar Panel Segmentation
I selected the Deeplabv3+ semantic segmentation model as the basis for panel extraction. The original Deeplabv3+ architecture uses an Xception backbone and an Atrous Spatial Pyramid Pooling (ASPP) module to capture multi-scale contextual information. However, the Xception backbone is heavy and slow for photovoltaic inspection applications. Therefore, I introduced three key improvements:
- MobileNet-v2 backbone: I replaced Xception with the lightweight MobileNet-v2 network, which uses depthwise separable convolutions and linear bottlenecks to significantly reduce the number of parameters and computational cost while maintaining high segmentation accuracy.
- Improved ASPP with heterogeneous receptive-field fusion: The original ASPP module uses parallel atrous convolutions with different dilation rates. My modified ASPP replaces the standard atrous convolution with atrous depthwise separable convolutions, and further enhances feature reuse by concatenating the output of a larger-dilation layer with the input of a smaller-dilation layer. This fusion scheme improves the utilization of multi-scale information without adding excessive parameters.
- CBAM attention in the encoder: I inserted CBAM at the end of the encoder to refine the extracted features. This helps preserve the fine-grained boundary details of the solar panel and improves the segmentation accuracy.
The computation of the standard convolution is given by:
$$Q_{1}=D_{i}\times D_{i}\times M\times D_{k}\times D_{k}\times N$$
where \(D_{i}\) is the input feature-map size, \(M\) is the input channels, \(D_{k}\) is the kernel size, and \(N\) is the output channels. The depthwise separable convolution computation is:
$$Q_{2}=D_{i}\times D_{i}\times M\times D_{k}\times D_{k}+D_{i}\times D_{i}\times M\times N$$
Thus, the ratio between \(Q_{2}\) and \(Q_{1}\) is:
$$\frac{Q_{2}}{Q_{1}}=\frac{1}{N}+\frac{1}{D_{k}^{2}}$$
which clearly demonstrates the efficiency advantage of depthwise separable convolutions.
3.2 Segmentation Results and Comparisons
To evaluate the segmentation performance, I employed the mean pixel accuracy (MPA) and mean intersection-over-union (mIoU) as the primary metrics. These are defined as:
$$MPA=\frac{1}{n+1}\sum_{i=0}^{n}\frac{R_{ii}}{\sum_{j=0}^{n}R_{ij}}$$
and
$$mIoU=\frac{1}{n+1}\sum_{i=0}^{n}\frac{R_{ii}}{\sum_{j=0}^{n}R_{ij}+\sum_{j=0}^{n}R_{ji}-R_{ii}}$$
where \(R_{ii}\) is the number of correctly classified pixels of class \(i\), \(R_{ij}\) is the number of pixels of class \(i\) classified as class \(j\), and \(R_{ji}\) is the opposite case.
Table 3 shows the segmentation performance of the improved Deeplabv3+ model versus several mainstream methods.
| Model | MPA (%) | mIoU (%) | GFLOPS | Parameters (M) |
|---|---|---|---|---|
| U-Net | 93.06 | 96.25 | 452.31 | 24.89 |
| PSPNet | 91.49 | 78.62 | 6.03 | 2.38 |
| Deeplabv3+ | 92.40 | 84.68 | 53.03 | 5.82 |
| Improved Deeplabv3+ | 95.01 | 97.62 | 93.40 | 2.28 |
As shown in Table 3, the improved Deeplabv3+ model achieves the highest MPA and mIoU values, while having the smallest parameter count. The GFLOPS is higher than that of the original Deeplabv3+, but the inference speed remains adequate for offline segmentation before detection. The qualitative segmentation results in my thesis clearly demonstrate that the improved model preserves the edges of solar panels more accurately and produces fewer mis-segmented regions in complex backgrounds.
3.3 Optimized YOLO v5 for Hot-Spot Detection after Segmentation
After segmenting the solar panels, I cropped and re-annotated the resulting panel regions for hot-spot detection. To further boost detection performance, I optimized the YOLO v5 model in the following ways:
- MobileNet-v3 backbone: I replaced the CSPDarknet53 backbone with MobileNet-v3, which integrates the advantages of MobileNet-v2 and adds a hardware-aware neural architecture search design, including the h-swish activation function and SE blocks.
- Additional small-object detection head: The original YOLO v5 has three detection scales (19×19, 38×38, 76×76). I added a fourth head that operates on a 128×128 feature map to better capture tiny hot spots that are common in solar panel infrared images.
- EIoU loss: I replaced the generalized IoU (GIoU) loss with the efficient IoU (EIoU) loss, which explicitly minimizes the central distance and width/height differences between the predicted and ground-truth boxes. The EIoU loss is expressed as:
$$L_{EIoU}=1-IoU+\frac{\rho^{2}(b,b^{gt})}{C^{2}}+\frac{\rho^{2}(w,w^{gt})}{C_{w}^{2}}+\frac{\rho^{2}(h,h^{gt})}{C_{h}^{2}}$$
where \(\rho(\cdot)\) denotes the Euclidean distance, \(b\) is the center point, \(w\) and \(h\) are the width and height of the box, and \(C\), \(C_{w}\), \(C_{h}\) are the diagonal, width, and height of the smallest enclosing box, respectively.
Ablation Study of the Detection Model
I performed ablation experiments to verify the contribution of each modification. The results are summarized in Table 4.
| MobileNet-v3 | Small head | EIoU | P (%) | R (%) | Parameters (M) | Inference time (ms) |
|---|---|---|---|---|---|---|
| – | – | – | 93.5 | 95.9 | 7.02 | 147.06 |
| ✓ | – | – | 96.1 | 94.3 | 1.40 | 78.13 |
| – | ✓ | – | 78.5 | 99.1 | 22.50 | 101.01 |
| – | – | ✓ | 82.0 | 100.0 | 7.01 | 147.06 |
| ✓ | ✓ | ✓ | 94.2 | 99.4 | 0.83 | 55.87 |
Table 4 reveals that combining all three improvements yields a remarkable reduction in model parameters (0.83 million) and inference time (55.87 ms per image), while achieving a precision of 94.2% and recall of 99.4%. The small-object detection head significantly improves recall, and the MobileNet-v3 backbone contributes to a lean model without sacrificing precision.
Comparison with Mainstream Detectors after Segmentation
Using the segmented panel images as input, I compared the proposed optimized YOLO v5 with several baseline detectors. The results are shown in Table 5.
| Model | P (%) | R (%) | Parameters (M) | mAP50 (%) |
|---|---|---|---|---|
| YOLO v5S | 93.5 | 95.9 | 7.02 | 98.1 |
| YOLO v5M | 71.1 | 98.1 | 20.97 | 97.8 |
| YOLO v5L | 83.1 | 98.3 | 46.27 | 98.4 |
| YOLO v5X | 84.6 | 98.3 | 86.38 | 98.6 |
| YOLO v3 | 81.6 | 98.1 | 61.67 | 97.9 |
| YOLO v3-spp | 78.8 | 98.8 | 62.71 | 97.9 |
| Proposed after segmentation | 94.2 | 99.4 | 0.83 | 99.2 |
From Table 5, the optimized YOLO v5 demonstrates the best overall performance, with the smallest parameter count and highest mAP50. The precision-recall (P-R) curves confirmed that the model accurately distinguishes between true hot spots and non-hot-spot bright regions (such as illumination effects), and it excels in detecting very small hot-spot regions.
4. RD-YOLO v5: Enhancing the Detector with RT-DETR Components
Although YOLO-based detectors are fast, they inherently rely on non-maximum suppression (NMS) as a post-processing step. NMS is difficult to optimize and can introduce latency. Transformer-based end-to-end detectors such as DETR avoid NMS but suffer from slow convergence and heavy computational overhead. The RT-DETR architecture reconciles these issues by using a real-time hybrid encoder and an efficient decoder. I therefore proposed a novel RD-YOLO v5 network that integrates two core components of RT-DETR—the attention-based intra-scale feature interaction (AIFI) and the CNN-based cross-scale feature-fusion module (CCFM)—into the neck of YOLO v5, replacing the original FPN+PAN structure.
The AIFI module applies a single-scale transformer encoder only to the highest-level feature (S5), which significantly reduces computational redundancy. The CCFM module consists of multiple convolutional blocks that fuse features from different scales in a path-aggregation fashion. The overall fusion process is described as:
$$F=reshape(Attn(Q,K,V))$$
where \(Q\), \(K\), \(V\) are the query, key, and value matrices derived from the flattened S5 features, and \(Attn\) denotes multi-head self-attention. The CCFM then takes S3, S4, and the transformed S5 as inputs and outputs a fused multi-scale representation for the detection head.
In addition, I introduced a lightweight mixed local channel attention (MLCA) mechanism into the backbone of YOLO v5. MLCA simultaneously captures global and local spatial information within channels by dividing the feature map into small blocks and applying a one-dimensional convolution for efficient channel interaction. The kernel size \(k\) of this convolution is adaptively determined by the channel dimension \(C\) using:
$$k=\frac{\mid \log_{2}(C)\mid}{\gamma}+\frac{b}{\gamma}$$
where \(\gamma=2\) and \(b=1\) in my experiments, with an odd-number constraint to ensure symmetry.
4.1 Ablation Experiments for RD-YOLO v5
I conducted ablation experiments on the original infrared images (without segmentation) to evaluate the effect of each modification. The results are presented in Table 6.
| Model | P (%) | R (%) | mAP50 (%) | Inference time (ms) |
|---|---|---|---|---|
| YOLO v5 | 89.1 | 97.7 | 99.1 | 147.06 |
| YOLO v5 + RT-DETR Neck | 87.1 | 99.6 | 98.9 | 140.84 |
| YOLO v5 + MLCA | 89.5 | 99.2 | 99.1 | 125.00 |
| RD-YOLO v5 (Proposed) | 92.2 | 99.0 | 98.6 | 76.92 |
Although the proposed RD-YOLO v5 shows a slightly lower mAP compared with the baseline YOLO v5, its precision is notably higher (92.2% vs. 89.1%), and the inference time is dramatically reduced from 147.06 ms to 76.92 ms. This speed improvement is critical for real-time drone inspection applications. The recall values remain stable, indicating that the model does not sacrifice detection completeness for speed.
4.2 Evaluation on Segmented Panel Images
To further test the generalizability of the RD-YOLO v5 model, I applied it to the segmented solar panel images produced by the improved Deeplabv3+ model. The comparison against the baseline and the previously proposed Deeplab-YOLO method is shown in Table 7.
| Model | P (%) | R (%) | Parameters (M) | mAP50 (%) |
|---|---|---|---|---|
| YOLO v5S | 93.5 | 95.9 | 7.02 | 98.1 |
| Deeplab-YOLO (Chapter 4) | 94.2 | 99.4 | 0.83 | 99.2 |
| RD-YOLO v5 (Chapter 5) | 93.9 | 98.7 | 5.30 | 99.1 |
From Table 7, the RD-YOLO v5 model performs very close to the Deeplab-YOLO method, achieving a precision of 93.9% and mAP50 of 99.1%. Its parameter count is higher than that of Deeplab-YOLO but still lower than the original YOLO v5. The visual detection results on segmented images showed that RD-YOLO v5 effectively detected most hot spots with high confidence, although a few tiny hot spots were missed. Overall, this method offers an excellent trade-off between speed and accuracy, especially when deployed on embedded hardware without specialized optimization for NMS.
5. Discussion and Future Directions
Throughout this research, I have systematically improved the detection of hot-spot defects in infrared images of solar panels by leveraging modern deep-learning techniques. The lightweight YOLO v5 variant delivered a fast and compact solution for scenarios where computational resources are limited. The segmentation-first Deeplab-YOLO framework proved to be the most accurate among all tested approaches, because it eliminates the interference of background heat sources. The RD-YOLO v5, on the other hand, offers a promising alternative for reducing inference latency by using transformer-inspired components without NMS.
Despite the promising results, there are several limitations that warrant future investigation. First, the dataset, although self-constructed, is still relatively small and lacks the variety of real-world photovoltaic plant settings. Expanding the dataset to include more diverse weather conditions, different camera angles, and various panel types would likely improve the generalization of the models. Second, the two-stage segmentation-then-detection pipeline, although accurate, is more complex and requires two separate models. Future research could explore a unified network that performs both segmentation and detection in a single forward pass, such as an instance-segmentation-based model. Third, the application of few-shot learning and domain adaptation techniques could help handle the long-tailed distribution of hot-spot defects in real-world deployments. Finally, deploying the proposed models on edge devices, such as embedded GPUs or FPGAs, and quantizing them for low-power inference would be a practical next step.
In conclusion, this work provides a comprehensive study of hot-spot detection in infrared images of solar panels using deep learning. The proposed methods achieve state-of-the-art performance in terms of precision, recall, model size, and speed, and they have significant potential for practical photovoltaic plant inspection. I hope that the findings and technical insights presented here will contribute to the advancement of automated solar panel condition monitoring and support the broader adoption of clean and renewable energy.