As a researcher in the field of electrical engineering, I have focused my work on the reliable diagnosis of photovoltaic systems. The increasing deployment of solar panels in both centralized and distributed power stations demands efficient and accurate inspection methods. Traditional manual patrols are time-consuming, costly, and sometimes dangerous, especially in remote or complex terrains. In this work, I have explored deep learning based approaches to automatically detect hot spot defects in infrared images of solar panels. Hot spots are critical defects that arise from partial shading, soiling, cell mismatch, or internal circuit faults. They severely reduce the power output and may lead to permanent damage or even fire hazards. My research targets the development of robust detection algorithms that can operate in real time with limited computational resources. In this article, I present a series of methods that progressively improve the detection of hot spots in solar panels.
Introduction and Background
The global push towards renewable energy has accelerated the deployment of solar photovoltaic systems. According to recent statistics, China installed more than 128.9 GW of solar capacity in the first three quarters of 2023 alone. This massive scale brings enormous operational challenges. Solar panels are exposed to harsh environmental conditions for decades. Dust, bird droppings, leaves, and snow can partially block incident sunlight, causing the affected cells to act as resistive loads. The current flowing through these shaded cells generates heat, resulting in localized temperature rise, known as the hot spot effect. The phenomenon is dangerous because it can crack the glass, melt the encapsulant, and degrade the entire module.
The main objective of my research is to develop accurate, fast, and lightweight detection models that can identify hot spots from infrared images. Infrared thermography captures the temperature distribution of solar panels, where hot spots appear as bright regions with high gray levels. However, in complex backgrounds, many objects such as building edges, metal frames, and reflective surfaces also emit heat and appear bright in infrared images. This similarity leads to false positives. My work addresses these challenges through several strategies: dataset construction, lightweight network design, segmentation followed by detection, and novel detection head architectures.
Dataset Creation and Preprocessing
Public infrared image datasets of solar panels are scarce. To conduct my experiments, I built an acquisition platform using a YSCP01-07 infrared camera. I captured images of healthy and defective photovoltaic panels under natural illumination. A total of 672 images were collected, of which 482 valid images contained clearly visible hot spots. These images were carefully screened to exclude blurred or redundant frames. The dataset includes diverse hot spot sizes, shapes, and locations on the panels. Some hot spots are caused by shading from leaves or bird droppings, while others result from cell cracks or solder failures.
Since the dataset is relatively small for deep learning, I applied several data augmentation techniques. The primary augmentation methods were geometric transformations, including rotation, flipping, scaling, and translation. Color adjustments such as contrast and brightness variations were also used to simulate different lighting conditions. Additionally, I added Gaussian noise to improve robustness. To further increase the training samples, I used Mosaic and Mixup augmentations. Mosaic combines four random images into one composite image, which enriches the context and background diversity. Mixup blends two images through linear interpolation. I compared the two techniques for downstream hotspot detection. The experimental results showed that Mosaic produced more reliable bounding box labels and better detection performance. Therefore, I adopted Mosaic augmentation as a fixed component of the training pipeline.
All images were annotated using the LabelImg tool. Two classes were defined: “hot spot” and “illumination effect”. The latter represents bright areas caused by non-uniform reflection, which are often mistaken for real hot spots. Every annotation was stored in a text file containing the class label and normalized bounding box coordinates. The final dataset was split into training, validation, and test sets with a ratio of 60:20:20 for the detection experiments. For the segmentation experiment, I used the Labelme tool to produce polygon annotations for solar panels. Each panel region was labeled as a single semantic class. The segmentation dataset was divided into 90% training and 10% validation.
| Stage | Number of images | Augmentation |
|---|---|---|
| Raw captured images | 672 | None |
| Valid images after screening | 482 | None |
| After applying Mosaic | 1928 | Mosaic (4-image combination) |
| After geometric and color transforms | 3374 | Rotation, flipping, scaling, brightness, noise |
Lightweight YOLO v5 for Hot Spot Detection
In the first stage of my research, I aimed to improve the speed and reduce the model size while maintaining high detection accuracy. The baseline YOLO v5S detector uses CSPDarknet53 as the backbone, which has a large number of parameters and is computationally heavy for embedded platforms. I replaced this backbone with a modified ShuffleNet-v2 network. The modification removed one redundant 1×1 convolution after the 3×3 depthwise convolution in the ShuffleNet units. This pruning reduces computational cost while preserving feature extraction capability. Additionally, I inserted the Convolutional Block Attention Module into the neck network. CBAM contains channel and spatial attention modules that adaptively refine features. The channel attention module uses both average and max pooling to capture global statistics, while the spatial attention module highlights position-relevant information. The improved network structure is illustrated in the complete model.
Let \(F\) be the input feature map of shape \(C \times H \times W\). The channel attention map is computed as:
$$
M_c(F) = \sigma \left( W_1 \left( W_0 \left( F_{avg}^c \right) \right) + W_1 \left( W_0 \left( F_{max}^c \right) \right) \right)
$$
where \(W_0\) and \(W_1\) are the weights of the shared MLP, and \(\sigma\) is the sigmoid activation. The spatial attention module uses the channel-refined feature \(F’\) as input:
$$
M_s(F) = \sigma \left( f^{7 \times 7} \left( \left[ F’_{avg}; F’_{max} \right] \right) \right)
$$
Here \(f^{7 \times 7}\) denotes a 7×7 convolution applied over the channel-wise max and average pooled features. The final refined output is produced by element-wise multiplication.
I performed ablation experiments to evaluate the contribution of each component. The experimental environment is listed in the table below.
| Operating System | Windows 10 |
| CPU | Intel(R) Core(TM) i9-12900K @ 3.19 GHz |
| GPU | NVIDIA GeForce RTX 3090 |
| Memory | 64 GB |
| Framework | PyTorch 1.12.0, YOLOv5-6.2 |
| Language | Python 3.8 |
The ablation results are summarized in the following table. I compared the baseline YOLO v5, YOLO v5 with ShuffleNet, YOLO v5 with CBAM, and the proposed combination.
| Model | Precision (%) | Recall (%) | Parameters (M) | Inference time (ms) |
|---|---|---|---|---|
| YOLO v5 | 89.1 | 97.7 | 7.02 | 147.06 |
| YOLO v5 + ShuffleNet | 79.5 | 96.5 | 5.20 | 119.05 |
| YOLO v5 + CBAM | 83.1 | 98.3 | 39.90 | 93.46 |
| Proposed | 93.5 | 95.9 | 5.30 | 92.59 |
The proposed model achieved a precision of 93.5%, a recall of 95.9%, and only 5.3 million parameters. The inference time was the lowest among all variants. A comparison with other popular detectors is also provided.
| Model | Precision (%) | Recall (%) | MAP@50 (%) | Parameters (M) |
|---|---|---|---|---|
| YOLO v5S | 89.1 | 97.7 | 98.1 | 7.02 |
| YOLO v5M | 71.1 | 98.1 | 97.8 | 20.97 |
| YOLO v5L | 83.1 | 98.3 | 98.4 | 46.27 |
| YOLO v5X | 84.6 | 98.3 | 98.6 | 86.38 |
| YOLO v3 | 81.6 | 98.1 | 97.9 | 61.67 |
| YOLO v3-spp | 78.8 | 98.8 | 97.9 | 62.71 |
| Proposed | 93.5 | 95.9 | 98.3 | 5.37 |
The qualitative results confirmed that the improved model correctly identified all hot spots on typical test images, while the baseline YOLO v3 and YOLO v5 missed some tiny hot spots. The attention mechanism helped the network concentrate on meaningful regions and suppress background clutter.
Deeplab-YOLO: A Segmentation-then-Detection Framework
Although the lightweight YOLO v5 achieved good accuracy on relatively simple backgrounds, real-world solar power plants often have complex backgrounds. Buildings, pipelines, and metal structures can be hotter than the solar panels and thus appear bright in infrared images. These objects may be falsely detected as hot spots. To solve this problem, I introduced a two-stage framework. The first stage is semantic segmentation using an improved Deeplabv3+ model to separate the solar panels from the background. The second stage is hot spot detection using an optimized YOLO v5 model restricted to the segmented panel regions. This design eliminates external false positives and improves detection robustness.
Improved Deeplabv3+ for Solar Panel Segmentation
The original Deeplabv3+ uses Xception as its backbone, which is very deep and computationally expensive. I replaced Xception with MobileNet-v2, a lightweight network based on inverted residual blocks and linear bottlenecks. MobileNet-v2 preserves low-level edge information while greatly reducing parameters. Additionally, I modified the Atrous Spatial Pyramid Pooling module. The original ASPP uses standard atrous convolutions with different dilation rates. I replaced them with atrous depthwise separable convolutions. Moreover, I introduced a cross-scale feature fusion strategy within the ASPP module. Each atrous output was concatenated with the input feature map before being fed to the next convolution layer. This enhances the exchange of information between different receptive fields. Finally, I added the CBAM attention mechanism at the end of the encoder to emphasize panel-relevant features.
The lightweight backbone reduces the computational cost. For a standard convolution with kernel size \(D_k \times D_k\), input channels \(M\), output channels \(N\), and feature map size \(D_i \times D_i\), the computation is:
$$
C_{std} = D_i \cdot D_i \cdot M \cdot N \cdot D_k \cdot D_k
$$
For a depthwise separable convolution, the computation is:
$$
C_{ds} = D_i \cdot D_i \cdot M \cdot D_k \cdot D_k + D_i \cdot D_i \cdot M \cdot N
$$
The ratio is:
$$
\frac{C_{ds}}{C_{std}} = \frac{1}{N} + \frac{1}{D_k^2}
$$
This ratio is much smaller than 1 for typical values, confirming the efficiency of depthwise separable convolution.
For the ASPP improvement, let \(x\) be the input feature. The atrous depthwise separable convolution with dilation rate \(r\) is expressed as:
$$
y_r = \text{Conv}_{1 \times 1} \left( \text{DConv}_{3 \times 3, rate=r} (x) \right)
$$
Then the cross-scale fusion can be written as:
$$
y = \text{Concat} \left( y_6, \; y_{12}, \; y_{18}, \; \text{AvgPool}(x) \right)
$$
The concatenated feature map is used for further processing.
I trained the segmentation model on the Labelme-annotated dataset. The loss curve showed stable convergence after about 20 epochs. The evaluation metrics are mean pixel accuracy and mean intersection over union. The experimental results are shown below.
| Model | MPA (%) | MIOU (%) | GFLOPS | Parameters (M) |
|---|---|---|---|---|
| U-Net | 93.06 | 96.25 | 452.31 | 24.89 |
| PSPNet | 91.49 | 78.62 | 6.03 | 2.38 |
| Deeplabv3+ | 92.40 | 84.68 | 53.03 | 5.82 |
| Improved Deeplabv3+ | 95.01 | 97.62 | 93.40 | 2.28 |
The improved Deeplabv3+ achieved the highest MPA of 95.01% and MIOU of 97.62%, with only 2.28 million parameters. Visual results showed that the improved model preserved fine panel edges and correctly identified the panels even when the background contained buildings and other heat sources. In contrast, the other models often over-segmented the boundaries or missed parts of the panel.
Optimized YOLO v5 for Hot Spot Detection on Segmented Panels
Once the solar panels are segmented, I performed hot spot detection on the segmented image. The problematic background objects are removed, but the hot spot sizes can vary from very small to relatively large. To address this multi-scale issue, I made three modifications to YOLO v5. First, I replaced the backbone CSPDarknet53 with MobileNet-v3. This backbone is lighter and faster while maintaining accuracy. Second, I added a fourth detection head that operates on a larger feature map of size \(128 \times 128\). The original YOLO v5 has three heads for feature maps of sizes \(19 \times 19\), \(38 \times 38\), and \(76 \times 76\). The extra small-object detection head significantly improves the sensitivity for tiny hot spots. Third, I replaced the GIOU loss with EIOU loss for bounding box regression. The EIOU loss is defined as:
$$
L_{EIOU} = L_{IOU} + L_{dis} + L_{asp}
$$
where
$$
L_{IOU} = 1 – IOU
$$
$$
L_{dis} = \frac{\rho^2(b, b^{gt})}{C^2}
$$
$$
L_{asp} = \frac{\rho^2(w, w^{gt})}{C_w^2} + \frac{\rho^2(h, h^{gt})}{C_h^2}
$$
Here, \(\rho(\cdot)\) is the Euclidean distance, \(b\) is the center point of the predicted box, \(b^{gt}\) is the ground truth center, \(C\) is the diagonal length of the minimum enclosing box, and \(C_w, C_h\) are the width and height of the minimal enclosing box. EIOU directly minimizes the differences in width and height, leading to faster convergence and better small-object localization.
I performed an ablation study using the segmented dataset. The results are shown below.
| MobileNet-v3 | Small head | EIOU | Precision (%) | Recall (%) | Parameters (M) | Inference time (ms) |
|---|---|---|---|---|---|---|
| – | – | – | 93.5 | 95.9 | 7.02 | 147.06 |
| ✓ | – | – | 96.1 | 94.3 | 1.40 | 78.13 |
| – | ✓ | – | 78.5 | 99.1 | 22.50 | 101.01 |
| – | – | ✓ | 82.0 | 100.0 | 7.01 | 147.06 |
| ✓ | ✓ | ✓ | 94.2 | 99.4 | 0.83 | 55.87 |
The full optimized model achieved the best balance of metrics. The precision reached 94.2%, recall reached 99.4%, parameters dropped to 0.83 M, and inference time was only 55.87 ms per image. The small detection head alone increased recall to 99.1% but reduced precision due to the increased number of false positives. When combined with the lightweight backbone and EIOU, the model retained high recall while regaining high precision.
A comparison with mainstream detectors on the segmented dataset is provided.
| Model | Precision (%) | Recall (%) | Parameters (M) | MAP@50 (%) |
|---|---|---|---|---|
| YOLO v5S | 93.5 | 95.9 | 7.02 | 98.1 |
| YOLO v5M | 71.1 | 98.1 | 20.97 | 97.8 |
| YOLO v5L | 83.1 | 98.3 | 46.27 | 98.4 |
| YOLO v5X | 84.6 | 98.3 | 86.38 | 98.6 |
| YOLO v3 | 81.6 | 98.1 | 61.67 | 97.9 |
| YOLO v3-spp | 78.8 | 98.8 | 62.71 | 97.9 |
| Proposed Deeplab-YOLO | 94.2 | 99.4 | 0.83 | 99.2 |
The P/R curves for the two classes indicated that the model performed exceptionally well on true hot spots, with an area near 1.0. For the class of illumination effects, the model was slightly less accurate, as expected, because this class is ambiguous and depends on the viewing angle.
Subjective tests on unseen images revealed that the original YOLO v5 missed several small hot spots and produced low-confidence detections. The Deeplab-YOLO framework, however, detected every visible hot spot with high confidence and did not confuse background heating sources with hot spots. This is because the segmentation step completely removed background pixels from the detection range.
RD-YOLO v5: A Real-Time Detector with Transformer Neck
Despite the advantages of YOLO v5, its detection head relies on non-maximum suppression. NMS is a post-processing step that cannot be fully optimized on GPU and often causes latency. The DETR family removes the need for NMS by using transformer-based object queries. However, the original DETR is slow. Recently, RT-DETR was introduced as a real-time detector that combines a transformer encoder with an efficient decoder. In this stage, I proposed a hybrid model called RD-YOLO v5. It uses the YOLO v5 framework but replaces its FPN+PAN neck with the hybrid encoder from RT-DETR.
The RT-DETR hybrid encoder consists of two components: the Attention-based Intra-scale Feature Interaction (AIFI) and the CNN-based Cross-scale Feature-fusion Module (CCFM). AIFI applies a transformer encoder to only the deepest feature map \(S_5\) to capture long-range dependencies. CCFM then fuses the features at different scales through a series of convolution-based fusion blocks. This design reduces the computational overhead while preserving powerful feature representations.
Let \(S_3, S_4, S_5\) be the feature maps extracted from the backbone at three scales. AIFI is applied to \(S_5\) as follows:
$$
Q = K = V = \text{Flatten}(S_5)
$$
$$
F = \text{Reshape} \left( \text{Attention}(Q, K, V) \right)
$$
Then CCFM processes \(\{S_3, S_4, F\}\) to produce the final multi-scale features. The fusion block in CCFM is implemented with repeated RepBlocks, enabling multi-scale information integration.
In addition, I introduced a lightweight mixed local channel attention module into the backbone. MLCA combines local and global information by performing local average pooling and global pooling on the input feature map. It then uses 1D convolution to compute channel attention weights without reducing channel dimensionality. The attention map is calculated as:
$$
M_{MLCA}(X) = \sigma \left( \text{Conv1d} \left( \text{LAP}(X) \right) + \text{Conv1d} \left( \text{GAP}(X) \right) \right)
$$
where LAP denotes local average pooling with a block size \(k_s = 5\), and GAP denotes global average pooling. The convolution kernel size \(k\) is determined adaptively from the channel number \(C\) by:
$$
k = \left| \frac{\log_2 C}{\gamma} + \frac{b}{\gamma} \right|_{odd}
$$
Here \(\gamma = 2\), \(b = 1\), and \(\left| \cdot \right|_{odd}\) rounds to the nearest odd number.
I performed ablation experiments to evaluate each modification. The results are summarized in the table below.
| Model | Precision (%) | Recall (%) | MAP@50 (%) | Inference time (ms) |
|---|---|---|---|---|
| YOLO v5 | 89.1 | 97.7 | 99.1 | 147.06 |
| YOLO v5 + Neck (RT-DETR) | 87.1 | 99.6 | 98.9 | 140.84 |
| YOLO v5 + MLCA | 89.5 | 99.2 | 99.1 | 125.00 |
| Proposed RD-YOLO v5 | 92.2 | 99.0 | 98.6 | 76.92 |
The proposed method reduced the inference time to 76.92 ms per image, roughly half of the baseline YOLO v5. The precision improved to 92.2%, while recall remained high. The MAP@50 decreased slightly from 99.1% to 98.6%, but the speed gain is significant for real-time applications.
I compared the RD-YOLO v5 with the previous Deeplab-YOLO method on the segmented dataset. This comparison is especially relevant because the segmentation step removes background interference, allowing the detector to focus on the panels.
| Model | Precision (%) | Recall (%) | Parameters (M) | MAP@50 (%) |
|---|---|---|---|---|
| YOLO v5S | 93.5 | 95.9 | 7.02 | 98.1 |
| Deeplab-YOLO (Section 4) | 94.2 | 99.4 | 0.83 | 99.2 |
| RD-YOLO v5 (Section 5) | 93.9 | 98.7 | 5.30 | 99.1 |
The RD-YOLO v5 achieved slightly lower precision and recall than Deeplab-YOLO, but it still outperformed the baseline YOLO v5 by a considerable margin. Moreover, the RD-YOLO v5 does not require NMS, which makes its inference pipeline more hardware-friendly. The parameter count is moderately large due to the transformer encoder, yet the inference time is lower, indicating that the transformer layers are efficient.
Visual inspection of the detection results confirmed that RD-YOLO v5 successfully identified hot spots in the segmented images, including several small ones that the original YOLO v5 missed. The model also produced higher confidence scores for the detected hot spots.
Experimental Evaluation and Discussion
Across all experiments, I used consistent training configurations for fair comparisons. The input image size was set to 640×640 for YOLO-based models and 512×512 for the segmentation model. The stochastic gradient descent optimizer with momentum 0.937 was employed. The initial learning rate was 0.01 for detection models and 0.0005 for the segmentation model. The batch size was set to 16 for detection and 4 for segmentation due to memory constraints. I trained all models for 100 epochs, with the segmentation model using a freezing strategy for the first 50 epochs. Every reported metric is the mean of five independent runs to reduce randomness.
The evaluation metrics used in this study are defined as follows. Precision measures the fraction of predicted hot spots that are true hot spots:
$$
P = \frac{TP}{TP + FP}
$$
Recall measures the fraction of true hot spots that are correctly detected:
$$
R = \frac{TP}{TP + FN}
$$
The mean average precision at an IoU threshold of 0.5 is denoted by MAP@50. For semantic segmentation, the mean pixel accuracy is:
$$
MPA = \frac{1}{n+1} \sum_{i=0}^{n} \frac{R_{ii}}{\sum_{j=0}^{n} R_{ij}}
$$
The mean intersection over union is:
$$
MIOU = \frac{1}{n+1} \sum_{i=0}^{n} \frac{R_{ii}}{\sum_{j=0}^{n} R_{ij} + \sum_{j=0}^{n} R_{ji} – R_{ii}}
$$
where \(n\) is the number of classes and \(R_{ij}\) represents the number of pixels of class \(i\) classified as class \(j\).
All experiments were executed on the same workstation to ensure comparability. The results demonstrate a clear trade-off between accuracy, model size, and speed. The lightweight YOLO v5 provides a compact solution for edge devices. The Deeplab-YOLO framework is ideal for complex scenes with high background interference. The RD-YOLO v5 offers a balance between speed and accuracy, especially when NMS-free inference is desired.
Conclusion and Future Work
In this article, I presented a series of deep learning approaches for detecting hot spot defects in infrared images of solar panels. I first constructed and augmented a dataset of infrared images. Then I proposed a lightweight YOLO v5 with ShuffleNet-v2 and CBAM, achieving 93.5% precision and 5.3 M parameters. Next, I designed a two-stage Deeplab-YOLO method that first segments solar panels using an improved Deeplabv3+ and then detects hot spots using an optimized YOLO v5. This approach achieved 94.2% precision and 99.2% MAP@50, effectively eliminating false detections from complex backgrounds. Finally, I introduced RD-YOLO v5, which integrates the RT-DETR neck and a lightweight MLCA attention mechanism, obtaining near-real-time inference while maintaining competitive accuracy.
The experimental results show that each proposed method has its own advantages. The lightweight model is suitable for embedded systems with limited resources. The segmentation-based method is the most accurate when the thermal background is challenging. The RD-YOLO v5 offers the best trade-off when low latency is critical. These findings provide practical solutions for the autonomous inspection of photovoltaic power plants.
For future work, I plan to expand the dataset with more diverse infrared images from actual utility-scale solar farms, including different weather conditions and various types of defects beyond hot spots. I also intend to investigate unsupervised domain adaptation and few-shot learning to handle rare defect categories. The integration of segmentation and detection into a single end-to-end architecture is another promising direction. Finally, deploying these models on edge hardware such as Jetson or mobile devices will be explored to facilitate real-time on-site inspections.
In summary, the research presented here offers efficient and accurate tools for monitoring the health of solar panels. By leveraging deep learning, the detection of hot spot defects becomes faster, safer, and more reliable, ultimately contributing to the sustainable operation of solar power systems.