In the context of global energy transition and the advancement of carbon neutrality goals, the photovoltaic industry has experienced rapid expansion. The International Energy Agency projects that global new solar photovoltaic installations will exceed 500 GW by 2025, with China contributing nearly half of this capacity. However, solar panels are susceptible to various defects during manufacturing, installation, and operation, such as cracks, grid breaks, and stains. These defects not only reduce power generation efficiency and shorten module lifespan but also pose safety risks. Traditional defect detection methods, relying on manual visual inspection or basic image processing, suffer from low efficiency and high miss rates, making them inadequate for intelligent operation and maintenance of large-scale solar plants. Therefore, developing efficient and accurate detection algorithms for solar panel surface defects is of paramount importance.
Deep learning-based object detection has shown great promise in industrial defect detection. Single-stage models like YOLO and two-stage models like Faster R-CNN have been applied to solar panel inspections. However, existing methods often face three common challenges: high parameter counts and computational complexity hinder deployment on resource-constrained edge devices; insufficient feature extraction for small, low-contrast defects leads to high miss rates; and poor robustness under complex illumination, occlusion, and background interference. To address these issues, we propose a lightweight detection algorithm named PV-RT-DETR (Photovoltaic Real-Time Detection Transformer), which integrates three key improvements over the baseline RT-DETR-R18 model.
Our contributions are as follows. First, we replace the original ResNet18 backbone with MobileNetV4, a state-of-the-art lightweight architecture optimized for mobile accelerators. It achieves a 43.2% reduction in parameter count and 30.6% reduction in computational cost while preserving strong multi-scale feature extraction capabilities. Second, we substitute the multi-head self-attention in the AIFI (Attention-based Internal Feature Interaction) module with a single-head self-attention mechanism. This modification enhances global contextual awareness and improves detection of small defects without increasing computational overhead. Third, we systematically optimize the upsampling and downsampling operators within the cross-scale feature fusion module. We replace nearest-neighbor interpolation with CARAFE (Content-Aware ReAssembly of Features) for upsampling and standard convolution with WTConv (Wavelet Transform Convolution) for downsampling. Through ablation studies, we identify the best combination strategy that balances detail preservation and computational efficiency.
The remainder of this paper is organized as follows. Section 2 details the proposed PV-RT-DETR architecture. Section 3 presents experimental results and ablation studies. Section 4 provides comparisons with state-of-the-art models. Section 5 concludes the paper and discusses future work.
Proposed Method
Overall Architecture
The overall structure of PV-RT-DETR is based on the RT-DETR framework, which consists of a backbone, an encoder (including the AIFI module and cross-scale feature fusion module), and a decoder with auxiliary prediction heads. The baseline model uses ResNet18 as the backbone. In our improved version, we replace it with MobileNetV4-conv-small, modify the AIFI module, and adjust the sampling operators in the CCFM (Cross-scale Feature Fusion Module). The following subsections describe each improvement in detail.
MobileNetV4 Backbone
MobileNetV4 introduces the Universal Inverted Bottleneck (UIB) block, which generalizes the classic inverted residual structure by optionally placing depthwise convolutions before the expansion layer or between expansion and projection layers. This design allows flexible trade-offs between spatial mixing, channel mixing, receptive field size, and computational efficiency. The UIB block supports four configurations, enabling hardware-efficient utilization on mobile accelerators. Additionally, MobileNetV4 incorporates an optimized multi-query attention mechanism that shares key and value projections across multiple query heads, reducing memory access and latency. Neural architecture search (NAS) is employed to automatically optimize layer parameters and structure, achieving high accuracy with minimal computational cost. We adopt the smallest variant, MobileNetV4-conv-small, which contains only 4.3M parameters in the backbone (our overall model with heads reaches 10.7M). This backbone significantly reduces the model footprint while maintaining competitive feature representation, as validated by our experiments.
Improved AIFI Module with Single-Head Self-Attention
The original AIFI module uses multi-head self-attention (MHSA) to enhance interactions between deep features. However, for small defect detection, the multi-head design introduces redundancy and insufficient attention to fine-grained details. We replace it with single-head self-attention (SHSA), which reduces computational complexity and focuses on global contextual information more efficiently. The SHSA mechanism is formulated as follows:
Given an input feature map \(X\), we first split it into attention part \(X_{att}\) and residual part \(X_{res}\) along the channel dimension:
$$
X_{att}, X_{res} = \text{Split}(X, C – C_p, C_p)
$$
where \(C\) is the total number of channels and \(C_p\) is the number of channels assigned to the attention branch. Then, the attention output is computed using a single-head self-attention:
$$
\tilde{X}_{att} = \text{Attention}(X_{att}W_Q, X_{att}W_K, X_{att}W_V)
$$
$$
\text{Attention}(Q,K,V) = \text{Softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V
$$
Finally, the output is obtained by concatenating the attention features and residual features, followed by a linear projection:
$$
\text{SHSA}(X) = \text{Concat}(\tilde{X}_{att}, X_{res}) W_O
$$
The key advantage of SHSA is that it aggregates global context with minimal parameters, and the residual connection preserves local details. This is particularly beneficial for detecting small defects such as thin cracks and grid breaks that require both global scene understanding and local texture awareness.
Optimization of Sampling Operators
In the cross-scale feature fusion module (CCFM), original RT-DETR uses nearest-neighbor interpolation for upsampling and standard convolution (stride=2) for downsampling. These naive operators cause loss of fine details and high parameter redundancy. We replace them with CARAFE (upsampling) and WTConv (downsampling). To determine the best replacement strategy, we design three candidate methods:
- Method A: Replace both upsampling and both downsampling operations entirely with CARAFE and WTConv respectively.
- Method B: Replace only the upsampling operation that enlarges the S4 feature map (the middle-level feature) with CARAFE, keep nearest-neighbor for the other upsampling (from F5 to S5), and replace both downsamplings with WTConv.
- Method C: Replace only the upsampling on S4 with CARAFE, and replace only the downsampling on S3 (the shallow feature) with WTConv, keeping the original operators for other branches.
Our experiments show that Method C achieves the best balance. The detailed comparison is presented in Table IV. For completeness, we describe the two operators.
CARAFE Upsampling: The CARAFE operator consists of two modules: kernel prediction module and content-aware reassembly module. Given an input feature map of size \(H \times W \times C\), it first compresses channels to \(C_m\) via a 1×1 convolution, then uses a content encoder (a convolution with kernel size \(k_{encoder}\)) to predict upsampling kernels of size \(k_{up} \times k_{up}\) for each target location. The kernels are normalized with Softmax. In the reassembly step, for each target position \(l’\), the corresponding source location \(l\) is determined, and the local neighborhood \(\mathcal{N}(X, k_{up})\) around \(l\) is weighted by the predicted kernel to produce the output at \(l’\):
$$
X’_{l’} = \sum_{n=-r}^{r} \sum_{m=-r}^{r} W_{l’}(n,m) \cdot X(i+n, j+m)
$$
where \(r = \lfloor k_{up}/2 \rfloor\). This content-aware mechanism significantly improves detail reconstruction compared to fixed interpolation.
WTConv Downsampling: WTConv (Wavelet Transform Convolution) decomposes the input feature map into low-frequency and high-frequency sub-bands using the Haar wavelet transform. The filters are:
$$
f_{LL} = \frac{1}{2}\begin{pmatrix}1 & 1 \\ 1 & 1\end{pmatrix}, \quad f_{LH} = \frac{1}{2}\begin{pmatrix}1 & -1 \\ 1 & -1\end{pmatrix}, \quad f_{HL} = \frac{1}{2}\begin{pmatrix}1 & 1 \\ -1 & -1\end{pmatrix}, \quad f_{HH} = \frac{1}{2}\begin{pmatrix}1 & -1 \\ -1 & 1\end{pmatrix}
$$
After applying these filters, we obtain four sub-bands: \(X_{LL}, X_{LH}, X_{HL}, X_{HH}\), each at half spatial resolution. Small kernel depthwise convolutions are applied to each sub-band. Then an inverse wavelet transform reconstructs the output. This approach expands the receptive field exponentially without increasing parameters, and by separating frequency components, it enhances robustness to low-frequency noise and shape perception. It is particularly effective for detecting defects with varying scales and textures in solar panel images.
Experiments
Dataset and Preprocessing
We use a dataset sourced from the Baidu PaddlePaddle AI Studio platform, originally containing 600 solar panel defect images. We apply data augmentation techniques including horizontal flipping and random rotation to expand the dataset to 2,400 images. Annotations are performed using LabelImg, and the labels include three defect categories: Crack, Grid (break), and Spot (stain). The dataset is split into training, validation, and test sets with a ratio of 7:2:1.
Experimental Setup
All experiments are conducted on a workstation with an NVIDIA RTX 4060 Ti GPU (8GB VRAM), Intel i5-12600KF CPU, Windows 10, Python 3.9.21, PyTorch 2.5.1, and CUDA 11.3. Key training hyperparameters are listed in Table I.
| Parameter | Value |
|---|---|
| Input image size | 640×640 |
| Training epochs | 200 |
| Batch size | 4 |
| Number of workers | 4 |
| Optimizer | AdamW |
| Loss function | GIoU |
| Initial learning rate | 0.0001 |
| Final learning rate factor | 1 |
| Learning rate momentum | 0.9 |
| Weight decay | 0.0001 |
Evaluation Metrics
We adopt the COCO evaluation protocol. Metrics include: Precision (P), Recall (R), F1-score, average precision at IoU=0.5 (AP50) for each class, mean AP50 (mAP50), number of parameters (Params), GFLOPs, and inference speed (FPS). Formulas are:
$$
P = \frac{TP}{TP+FP}, \quad R = \frac{TP}{TP+FN}
$$
$$
F1 = 2 \times \frac{P \times R}{P + R}
$$
$$
AP = \int_0^1 P(R) dR
$$
$$
mAP = \frac{1}{N} \sum_{i=1}^N AP_i
$$
$$
FPS = \frac{N_{\text{total}}}{T_{\text{total}}}
$$
We also use GradCAM++ for visual explanation, which improves upon GradCAM by incorporating higher-order gradients. The saliency map for class \(c\) is computed as:
$$
L_c^{\text{GradCAM++}} = \text{ReLU}\left( \sum_k w_k^c A_k \right)
$$
where \(w_k^c = \sum_i \sum_j \alpha_{ij}^{kc} \cdot \text{ReLU}\left(\frac{\partial Y^c}{\partial A_k^{ij}}\right)\) and \(\alpha_{ij}^{kc}\) is a spatial weight coefficient.
Ablation Studies
We conduct ablation experiments to validate each improvement module. The baseline is RT-DETR-R18. Module A: backbone replaced by MobileNetV4; Module B: SHSA in AIFI; Module C: sampling operator optimization using CARAFE+WTConv (Method C). Results are shown in Table II.
| Method | AP(Crack)/% | AP(Grid)/% | AP(Spot)/% | mAP50/% | Params/M | GFLOPs |
|---|---|---|---|---|---|---|
| Baseline | 92.2 | 94.4 | 94.3 | 93.6 | 19.9 | 56.9 |
| +A | 90.1 | 94.1 | 97.2 | 93.8 | 11.3 | 39.5 |
| +B | 91.1 | 95.1 | 96.6 | 94.3 | 19.7 | 57.0 |
| +C | 94.4 | 95.7 | 96.2 | 95.4 | 19.4 | 55.5 |
| +A+B | 92.1 | 94.7 | 96.1 | 94.3 | 11.1 | 39.5 |
| +A+C | 95.2 | 96.1 | 97.1 | 96.1 | 10.8 | 38.0 |
| +B+C | 96.5 | 94.4 | 96.6 | 95.8 | 19.2 | 55.5 |
| Full (A+B+C) | 97.2 | 94.5 | 97.2 | 96.3 | 10.7 | 38.1 |
The full model achieves mAP50 of 96.3% with only 10.7M parameters and 38.1 GFLOPs, demonstrating significant improvements while reducing model size by 46.2% compared to baseline. Each module contributes positively; in particular, the combination of A and C yields a large boost, indicating that the lightweight backbone and optimized sampling operators are complementary.
Backbone Comparison
To validate the effectiveness of MobileNetV4, we fix the other improvements (B and C) and replace the backbone with various popular lightweight architectures. Results are shown in Table III.
| Backbone | P/% | R/% | F1/% | mAP50/% | Params/M |
|---|---|---|---|---|---|
| ResNet18 | 94.4 | 93.4 | 93.9 | 95.8 | 19.2 |
| StarNet | 92.1 | 90.4 | 91.2 | 92.6 | 11.3 |
| EfficientViT | 91.8 | 93.0 | 92.3 | 93.5 | 10.1 |
| FasterNet | 92.9 | 94.3 | 93.6 | 94.6 | 10.2 |
| ConvNextV2 | 91.5 | 91.9 | 91.6 | 94.3 | 11.7 |
| MobileNetV4 (Ours) | 95.6 | 95.2 | 95.4 | 96.3 | 10.7 |
MobileNetV4 achieves the best F1 (95.4%) and mAP50 (96.3%) among all backbones, with competitive parameter count. This confirms that the specialized UIB blocks and multi-query attention provide superior feature extraction for solar panel defect detection.
Sampling Operator Strategy Comparison
We evaluate three replacement strategies for sampling operators (Section 2.4) while keeping the backbone and AIFI improvements fixed. Results are in Table IV.
| Strategy | P/% | R/% | F1/% | mAP50/% | Params/M |
|---|---|---|---|---|---|
| Method A | 92.2 | 93.0 | 92.5 | 93.7 | 10.2 |
| Method B | 93.0 | 92.8 | 92.8 | 94.1 | 10.1 |
| Method C (Ours) | 95.6 | 95.2 | 95.4 | 96.3 | 10.7 |
Method C yields the highest P, R, F1, and mAP50. Method B has the fewest parameters, but its accuracy is considerably lower. This indicates that selectively applying CARAFE on the middle-level upsampling and WTConv on shallow downsampling effectively preserves high-level semantics while enhancing detail capture for shallow features. Method A, which applies both operators globally, suffers from information loss in high-level feature maps.
To visually verify the effectiveness, we generate GradCAM++ heatmaps for the three strategies. An example solar panel defect detection result is shown below. The heatmap reveals that Method C focuses more precisely on the actual defect regions (cracks and grid breaks) and effectively suppresses background clutter, whereas Method A and B show diffused attention or missed detections.
Comparison with State-of-the-Art Models
We compare PV-RT-DETR with several mainstream object detection models under the same experimental conditions. Table V summarizes the results.
| Model | P/% | R/% | F1/% | mAP50/% | Params/M | FPS |
|---|---|---|---|---|---|---|
| Faster-RCNN (ResNet50) | 91.9 | 89.7 | 90.8 | 91.3 | 28.3 | 16.1 |
| YOLOv5-DETR | 88.5 | 89.6 | 89.0 | 90.1 | 5.6 | 183.6 |
| YOLOv8-DETR | 94.9 | 92.8 | 93.8 | 95.2 | 6.1 | 189.9 |
| YOLOv9m | 90.0 | 87.3 | 88.6 | 92.2 | 16.7 | 94.1 |
| YOLOv10m | 79.3 | 83.2 | 81.2 | 88.3 | 15.3 | 108.8 |
| YOLO11m | 93.1 | 90.7 | 91.8 | 95.3 | 20.0 | 90.7 |
| YOLO12m | 88.6 | 92.8 | 90.6 | 95.0 | 19.6 | 63.1 |
| RT-DETR-R18 (Baseline) | 92.5 | 91.5 | 92.0 | 93.6 | 19.9 | 105.8 |
| PV-RT-DETR (Ours) | 95.6 | 95.2 | 95.4 | 96.3 | 10.7 | 88.1 |
Our model achieves the highest mAP50 (96.3%) and F1-score (95.4%), with a parameter count of only 10.7M—53.8% of the baseline. Although the FPS (88.1) is lower than some lightweight YOLO variants, it still meets real-time requirements (≥30 FPS). Importantly, our model surpasses larger models like YOLOv9m and YOLO11m in both accuracy and efficiency. The bubble chart analysis (not shown due to space) further confirms that PV-RT-DETR strikes the best trade-off among accuracy, F1-score, and model size.
Conclusion
We have proposed PV-RT-DETR, a lightweight real-time object detector specifically designed for solar panel defect detection. By replacing the backbone with MobileNetV4, adopting single-head self-attention in the AIFI module, and optimizing sampling operators with CARAFE and WTConv (using a selective replacement strategy), our model achieves a mAP50 of 96.3% and an F1-score of 95.4%, with only 10.7M parameters and 38.1 GFLOPs. Compared to the baseline RT-DETR-R18, the parameter count is reduced by 46.2% while detection accuracy improves by 2.7% in mAP50 and 3.4% in F1-score. Extensive experiments demonstrate that our method outperforms multiple state-of-the-art YOLO variants and maintains real-time inference speed.
Despite these promising results, there are limitations. Our dataset covers only three common defect types (crack, grid break, stain) and does not include more complex defects such as potential-induced degradation (PID) or hot spots. The model may not generalize well to all solar panel environments. In future work, we plan to construct a more comprehensive dataset containing diverse defect types and varying imaging conditions. We also aim to deploy the model on edge devices for real-time monitoring and explore domain adaptation techniques to handle cross-station variability. Additionally, further architectural improvements such as knowledge distillation or pruning could be investigated to reduce the parameter count even further while preserving accuracy.