In this work, we present a comprehensive deep learning framework for the automatic classification and localization of defects in solar panels. The proposed system integrates image stitching, segmentation, and neural network classification to overcome the limitations of traditional manual inspection. By utilizing unmanned aerial vehicle (UAV) acquired infrared and visible light images, we generate high-resolution panoramic views of large photovoltaic plants and accurately identify defective modules. Our method achieves high accuracy, precision, recall, and F1 scores, demonstrating its effectiveness for real-world solar panel maintenance.
Introduction
The rapid expansion of photovoltaic installations worldwide demands efficient and reliable methods for detecting defects in solar panels. Traditional manual inspection is time‑consuming, labor‑intensive, and often fails to cover large areas consistently. Computer vision techniques, especially deep learning, offer a promising alternative by enabling non‑contact, fast, and accurate defect detection. However, challenges remain due to the limited field of view of single aerial images and the need for robust classification under varying environmental conditions. To address these issues, we propose a pipeline that first stitches multiple local aerial images into a panoramic view, then segments individual solar panels using color and spatial information, and finally classifies each panel into normal or defective categories using a state‑of‑the‑art convolutional neural network. Our approach ensures that defects are not missed due to perspective limitations and provides precise localization within the solar panel array.
Image Stitching for Panoramic View Creation
Aerial images of solar panels captured by UAVs typically cover only a small portion of a large photovoltaic plant. To obtain a complete overview, we employ an image stitching algorithm based on the AKAZE (Accelerated‑KAZE) feature detector. Unlike traditional methods that rely on linear scale spaces (e.g., SIFT), AKAZE operates in a nonlinear scale space using nonlinear diffusion filtering, which preserves edge information more effectively and is robust to blur and noise. The nonlinear diffusion equation is defined as:
$$
\frac{\partial L}{\partial t} = \mathrm{div}\bigl(c(x,y,t) \cdot \nabla L\bigr)
$$
where $L$ represents the image luminance, $\mathrm{div}$ and $\nabla$ denote divergence and gradient operators, $t$ is the time parameter, and $c(x,y,t)$ is the conduction function that controls the diffusion speed. The conduction function is typically given by:
$$
c(x,y,t) = g\bigl(|\nabla L_{\sigma}(x,y,t)|\bigr),\quad g = \frac{1}{1 + \frac{|\nabla L_{\sigma}|^{2}}{\lambda^{2}}}
$$
Here, $\nabla L_{\sigma}$ is the gradient of the Gaussian‑smoothed image, and $\lambda$ is the diffusion factor. The discrete solution of the nonlinear diffusion equation is obtained using the Fast Explicit Diffusion (FED) scheme:
$$
L_{i+1} = \bigl[I + \tau A(L_i)\bigr] L_i
$$
where $I$ is the identity matrix, $A(L_i)$ is the conduction matrix at scale $i$, and $\tau$ is the time step. After constructing the nonlinear scale space, we detect keypoints by computing the Hessian matrix at each pixel and finding extrema across scales. The Hessian matrix used in AKAZE is:
$$
L_{\mathrm{Hessian}} = \sigma^{2}\bigl(L_{xx} L_{yy} – L_{xy}^{2}\bigr)
$$
where $\sigma$ is the scale coefficient, $L_{xx}$ and $L_{yy}$ are second‑order derivatives in the horizontal and vertical directions, and $L_{xy}$ is the mixed derivative. Keypoints with large Hessian responses are selected as feature points. Next, we compute descriptors for these points and perform feature matching using the k‑nearest neighbor algorithm with a distance ratio test:
$$
\frac{d_m}{d_n} < T
$$
where $d_m$ and $d_n$ are the distances to the nearest and second‑nearest neighbors, respectively, and $T$ is a threshold (typically 0.7). To filter out false matches, we apply the RANSAC (Random Sample Consensus) algorithm, which iteratively estimates the homography matrix that best aligns the matched points. Finally, we warp the images using perspective transformation:
$$
\begin{bmatrix} x’ \\ y’ \\ z’ \end{bmatrix} = \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix} \begin{bmatrix} u \\ v \\ 1 \end{bmatrix}
$$
where $(u,v)$ are the coordinates in the source image, and the transformed coordinates are $(x,y) = (x’/z’,\;y’/z’)$. The resulting warped images are then blended using bilinear interpolation to produce a seamless panoramic view of the entire solar panel array.
Segmentation of Solar Panels
Once the panoramic image (both visible and infrared) is obtained, we need to isolate each individual solar panel for defect classification. Since solar panels exhibit a distinct color compared to the background (e.g., dark blue or black against green grass or concrete), we convert the visible image to the HSV color space and apply color thresholding to generate a binary mask. The mask is further refined using morphological operations (erosion and dilation) to remove small noise and fill gaps. Figure 8 in the original paper (not cited here) shows the step‑by‑step segmentation process. The resulting mask is then applied to the co‑registered infrared image to extract only the panel regions. Finally, we perform contour detection to obtain the minimum bounding rectangle of each panel, and apply a perspective transformation to correct any skew, yielding a normalized 24×40 pixel infrared image of each solar cell. This preprocessing ensures that the neural network receives consistent inputs regardless of the panel’s orientation in the original scene.
Defect Classification Using EfficientNet
For the classification of defects in solar panels, we adopt the EfficientNet‑B0 architecture, which balances network depth, width, and input resolution to achieve high performance with low computational cost. The core building block of EfficientNet is the Mobile Inverted Residual Bottleneck Convolution (MBConv) module, which uses depthwise separable convolutions to reduce parameters, and incorporates Squeeze‑and‑Excitation (SE) attention to adaptively recalibrate channel‑wise features. The structure of MBConv is illustrated in Figure 3 of the reference paper (not shown). The SE block first performs global average pooling, then passes through two fully connected layers to generate a set of channel weights, which are multiplied with the original feature maps. This mechanism enhances the representational power of the network.
We train the classifier on a dataset of 7,000 infrared images of solar cells, each resized to 24×40 pixels. The images are divided into four categories: normal, hot‑spot defect, diode open‑circuit, and crack. The data distribution is shown in Table 1.
| Dataset | Category | Count |
|---|---|---|
| Training | Normal | 1,571 |
| Hot‑spot defect | 1,469 | |
| Diode open‑circuit | 1,172 | |
| Crack | 688 | |
| Validation | Normal | 228 |
| Hot‑spot defect | 217 | |
| Diode open‑circuit | 171 | |
| Crack | 84 | |
| Test | Normal | 461 |
| Hot‑spot defect | 440 | |
| Diode open‑circuit | 331 | |
| Crack | 168 |
To improve generalization, we apply data augmentation including horizontal/vertical flips and random rotations. The network is trained using the Adam optimizer with an initial learning rate of 0.001, a batch size of 16, and gradient clipping. The loss function is the cross‑entropy loss:
$$
L = -\frac{1}{N}\sum_{i=1}^{N}\sum_{c=1}^{M} y_{ic} \log(p_{ic})
$$
where $M$ is the number of classes (4), $y_{ic}=1$ if sample $i$ belongs to class $c$, otherwise 0, and $p_{ic}$ is the predicted probability. Training is stopped after 200 epochs to avoid overfitting. The accuracy curve during training (as shown in Figure 11 of the original paper) indicates that the model converges well.
Experimental Results and Model Comparison
We evaluate the performance of our proposed model (EfficientNet‑B0) against several popular architectures: ResNet101, MobileNet_V3, and RegNet. The evaluation metrics include accuracy, precision, recall, and F1 score, defined as:
$$
\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}
$$
$$
\text{Precision} = \frac{TP}{TP + FP}
$$
$$
\text{Recall} = \frac{TP}{TP + FN}
$$
$$
F1 = \frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}
$$
where $TP$, $TN$, $FP$, $FN$ denote true positives, true negatives, false positives, and false negatives, respectively. The results on the test set are summarized in Table 2.
| Model | Accuracy | Precision | Recall | F1 Score |
|---|---|---|---|---|
| ResNet101 | 0.8957 | 0.8951 | 0.8925 | 0.8936 |
| MobileNet_V3 | 0.9214 | 0.9169 | 0.9134 | 0.9148 |
| RegNet | 0.9085 | 0.9029 | 0.9034 | 0.9028 |
| EfficientNet_B0 | 0.9371 | 0.9313 | 0.9320 | 0.9311 |
As shown in Table 2, the EfficientNet‑B0 model outperforms all other architectures across all four metrics. Its accuracy reaches 93.71%, precision 93.13%, recall 93.20%, and F1 score 93.11%. This demonstrates that the balanced scaling strategy and attention mechanisms in EfficientNet are particularly effective for the fine‑grained classification task of solar panel defects.
Integration and Final Localization Results
After training, the classifier is applied to the segmented infrared images of every solar panel in the panoramic view. For each panel, the model outputs a probability distribution over the four classes. The panel is then color‑coded on the original visible image: green for normal, red for crack, blue for hot‑spot, and cyan for diode open‑circuit. This provides an intuitive and accurate visualization of defect locations across the entire solar panel array. An example of such localization is shown in Figure 12 of the original paper (not cited here). The integration of stitching, segmentation, and deep learning classification enables fast, non‑contact inspection of large‑scale solar panel installations, significantly reducing manual effort and improving detection reliability.
Conclusion
We have developed a complete deep learning based system for the classification and localization of defects in solar panels. By combining AKAZE feature stitching to create panoramic views, HSV‑based segmentation to isolate individual panels, and an EfficientNet‑B0 classifier to identify four types of defects, our approach achieves high accuracy and robustness. The experimental results confirm that the proposed method outperforms several baseline networks, with an accuracy of 93.71% and F1 score of 93.11%. This system can be deployed in real‑world photovoltaic plants to automate the inspection process, reduce downtime, and enhance the efficiency of solar panel maintenance. Future work may extend the method to handle additional defect types and incorporate temporal information from repeated flights for predictive maintenance.