1. Introduction
As global energy demand continues to rise, renewable energy sources have become increasingly important. Among them, solar energy is one of the most promising due to its abundance and low environmental impact. In recent years, the scale of photovoltaic power generation has grown exponentially, especially in countries like China. Most solar power plants are located in arid or semi-arid regions such as the northwest and northeast, where dust, wind, and sand are common. These environmental conditions lead to the accumulation of dust, bird droppings, and other contaminants on the surface of solar panels. Such soiling reduces the transmittance of sunlight and can cause a localized temperature rise known as the “hot spot effect”, which may permanently damage the panel. Therefore, regular cleaning and defect inspection are essential for maintaining the efficiency and lifespan of solar panels.
Manual inspection and cleaning of large photovoltaic arrays are labor-intensive and costly. Automated solutions, such as cleaning robots equipped with vision systems, offer an efficient alternative. This thesis focuses on the software aspects of a solar panel cleaning robot, including surface defect detection, internal defect detection using infrared imaging, and precise localization of detected defects. The goal is to develop reliable algorithms that can operate in real time to support autonomous cleaning and maintenance.
2. Traditional Pixel-Based Defect Detection
In this chapter, I present a classic computer vision approach for detecting surface defects on solar panels. This method relies on pixel-level operations and is designed to be simple, interpretable, and computationally efficient.
2.1 Image Characteristics
Visible-light images of solar panels typically show a blue background with white gridlines (busbars and fingers). Defects such as dust patches, bird droppings, cracks, or stains appear as irregular regions that differ in color or texture from the surrounding area. The blue background arises from the anti-reflective coating on silicon cells. The gridlines are thin, but the main busbars are relatively thick and can be confused with defects. Therefore, the detection algorithm must be able to distinguish defects from gridlines while being robust to varying illumination and noise.
2.2 Overall Algorithm Design
The proposed pixel-based detection algorithm consists of the following steps:
1. **Image preprocessing**: Apply a median filter to reduce noise and eliminate thin gridlines.
2. **Color space conversion**: Convert the image from RGB to HSV to separate the blue background from defects.
3. **Binarization**: Use Otsu’s method to produce a binary image.
4. **Morphological operations**: Apply an opening operation (erosion followed by dilation) to remove small residual gridline fragments while preserving defect shapes.
5. **Connected component labeling**: Label connected regions and filter out those with an area smaller than a threshold to identify candidate defects.
The block diagram of this algorithm is shown below.
2.3 Key Techniques
2.3.1 Median Filtering
Given an input image \(I(x,y)\), median filtering replaces each pixel with the median value of its neighborhood. The output is given by:
\[
g(x,y) = \text{median}\{f(x+i, y+j) \mid (i,j) \in W\}
\]
where \(W\) is a \(3\times3\) or \(5\times5\) window. In my experiments, a \(3\times3\) median filter effectively removed salt-and-pepper noise and fine gridlines while preserving the edges of defects. As shown in Table 2.1, median filtering outperformed mean and Gaussian filtering in terms of defect visibility and background suppression.
2.3.2 HSV Color Space Conversion
The RGB to HSV conversion is performed using the following standard formulas. Suppose \(R,G,B \in [0,1]\), define \(Max = \max(R,G,B)\), \(Min = \min(R,G,B)\), and \(\Delta = Max – Min\). Then:
\[
H =
\begin{cases}
0 & \text{if } Max = Min \\
60^\circ \times \frac{G-R}{\Delta} + 60^\circ & \text{if } Min = R \\
60^\circ \times \frac{B-G}{\Delta} + 180^\circ & \text{if } Min = G \\
60^\circ \times \frac{R-B}{\Delta} + 300^\circ & \text{if } Min = B
\end{cases}
\]
\[
S =
\begin{cases}
0 & \text{if } Max = 0 \\
\frac{\Delta}{Max} & \text{otherwise}
\end{cases}
\]
\[
V = Max
\]
After converting the preprocessed solar panel image to HSV, the background (blue) becomes distinguishable from defects (which appear dark). When mapped back to RGB for display, the result is a blue background with black defects and gridlines.
2.3.3 Otsu Binarization
Otsu’s method automatically determines an optimal threshold \(T\) that minimizes the intra-class variance. The weighted within-class variance is:
\[
\sigma^2_W(T) = \omega_0(T)\sigma^2_0(T) + \omega_1(T)\sigma^2_1(T)
\]
where \(\omega_0\) and \(\omega_1\) are the probabilities of background and foreground, and \(\sigma^2_0\), \(\sigma^2_1\) are their variances. The threshold \(T^*\) is chosen to minimize \(\sigma^2_W(T)\). In my implementation, the binary image is represented as:
\[
B(x,y) =
\begin{cases}
255 & \text{if } I(x,y) > T^* \\
0 & \text{otherwise}
\end{cases}
\]
After Otsu binarization, only the thick busbars and defect regions remain visible.
2.3.4 Morphological Opening
Opening is defined as erosion followed by dilation:
\[
I \circ S = (I \ominus S) \oplus S
\]
where \(S\) is a structuring element (e.g., a \(5\times5\) square). This operation removes small objects and thin protrusions while preserving the size and shape of larger objects. In the solar panel defect image, opening eliminates the leftover gridline fragments, leaving only the defect regions.
2.3.5 Connected Component Labeling
I used the seed-filling algorithm with 8-connectivity to label connected regions. For a binary image, two pixels are considered connected if they are adjacent horizontally, vertically, or diagonally. The algorithm assigns a unique label to each connected component. Components with area smaller than a predefined threshold (e.g., 50 pixels) are discarded as noise. The centroid of each remaining component is computed as:
\[
\bar{x} = \frac{1}{N}\sum_{i=1}^{N}x_i, \quad \bar{y} = \frac{1}{N}\sum_{i=1}^{N}y_i
\]
where \(N\) is the number of pixels in the component.
2.4 Experimental Results and Analysis
I tested the pixel-based algorithm on 200 randomly selected solar panel images containing varying numbers and sizes of defects. Table 2.1 shows the detection results for a subset of these images.
| Image ID | Detected defects | Actual defects | Accuracy (%) | Time (s) |
|———-|——————|—————-|————–|———-|
| 002 | 1 | 1 | 100 | 0.81 |
| 012 | 4 | 4 | 100 | 1.34 |
| 030 | 10 | 11 | 90.9 | 1.55 |
| 034 | 7 | 5 | 71.4 | 1.33 |
| 105 | 2 | 2 | 100 | 0.91 |
| 106 | 3 | 4 | 75 | 0.99 |
| 132 | 4 | 5 | 80 | 1.03 |
| 148 | 10 | 12 | 83.3 | 1.54 |
| 190 | 4 | 3 | 75 | 0.97 |
| 200 | 3 | 4 | 75 | 0.89 |
The average detection accuracy was 79.23%, and the average processing time per image was 1.04 seconds. Some false detections and missed detections occurred when defects had similar color to the background or when defects were very small and were removed during preprocessing. This low accuracy and speed make the pixel-based method insufficient for real-time applications.
3. Deep Learning-Based Defect Detection
To improve detection accuracy and speed, I turned to deep learning models based on convolutional neural networks (CNNs). In this chapter, I compare two popular object detection frameworks: Faster R-CNN and YOLOv3, and then propose improvements to YOLOv3 for solar panel defect detection.
3.1 Dataset Preparation
Since no public dataset for solar panel surface defects was available, I created my own dataset following the Pascal VOC format. I captured 30 visible-light images of solar panels with defects using a camera. After cropping and selecting regions of interest, I obtained 200 images of size 400×400 pixels. To augment the dataset and prevent overfitting, I applied grayscale conversion, rotation by 90 degrees, and horizontal flipping, resulting in 1400 images in total. I then labeled all defects using the LabelImg tool. Each annotation includes the bounding box coordinates and the class label “stain”. The dataset was randomly split into training, validation, and test sets with ratios of 64%, 16%, and 20%, respectively.
3.2 Faster R-CNN for Solar Panel Defect Detection
Faster R-CNN is a two-stage detector. It first generates region proposals using a Region Proposal Network (RPN), then classifies each proposal and refines its bounding box. The architecture is based on a VGG16 or ResNet backbone. In my experiments, I used the TensorFlow implementation with a ResNet101 backbone.
The training process can be summarized as:
1. Train an RPN network to generate candidate boxes.
2. Train a Fast R-CNN network using these proposals.
3. Share convolutional features between RPN and Fast R-CNN to form a unified network.
4. Fine-tune the model end-to-end.
For evaluation, I used precision, recall, and mean average precision (mAP). The formulas are:
\[
\text{Precision} = \frac{TP}{TP + FP}, \quad \text{Recall} = \frac{TP}{TP + FN}
\]
where \(TP\), \(FP\), and \(FN\) denote true positives, false positives, and false negatives, respectively. The PR curve for Faster R-CNN after 3000 iterations is shown in Figure 3.1. The mAP was 0.818, with a precision of 0.950 and recall of 0.868. However, the inference speed was only 1.6 frames per second (FPS), which is too slow for real-time detection.
3.3 YOLOv3 for Solar Panel Defect Detection
YOLOv3 is a one-stage detector that directly predicts bounding boxes and class probabilities from a single network pass. It uses a DarkNet-53 backbone with residual connections and adopts a feature pyramid network (FPN) for multi-scale detection. The network divides the input image into an \(S \times S\) grid, and each grid cell predicts a fixed number of bounding boxes, each with a confidence score and class probabilities.
I trained YOLOv3 using PyTorch with an input size of 416×416 pixels. After 3000 iterations, the mAP was 0.687, precision 0.897, and recall 0.864. Although the mAP was lower than Faster R-CNN, the inference speed was 31 FPS, making it much more suitable for real-time applications. Figure 3.2 shows example detection results using YOLOv3. The bounding boxes had reasonable IoU values, but some small defects were missed.
3.4 Improved YOLOv3 for Solar Panel Defect Detection
To boost the accuracy of YOLOv3 while maintaining its speed advantage, I proposed three modifications: adaptive learning rate scheduling, K-means anchor box clustering, and a lightweight backbone (Tiny-YOLOv3).
3.4.1 Adaptive Learning Rate
The standard YOLOv3 uses a fixed learning rate schedule. I implemented an adaptive learning rate that changes based on the loss function trend. Let \(L(n)\) be the loss at epoch \(n\), and \(\eta(n)\) be the learning rate. The update rule is:
\[
\eta(n+1) =
\begin{cases}
k_{inc}\eta(n) & \text{if } \bar{L}(n – k, n) < \bar{L}(n – 2k, n – k) \\
k_{dam}\eta(n) & \text{otherwise}
\end{cases}
\]
where \(k_{inc} > 1\) is the increment factor, \(k_{dam} < 1\) is the damping factor, and \(\bar{L}(a,b)\) denotes the average loss over epochs \(a\) to \(b\). In my implementation, \(k_{inc} = 1.2\), \(k_{dam} = 0.8\), and \(k = 50\). This approach allowed the model to converge in 2300 iterations instead of 3000, saving about 22% of training time.
3.4.2 K-means Anchor Clustering
YOLOv3 uses predefined anchor boxes. Instead of using the default anchors, I applied K-means clustering on the ground-truth bounding boxes to find the best anchor dimensions. The distance metric was defined as:
\[
d(box, centroid) = 1 – IoU(box, centroid)
\]
where \(IoU\) is the intersection-over-union. This metric avoids the bias toward large boxes that occurs with Euclidean distance. I clustered the training annotations into 9 anchor boxes. The resulting anchors improved the matching between predicted and ground-truth boxes, leading to a 7.8% increase in mAP compared to the default anchors.
3.4.3 Lightweight Model: Tiny-YOLOv3
Tiny-YOLOv3 is a simplified version of YOLOv3 with 13 convolutional layers and 6 max-pooling layers. It uses only two-scale feature maps instead of three, which significantly reduces computational complexity. As a result, the model is faster and requires less memory, making it suitable for embedded systems on cleaning robots.
3.5 Experimental Results of Improved YOLOv3
The improved YOLOv3 model was trained using the same dataset and PyTorch environment. The training settings were: initial learning rate 0.001, batch size 16, and 2300 iterations. After training, the PR curve is shown in Figure 3.3. The final performance metrics are summarized in Table 3.1.
| Method | mAP | Precision | Recall | FPS | Inference time (s) |
|———————-|——-|———–|——–|——-|——————–|
| Faster R-CNN | 0.818 | 0.950 | 0.868 | 1.6 | 0.625 |
| YOLOv3 | 0.687 | 0.897 | 0.864 | 31.0 | 0.032 |
| Improved YOLOv3 | 0.809 | 0.927 | 0.904 | 46.5 | 0.021 |
| Pixel-based method | – | 0.79 | – | ~1.0 | 1.040 |
From Table 3.1, the improved YOLOv3 achieved an mAP of 0.809, a precision of 0.927, and a recall of 0.904, which are close to those of Faster R-CNN, while the inference speed was 46.5 FPS, which is even faster than the original YOLOv3. Therefore, the improved YOLOv3 is the best choice for real-time solar panel defect detection in the cleaning robot.
4. Infrared Image-Based Defect Detection
In addition to surface defects, solar panels may suffer from internal faults such as micro-cracks, broken cells, or solder joint failures. These defects often manifest as localized heating, known as hot spots. Infrared thermography is an effective non-destructive technique for detecting such defects because it captures the temperature distribution of the panel surface.
4.1 Characteristics of Infrared Images
Infrared images have several distinct characteristics:
– They reflect temperature distribution; hotter regions appear brighter.
– They are grayscale images without color information.
– The contrast is usually low because temperature differences are small.
– They contain significant noise from the environment and sensor.
– The resolution is typically lower than visible-light images.
For a solar panel with a hot spot, the defective region has a higher temperature than the surrounding healthy cells. Thus, in the infrared image, the hot spot appears as a bright region.
4.2 Infrared Imaging Equipment
I selected the IRM-80X62-USB infrared thermal imaging module for capturing infrared images of solar panels. This module has an array size of 80×62 pixels, a temperature range from -20°C to +500°C, and an accuracy of ±2°C. It communicates via a virtual USB serial port with a baud rate of 921600. The temperature data are transmitted as 9920 bytes (2 bytes per pixel for 4960 pixels). The actual temperature is calculated as:
\[
T = ( \text{HighByte} \times 256 + \text{LowByte} – 2731) / 10 \,^{\circ}C
\]
For example, if the high byte is 0x0B and the low byte is 0x15, the combined value is 0x0B15 = 2837 decimal, then \(T = (2837 – 2731)/10 = 10.6^\circ C\).
The module was connected to a computer via USB, and images were captured when the camera was pointed at a hot spot on a solar panel. An example infrared image is shown in Figure 4.1.
4.3 Hot Spot Detection Algorithm
The algorithm for detecting hot spots in infrared images consists of the following steps:
1. **Grayscale conversion**: Convert the raw sensor data to a grayscale image.
2. **Image enhancement**: Use histogram non-uniform stretching to improve contrast.
3. **Image filtering**: Apply median filtering to reduce noise.
4. **Threshold segmentation**: Use Kittler’s minimum error thresholding to binarize the image and extract the hot spot region.
4.3.1 Histogram Non-Uniform Stretching
This method reassigns the gray levels so that the histogram is stretched non-uniformly. The original gray range \([m_0, m_n]\) is expanded to \([g_0, g_k]\), where \(k > n\). The goal is to make the area of each sub-interval equal. The stretching is described by:
\[
g_i’ = g_0 + \sum_{j=0}^{i-1} k_j
\]
where \(k_j\) is the number of sub-intervals assigned to the original interval \(j\). This technique enhances the contrast of the hot spot region with respect to the background.
I compared three enhancement methods: global histogram equalization, adaptive histogram equalization, and non-uniform stretching. The non-uniform stretching produced the best visual result, making the hot spot clearly distinguishable while preserving background details.
4.3.2 Median Filtering
Similar to the visible-light case, median filtering was applied to remove salt-and-pepper noise. A \(3\times3\) median filter was sufficient to improve the signal-to-noise ratio without significantly blurring the hot spot edges.
4.3.3 Kittler’s Minimum Error Thresholding
Kittler’s method assumes that the image histogram is a mixture of two Gaussian distributions representing background and foreground. The threshold \(t\) is selected by minimizing the criterion function \(J(t)\):
\[
J(t) = P_0(t)\log\sigma_0^2(t) + P_1(t)\log\sigma_1^2(t) – 2[P_0(t)\log P_0(t) + P_1(t)\log P_1(t)]
\]
where \(P_0(t)\) and \(P_1(t)\) are the prior probabilities of background and foreground, and \(\sigma_0^2(t)\), \(\sigma_1^2(t)\) are their variances. The optimal threshold \(t^*\) is:
\[
t^* = \arg\min_{0 \le t \le L-1} J(t)
\]
After thresholding, the binary image \(B(x,y)\) is obtained as:
\[
B(x,y) =
\begin{cases}
255 & \text{if } f(x,y) > t^* \\
0 & \text{otherwise}
\end{cases}
\]
The result of applying this segmentation to a solar panel infrared image is shown in Figure 4.2. The hot spot is accurately extracted from the background.
4.4 Experiments and Discussion
I tested the infrared detection algorithm on several infrared images containing hot spots of different sizes and intensities. In all tested cases, the algorithm successfully segmented the hot spot regions. The processing time for each 80×62 image was less than 50 ms, which is fast enough for real-time monitoring. The main limitations come from the low resolution of the thermal sensor, which may cause small hot spots to appear blurred. However, for the purpose of detecting the presence and approximate location of a hot spot, the algorithm proved reliable.
5. Localization of Defects on Solar Panels
After detecting a defect in an image, the cleaning robot needs to know where the defect is located on the physical solar panel. This requires two steps:
1. Determining the robot’s own position on the solar panel.
2. Computing the distance from the defect to the robot using monocular vision.
5.1 Robot Localization Based on Encoders
The robot follows a boustrophedon (zigzag) path to cover the entire panel. The path is shown in Figure 5.1. Starting from the bottom-left corner, the robot moves straight upward until it reaches the top edge, then rotates 90 degrees to the right, moves a distance equal to the robot’s width, rotates again, and moves downward. By repeating this pattern, the robot covers the full rectangular panel.
An incremental encoder is mounted on each wheel to measure the travel distance. In each sampling period \(\Delta T\), the number of pulses \(n\) is counted. The distance travelled by the wheel is:
\[
l = \frac{\pi d n}{h}
\]
where \(d\) is the wheel diameter, and \(h\) is the number of encoder pulses per revolution. Since the robot moves along a known path, the distance \(l\) can be decomposed into vertical and horizontal components. Let \(H\) be the panel height, \(W\) the panel width, and \(w\) the robot width. The number of vertical segments is:
\[
r = \left\lfloor \frac{l}{H – w} \right\rfloor
\]
The vertical distance from the bottom edge is:
\[
D_y =
\begin{cases}
l – r(H – w) & \text{if } r \text{ is even} \\
(H – w) – (l – r(H – w)) & \text{if } r \text{ is odd}
\end{cases}
\]
The horizontal distance from the left edge is:
\[
D_x = r \cdot w
\]
Thus, the robot’s position is \((D_x, D_y)\) in a coordinate system where the bottom-left corner is \((0,0)\).
5.2 Monocular Distance Measurement
To measure the distance from a detected defect to the robot, I used a single-frame static imaging model. The camera is mounted at a fixed height \(h_c\) and tilted at an angle \(\alpha\) with respect to the horizontal plane. The geometry is shown in Figure 5.2. For a defect point \(P\) on the panel, its projection on the image plane is at pixel coordinate \(v\) (vertical axis). The distance \(L\) from the camera to the defect is given by:
\[
L = \frac{h_c}{\tan\left(\alpha + \arctan\left(\frac{v – v_0}{f_y}\right)\right)}
\]
where \(f_y\) is the focal length in pixels, and \(v_0\) is the vertical coordinate of the principal point.
5.3 Camera Calibration
To obtain the intrinsic parameters and distortion coefficients, I used Zhang’s calibration method. A checkerboard pattern with 10×7 corners and a square size of 26 mm was captured from 37 different angles. The calibration was performed using MATLAB R2019b camera calibration toolbox. The intrinsic parameters obtained for the visible-light camera are:
\[
f_x = 1362.70, \quad f_y = 1369.27, \quad c_x = 661.83, \quad c_y = 455.55
\]
The radial distortion coefficients are:
\[
k_1 = -0.1303, \quad k_2 = 0.8904
\]
The tangential distortion coefficients are:
\[
p_1 = 0.0226, \quad p_2 = 0.0088
\]
These parameters allow the image coordinates to be transformed into world coordinates using:
\[
\begin{bmatrix}
u \\
v \\
1
\end{bmatrix}
=
\begin{bmatrix}
f_x & 0 & c_x \\
0 & f_y & c_y \\
0 & 0 & 1
\end{bmatrix}
\begin{bmatrix}
R & T
\end{bmatrix}
\begin{bmatrix}
X_w \\
Y_w \\
Z_w \\
1
\end{bmatrix}
\]
In our setup, the camera is oriented perpendicular to the direction of motion, and the robot travels on a flat panel, so the extrinsic parameters are fixed and can be measured directly.
5.4 Experimental Results
I experimentally verified the localization algorithm. In one test, the defect was found at image pixel coordinates \((191, 178)\) in a \(510 \times 320\) image. The camera height was 0.3 m, the tilt angle was \(25^\circ\), and the focal length was \(f_y = 1369.27\) pixels. Using the distance formula:
\[
L = \frac{0.3}{\tan(25^\circ + \arctan((178 – 455.55)/1369.27))} \approx 0.473 \text{ m}
\]
The measured distance matched the actual distance within 2 cm, confirming the accuracy of the monocular distance estimation. Combined with the robot’s encoder position, the absolute position of the defect on the solar panel was determined with an error of less than 5 cm in the test scenarios.
6. Conclusion and Future Work
In this thesis, I investigated vision-based defect detection and localization for solar panels. I first designed a traditional pixel-based detection algorithm that used median filtering, HSV conversion, Otsu binarization, morphological opening, and connected component labeling. Although this method could identify defects, its accuracy and speed were insufficient for real-time use.
I then explored deep learning approaches. After comparing Faster R-CNN and YOLOv3, I selected YOLOv3 for its superior speed and modified it with adaptive learning rate, K-means anchor clustering, and a lightweight Tiny-YOLOv3 backbone. The improved YOLOv3 achieved 0.809 mAP, 0.927 precision, 0.904 recall, and 46.5 FPS, making it the most practical solution for the cleaning robot.
For internal defects such as hot spots, I used infrared thermography. I designed an algorithm based on histogram enhancement, median filtering, and Kittler’s minimum error thresholding. This algorithm successfully segmented hot spot regions in infrared images with low computational cost.
Finally, I addressed defect localization by combining encoder-based robot position estimation with monocular distance measurement using a static imaging model and camera calibration. The total localization error was acceptable for cleaning and reporting purposes.
In the future, I plan to extend the infrared detection algorithm to classify different types of internal defects (micro-cracks, broken cells, finger interruptions, and dark cells). Additionally, I will consider using more robust simultaneous localization and mapping (SLAM) techniques to reduce drift in the robot’s encoder readings, especially on slippery surfaces. The deep learning model could also be improved via model pruning or quantization to run even faster on embedded hardware.
In summary, this work provides a comprehensive study of solar panel defect detection and localization, from traditional image processing to modern deep learning techniques. The proposed methods have shown high efficacy in real-world experimental settings and are ready to be integrated into an autonomous solar panel cleaning robot.