1. Introduction
In recent years, China’s solar power generation technology has developed rapidly, and the scale of photovoltaic power generation has been multiplying. Most photovoltaic power stations in China are distributed in the northwest, northeast, and other regions, where there is little precipitation, strong wind and sand, and much dust. Dust and dirt are easy to accumulate on the surface of solar panels. At the same time, these stains will cover the surface of solar panels, resulting in the failure of solar panels due to the “hot spot effect”. Manual cleaning and detection of solar panel defects requires a lot of manpower and material resources. Therefore, it is necessary to detect solar panel defects independently through a photovoltaic cleaning robot.
This paper mainly studies the software part of the photovoltaic cleaning robot. The main work contents are as follows: first, the traditional pixel-based operation method is used to detect defect images of solar panels; second, the deep learning method is used to detect defect images of solar panels; third, infrared images of solar panels with internal defects are detected; fourth, the defect location technology of solar panels is studied.
2. Defect Detection Algorithm Based on Pixel Operations
According to the characteristics of solar panel defect images, I designed a set of detection algorithms based on pixel operations, including key technologies such as HSV space conversion and connected domain label detection.
2.1 Overall Design of the Detection Algorithm
The algorithm flow includes the following key steps:
(1) The visible light camera is used to collect solar panel images containing surface defects, and the input image is preprocessed, that is, the image is filtered to effectively remove the fine grid lines and noise in the solar panel image and improve image features.
(2) HSV space conversion is performed on the preprocessed image. The captured visible light image is an RGB color model. In addition to the color of surface dirt, the solar panel image containing defects has white coarse grid lines and dark blue backgrounds with different saturations. Therefore, in order to remove the blue background, the HSV space conversion method is adopted.
(3) The image is binarized and opened. Since the computer can only display images according to the RGB channel, the result obtained in the previous step is mapped in the RGB color space to obtain a blue background with black spots. In order to more clearly display the shape of surface defects, the Otsu method is used for binarization, followed by the operation of erosion and then dilation to filter out small spots and highlight the position of surface defects.
(4) Connected domain label detection is performed on the image, and connected domains whose area is less than a specific threshold are deleted, which can mark the number of surface defect points and the position of each surface defect point in each image.
2.2 Key Technologies
2.2.1 Image Preprocessing
Due to differences in environment, acquisition equipment, and other conditions, solar panel images are often disturbed by noise when captured and transmitted, which affects image quality and may even erase very important detailed information in the image, causing a non-negligible impact on subsequent image recognition, surface defect detection, and other processes. The solar panel image has both thick and thin grid lines. The thin ones are secondary grid lines, which play the role of guiding the current, and the thick ones are main grid lines, which are used to collect the current from the secondary grid lines for aggregation. In subsequent processing, the grid lines will have a relatively obvious impact on the detection of surface defects, so it is necessary to filter the captured solar panel images.
Commonly used image filtering methods include mean filtering, median filtering, and Gaussian filtering. The mean filter formula is:
$$g(x,y) = \frac{1}{m \times n} \sum_{(i,j) \in S} f'(i,j)$$
The median filter formula is:
$$g(x,y) = \text{med}\{f(x-k, y-l), (k,l) \in W\}$$
The Gaussian filter formula is:
$$g(x,y) = e^{-\frac{x^2 + y^2}{2\sigma^2}}$$
To compare the effects of these three methods, I selected a captured solar panel image containing surface defects for testing. The experimental results show that the image processed by median filtering can not only remove noise and fine grid lines but also highlight the position of surface defect points, obtaining a smooth solar panel surface defect image, which is conducive to further processing. Therefore, the median filtering method is chosen in preprocessing.
2.2.2 HSV Space Conversion
Most solar panels on the market are dark blue with varying saturation. When the solar panel surface defect image is preprocessed, the noise and fine grid lines are filtered out, leaving the blue background, white coarse grid lines, and gray-white stains on the surface. In order to further identify the surface defects on the image, the influence of the blue background needs to be removed first, so the RGB color space to HSV color space conversion method is considered.
The conversion formula from RGB to HSV is:
$$
H =
\begin{cases}
0 & \text{if } Max = Min \\
60 \times \frac{G-R}{Max-Min} + 60 & \text{if } Min = B \\
60 \times \frac{B-G}{Max-Min} + 180 & \text{if } Min = R \\
60 \times \frac{R-B}{Max-Min} + 300 & \text{if } Min = G
\end{cases}
$$
$$
S = Max – Min
$$
$$
V = Max
$$
where the RGB values are in the range [0,1], $Max = \max(R,G,B)$, and $Min = \min(R,G,B)$. Since the computer can only display images according to the RGB channel, after the HSV space conversion of the solar panel surface defect image, the image is mapped in the RGB color space to obtain an image with a blue background and black spots, which means that the surface defects and coarse grid lines on the solar panel are displayed as black areas.
2.2.3 Otsu Binarization
The image obtained through HSV space conversion is blue with black spots. For the convenience of subsequent processing, binarization is first performed to convert the color image into a black-and-white image. Since the image obtained in the previous step only has black and blue colors, the Otsu method is used for binarization.
The Otsu method first assumes that the gray level of an image is $(0 \sim m)$, and the number of pixels with gray level $i$ is $n_i$. The total number of pixels in the entire image is $N$. The probability of each gray level appearing is $P_i = n_i / N$. According to the threshold $T$, the image can be divided into two parts: $C_0 = \{0,1,2,\ldots,T\}$ and $C_1 = \{T+1, T+2, \ldots, m\}$. The variance between classes is:
$$\sigma^2 = \omega_0 \omega_1 (\mu_0 – \mu_1)^2$$
When the variance is maximized, the corresponding $T$ is the binarization threshold. After processing the solar panel defect image with the Otsu method, only the surface defects and the most obvious coarse grid lines remain, and some detailed information of the defects can be preserved.
2.2.4 Morphological Processing
From the binarized image, it can be seen that surface defects are relatively large patches, while the grid lines are some intermittent points. For images where the main objects should be retained and small interferences removed, morphological processing is usually adopted. The opening operation (erosion followed by dilation) can separate elements that are not closely connected and remove small details. The formulas are:
$$open(X,B) = (X \ominus B) \oplus B$$
$$close(X,B) = (X \oplus B) \ominus B$$
Compared with the mode filtering method, the opening operation better preserves the shape and contour of the defects while completely eliminating the influence of grid lines. Therefore, the opening operation is chosen to process the solar panel defect image.
2.2.5 Connected Domain Label Detection
After the previous processing, only the defect positions remain in the binarized image. To further confirm the position, size, and quantity of defects, the connected domain label detection method can be used. A connected domain is a collection of certain pixels that are adjacent and have similar properties in an image. The connectivity judgment standards are mainly divided into two types: four-neighborhood connectivity and eight-neighborhood connectivity. I chose the seed-filling method based on eight-neighborhood connectivity. The center coordinate of the defect in the test image is [191, 178].
2.3 Experimental Results and Analysis
To test the accuracy and efficiency of this algorithm, I randomly selected 200 solar panel defect images containing different numbers and sizes of surface defect points for testing. Some of the experimental results are shown in Table 1.
| Image No. | Detected Defects | Actual Defects | Accuracy (%) | Time (s) |
|---|---|---|---|---|
| 002 | 1 | 1 | 100 | 0.81 |
| 012 | 4 | 4 | 100 | 1.34 |
| 030 | 10 | 11 | 90.9 | 1.55 |
| 034 | 7 | 5 | 71.4 | 1.33 |
| 105 | 2 | 2 | 100 | 0.91 |
| 106 | 3 | 4 | 75 | 0.99 |
| 132 | 4 | 5 | 80 | 1.03 |
| 148 | 10 | 12 | 83.3 | 1.54 |
| 190 | 4 | 3 | 75 | 0.97 |
| 200 | 3 | 4 | 75 | 0.89 |
The average accuracy of this algorithm is 79.23%, and the average detection time is 1.04 seconds per image. The accuracy and efficiency are not very satisfactory. Some defects are missed because their color is too close to the background or the defect is too small, and they are ignored during preprocessing or morphological processing. Some non-defect spots with slightly different colors from the background are misidentified as defects.
3. Defect Detection Algorithm Based on Deep Learning
3.1 Dataset Preparation
Since there is no ready-made relevant dataset on the Internet, I prepared my own solar panel defect dataset according to the Pascal VOC dataset format. First, 30 images of solar panels containing defects were captured with a visible light camera. Since the captured images include backgrounds, frames, and other irrelevant elements, a total of 200 images were cropped from them, uniformly in .jpg format with a size of 400*400 pixels. Since too few images during neural network training can easily cause overfitting, the dataset was expanded. First, some of the initial 200 images were grayscaled to expand the dataset to 350 images, and then these 350 images were rotated 90 degrees and flipped left-right to expand to 1400 images.
I used the LabelImg annotation software to prepare the solar panel defect image dataset. The defects in the images were framed and recorded with the label “stain”. After completing the .xml files, the dataset was divided into training set, validation set, and test set in proportion, and the VOC format dataset was completed.
3.2 Faster R-CNN Algorithm
Faster R-CNN is an improvement on the R-CNN algorithm series. The network structure integrates feature extraction and candidate region selection, greatly improving the network performance, especially the detection speed. The training steps are as follows:
(1) Train an RPN network first, and use this RPN network to select candidate boxes for different input images.
(2) Train a Fast R-CNN network, reading the annotation information in the input images for training.
(3) Input the candidate box information, feature map information, and image channel number, scaling ratio, and other information into the trained RPN network to generate ROI candidate boxes for each image.
(4) Train the Faster R-CNN network, select the Res101 network for feature extraction, and output the prediction confidence and prediction boxes.
The computer configuration is shown in Table 2.
| Item | Configuration |
|---|---|
| CPU | Intel(R) Core(TM) i9-9900K |
| GPU | NVDIA GeForce RTX 2080Ti |
| Memory | 32G |
| OS | Windows10 |
| CUDA | CUDA V10.0.130 |
| Framework | tensorflow-gpu1.15.0rc2 |
| Python | 3.6 |
The evaluation metrics include average precision (AP), precision, recall, and detection speed (FPS). The precision and recall are calculated as:
$$precision = \frac{TP}{TP + FP}$$
$$recall = \frac{TP}{TP + FN}$$
After 3000 iterations, the AP of Faster R-CNN is 0.818, the recall is 0.868, and the precision is 0.950. The FPS is 1.6 frames/s.
3.3 YOLOv3 Algorithm
YOLO (You Only Look Once) is another framework proposed after the R-CNN series to improve target detection speed. Its core idea is to replace the two-stage algorithm with a one-stage algorithm, directly outputting the regression border position and category in the output layer. YOLOv3 has two improvements over YOLOv2: one is the use of the residual model DarkNet-53, which deepens the network structure; the other is the use of the FPN structure to achieve multi-scale detection.
The training process of YOLOv3 is as follows:
(1) Load the pre-trained model and the yolov3.weights file for iterative training.
(2) Adjust the image resolution. In this experiment, the image size is set to 416×416, and the image labels are read from a batch to obtain relevant information, which is then input into the model for detection.
(3) Calculate the loss value, including the bounding box loss, confidence loss, and category loss, and then perform backpropagation and parameter optimization.
(4) After iterating the specified number of times, save the trained model.
After 3000 iterations, the mAP of YOLOv3 is 0.687, the precision is 0.897, the recall is 0.864, and the FPS is 31.0 frames/s. Compared with Faster R-CNN, YOLOv3 has greatly improved detection speed, but the detection accuracy is lower.
3.4 Improved YOLOv3 Algorithm
Considering the detection speed requirements, I chose YOLOv3 as the system detection network. In order to improve the detection accuracy, I proposed the following improvements to the YOLOv3 network:
3.4.1 Adaptive Learning Rate Configuration
Learning rate is very important for training a good model. A high learning rate can cause oscillation and loss explosion, while a low learning rate makes training very slow and may cause overfitting. I proposed an adaptive learning rate method, where the system adjusts different learning rates in each iteration. The formula is:
$$\eta(n+1) =
\begin{cases}
k_{inc} \eta(n) & \text{if } L(n) \leq L(n-1) \\
k_{dam} \eta(n) & \text{if } L(n) > L(n-1)
\end{cases}$$
where $k_{inc} > 1$ is the increment factor and $k_{dam} < 1$ is the damping factor. When the training starts after $(2k + m)$ rounds, the loss function is calculated every $k$ rounds, and the learning rate is automatically adjusted by comparing the average loss function values. I compared three learning rate configuration methods, and the results are shown in Table 3.
| Method | Iterations | Training Time (h) |
|---|---|---|
| Method 1 (fixed 0.001) | 3000 | 11.2 |
| Method 2 (max 0.006, min 0.0001) | 2700 | 12.2 |
| Adaptive method | 2300 | 8.7 |
The adaptive learning rate configuration method effectively improves training efficiency and reduces the number of iterations.
3.4.2 K-means Clustering for Anchor Boxes
Anchor boxes are key to target detection quality. YOLOv3 uses the K-means clustering algorithm to cluster the true bounding boxes of the training set. However, the traditional K-means clustering algorithm uses Euclidean distance, which can cause larger boxes to have greater errors. To avoid this, the distance is defined as:
$$d(box, centroid) = 1 – IoU(B, C)$$
The improved anchor box clustering algorithm process is as follows:
(1) For the original training set, generate a file containing position and category information.
(2) Given $k$ cluster centers $\{(W_j, H_j), j \in \{1,2,\ldots,k\}\}$, calculate the distance from each box to the cluster center and assign the box to the nearest cluster center.
(3) Recalculate the cluster center of each cluster by calculating the average width and height of all boxes in each cluster.
(4) Repeat steps (3) and (4) until the cluster centers barely change.
The experimental results show that the method of using the K-means clustering algorithm to determine the anchor box size improves the accuracy, and the mAP value is about 7.8% higher.
3.4.3 Lightweight Model Tiny-YOLOv3
The backbone network of YOLOv3 uses the darknet-53 network with 53 convolutional layers, which is relatively complex and requires high-performance computer hardware. Therefore, I considered using Tiny-YOLOv3 to replace YOLOv3. Tiny-YOLOv3 is a lightweight version of YOLOv3, containing 13 convolutional layers and 6 max-pooling layers. It uses a pyramid structure with two scales of feature maps for prediction. The original image is divided into $S \times S$ small regions, where the feature map sizes are 13×13 and 26×26. Experimental results show that the lightweight model Tiny-YOLOv3 can effectively further accelerate model training and detection speed.
3.5 Experimental Results and Analysis
The training process of the improved YOLOv3 algorithm is basically the same as that of YOLOv3, except that the pre-trained weight file is replaced with tiny-yolov3.weights, the number of iterations is set to 2300, and the initial learning rate is set to 0.001. The improved network’s PR curve shows that the mAP increased to 0.809, the precision increased to 0.927, and the recall increased to 0.904. The FPS increased to 46.5 frames/s. A comparison of the different methods is shown in Table 4.
| Method | AP (%) | Precision (%) | Recall (%) | FPS |
|---|---|---|---|---|
| Faster R-CNN | 81.8 | 95.0 | 86.8 | 1.6 |
| YOLOv3 | 68.7 | 89.7 | 86.4 | 31.0 |
| Improved YOLOv3 | 80.9 | 92.7 | 90.4 | 46.5 |
| Pixel-based method | 79.2 | — | — | 0.96 |
Although the precision and recall of the improved YOLOv3 network are still slightly lower than those of Faster R-CNN, its detection speed is much higher, which is suitable for real-time defect detection systems. Compared with the pixel-based method, both speed and accuracy have been greatly improved. Therefore, I chose the deep learning method using the improved YOLOv3 network to detect defects on solar panels.
4. Defect Detection Algorithm Based on Infrared Images
4.1 Infrared Characteristics of Solar Panel Hot Spots
Solar panels may have internal defects such as hidden cracks, fragments, broken grid lines, and black cells in addition to visible surface stains or damage. Since these defect points often generate heat when energized, the infrared image detection method is considered to detect internal defects of solar panels.
Infrared images have the following characteristics: they can reflect the temperature distribution of the object surface; they do not contain color information; the gray value variation range is small; the image is relatively blurred; the image contains much noise; and the infrared images captured during the day have higher clarity than those at night.
In the captured infrared images of solar panels, the temperature at the hot spot is the highest on the entire solar panel, meaning the gray value at the hot spot is higher than that of normally working solar panel components. The temperature at the center of the hot spot is also significantly higher than the average temperature around the hot spot.
4.2 Infrared Image Acquisition Equipment
Common image acquisition equipment includes smart cameras, near-infrared CCD cameras, and infrared thermal imagers. A comparison of these devices is shown in Table 5.
| Type | Smart Camera | Near-infrared CCD | Infrared Thermal Imager |
|---|---|---|---|
| Processing speed | Fast | Slow | Medium |
| Filter visible light | Needed | Needed | Not needed |
| Sensitive wavelength | Visible light | 560-1100 μm | Above 800 μm |
I chose the infrared thermal imager for capturing infrared images of solar panel hot spots for the following reasons: (1) smart cameras and near-infrared CCD cameras need to filter out visible light, and smart cameras can only detect surface defects of solar panels; (2) the detection principle of near-infrared CCD cameras is electroluminescence, but solar panels use photo-generation; (3) the sensitive wavelength range of silicon solar panels is about 800-1000 μm, which matches the infrared thermal imager; (4) the infrared thermal imager allows non-contact detection without damaging solar panels.
4.3 Infrared Thermal Imager Module
I selected the IRM-80X62-USB serial port infrared thermal imager module for infrared image acquisition. The main parameters are shown in Table 6.
| Parameter | Value |
|---|---|
| Image array size | 80(H)x62(V) |
| Temperature measurement range | -20℃~+500℃ |
| Industrial version accuracy | ±2℃ |
| Operating voltage | +4~5.5V |
| Power consumption | ≤0.5W |
The module is connected to the computer via a USB interface, and the image can be read directly through the serial port function. The temperature calculation method is as follows: all pixel temperatures and background temperatures use absolute temperature representation with a temperature resolution of 0.1 degrees. The formula is:
$$\text{actual temperature} = (\text{temperature high byte} \times 256 + \text{temperature low byte} – 2731) / 10$$
For example, if the temperature high byte is 0x0B and the low byte is 0x15, the temperature is 0x0B15 = 2837 in decimal, and the actual temperature is (2837 – 2731) / 10.0 = 10.6℃.
4.4 Hot Spot Detection Algorithm
The captured infrared images have low signal-to-noise ratio and low contrast. To segment the hot spot from the image, the contrast between the hot spot and the background needs to be enhanced. The algorithm flow is shown in Figure 4.
4.4.1 Image Enhancement
I used the histogram correction method for image enhancement. I compared three methods: global histogram equalization, adaptive histogram equalization, and non-uniform histogram stretching. The experimental results show that the non-uniform histogram stretching method achieves the best effect, making the contrast between the hot spot and the background relatively large.
4.4.2 Image Filtering
Using the histogram correction method, I obtained an image with relatively prominent hot spot information. However, due to various noises during the shooting process, image filtering is needed to enhance the signal-to-noise ratio. The infrared images mainly contain salt-and-pepper noise and Gaussian noise. I compared mean filtering, median filtering, and Gaussian filtering. The median filtering method is most effective in removing the salt-and-pepper noise.
4.4.3 Threshold-based Image Segmentation
I adopted the adaptive minimum threshold segmentation algorithm proposed by Kittler. Assuming the background is dark and the target is bright, and both satisfy the mixture of Gaussian distributions:
$$p(i) = \sum_{j=0}^{1} p_j \cdot p(i|j)$$
where $p_0$ and $p_1$ are the prior probabilities of the background $C_0$ and target $C_1$, respectively, and both follow normal distributions with mean $\mu_j$ and variance $\sigma_j^2$. The optimal threshold $t^*$ is obtained when the minimum error objective function $J(t)$ takes the minimum value:
$$J(t) = P_0(t) \ln \frac{\sigma_0^2(t)}{[P_0(t)]^2} + P_1(t) \ln \frac{\sigma_1^2(t)}{[P_1(t)]^2}$$
The final binary image is obtained by:
$$bin(x,y) =
\begin{cases}
0 & \text{if } f(x,y) \leq t^* \\
255 & \text{if } f(x,y) > t^*
\end{cases}$$
The experimental results show that this algorithm can clearly and accurately segment the hot spot from the infrared image of the solar panel, meaning the algorithm can detect the position of the hot spot on the solar panel.
5. Defect Location Technology of Solar Panels
5.1 Positioning of the Photovoltaic Cleaning Robot
The positioning of defects detected by a monocular camera mainly involves two processes: first, obtaining the position of the photovoltaic cleaning robot on the solar panel; second, obtaining the position of the defect relative to the robot.
I used the relative positioning method based on odometry. The basic principle is that when the initial position of the mobile robot is known, the change in position and direction angle relative to the initial position can be calculated through sensor values, enabling real-time position estimation. In the case where the robot’s walking path is determined, the distance the robot’s wheels move relative to the ground is calculated through the pulse changes of the photoelectric encoder in the sampling period, thereby calculating the robot’s position on the solar panel.
The photovoltaic cleaning robot adopts a back-and-forth path. The operation steps are: (1) the robot starts from the lower left corner of the solar panel and walks forward along a straight line parallel to the edge; (2) when encountering the upper frame, it stops, rotates 90 degrees clockwise, and moves to the right; (3) after moving a distance equal to the robot’s bottom edge length, it stops, rotates 90 degrees clockwise again, and moves in the opposite direction; (4) when encountering the lower frame, it stops, rotates 90 degrees counterclockwise, and continues moving to the right; (5) it continues according to the previous movement pattern until reaching the end point.
The incremental encoder is installed on the wheels of the photovoltaic cleaning robot. The movement distance is calculated by:
$$l = n \cdot \frac{\pi d}{h}$$
where $n$ is the number of pulse signals in the sampling period, $d$ is the wheel diameter, and $h$ is the encoder line number. When the robot walks along the planned path and detects a defect, it stops. The walking distance can be obtained through the encoder. According to the following formulas, the robot’s position on the solar panel can be calculated:
$$r = \frac{l}{H + w}$$
$$D_y =
\begin{cases}
H – \{l – r(H + w)\} & \text{if } [r] \text{ is odd} \\
l – [r](H + w) & \text{if } [r] \text{ is even}
\end{cases}$$
5.2 Defect Positioning Based on Monocular Vision
The distance measurement model based on a single-frame static image uses geometric perspective transformation to describe the projection process. Both the visible light camera module and the infrared thermal imager module are fixed at a certain inclination angle with the robot support. After calibrating the internal and external parameters of the camera, any point P in space can be mapped to point p in the image coordinate system.
In the single-frame static image distance measurement model, $f$ represents the focal length, $h$ represents the installation height of the camera, and $\alpha$ represents the pitch angle of the camera. The distance from the defect center to the robot is:
$$l = \frac{h}{\tan[\arctan(\frac{y – y_0}{f}) + \alpha]}$$
where $h$ and $\alpha$ can be obtained by measurement, and $f$, $y$, and $y_0$ are obtained through camera calibration.
5.3 Spatial Coordinate Transformation
To extract the final defect position information, the mapping relationships among the world coordinate system, camera coordinate system, pixel coordinate system, and image coordinate system need to be established. The relationship between the image coordinate system and the pixel coordinate system is:
$$u = \frac{x}{d_x} + u_0, \quad v = \frac{y}{d_y} + v_0$$
The relationship between the camera coordinate system and the image coordinate system is:
$$x = f \frac{x_c}{z_c}, \quad y = f \frac{y_c}{z_c}$$
The transformation between the world coordinate system and the camera coordinate system is:
$$\begin{bmatrix} x_c \\ y_c \\ z_c \\ 1 \end{bmatrix} = \begin{bmatrix} R & T \\ O & 1 \end{bmatrix} \begin{bmatrix} x_w \\ y_w \\ z_w \\ 1 \end{bmatrix}$$
The final transformation relationship between the image coordinate system and the world coordinate system is:
$$\begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = \begin{bmatrix} f_x & 0 & u_0 & 0 \\ 0 & f_y & v_0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix} \begin{bmatrix} R & T \\ O & 1 \end{bmatrix} \begin{bmatrix} x_w \\ y_w \\ z_w \\ 1 \end{bmatrix}$$
5.4 Camera Distortion Model and Calibration
In actual use, the camera will produce different degrees of distortion, including radial distortion and tangential distortion. The nonlinear distortion can be expressed as:
$$x’ = x + \delta_x(x,y)$$
$$y’ = y + \delta_y(x,y)$$
The distortion parameters $k_1$, $k_2$, $p_1$, $p_2$, $s_1$, and $s_2$ can be determined. I used Zhang’s calibration method with a black-and-white checkerboard template for camera calibration. The calibration steps are:
(1) Determine the size and number of corner points of the checkerboard template, print and attach the template to a flat surface.
(2) Use the camera to be calibrated to capture images of the checkerboard from different angles.
(3) Detect the corner points in the images to obtain their coordinates in the pixel coordinate system.
(4) Calculate the internal parameters and distortion parameters of the camera.
(5) Optimize the parameters by solving the minimum parameter values.
I used a 10×7 checkerboard with a square side length of 26mm and captured 37 valid calibration images. The calibration results are shown in Table 7.
| Parameter | Value |
|---|---|
| $f_x$ (horizontal focal length) | 1362.70337 |
| $f_y$ (vertical focal length) | 1369.27121 |
| $u_0$ | 661.83015 |
| $v_0$ | 455.54694 |
| $k_1$ (radial distortion) | -0.1303 |
| $k_2$ (radial distortion) | 0.8904 |
| $p_1$ (tangential distortion) | 0.0226 |
| $p_2$ (tangential distortion) | 0.0088 |
5.5 Experimental Results and Analysis
For the visible light camera on the photovoltaic cleaning robot, the distance from the camera to the solar panel surface $h$ is 30cm, and the angle between the lens and the horizontal plane $\alpha$ is 25 degrees. Taking the detection result of the pixel-based algorithm as an example, the defect center is located at coordinate [191, 178] in the image. Substituting into the distance formula:
$$l = \frac{0.3}{\tan[\arctan(\frac{178 – 0}{1369.27}) + 25^\circ]} \approx 0.473 \text{ m}$$
The distance from the defect to the photovoltaic cleaning robot is 0.473m. Combined with the robot’s position on the solar panel, the accurate position of the defect can be obtained.
6. Conclusion and Outlook
This paper mainly studied the software part of the photovoltaic cleaning robot. The work contents include:
(1) For the traditional visual detection method to detect defects on solar panels, I designed an algorithm based on the characteristics of surface defect images. After image preprocessing, the defects in the image were detected through HSV space conversion, opening operation, connected domain label detection, and other steps. The efficiency and accuracy of the algorithm were measured through experiments.
(2) For the deep learning method to detect defects on solar panels, I first prepared a VOC-format solar panel defect image dataset. Then, I used the Faster R-CNN network and the YOLOv3 network for training and detection respectively. Through comparison of parameters such as PR curves and IoU values, I concluded that YOLOv3 has more advantages. On this basis, I proposed several improvements to YOLOv3, including improving the learning rate configuration, using the K-means clustering algorithm, and using the more lightweight model Tiny-YOLOv3. After retraining and testing the improved network, the experiments proved that the improved network has higher efficiency and accuracy. Therefore, this algorithm was chosen as the algorithm for the photovoltaic cleaning robot to detect defects.
(3) For the detection of hot spots on solar panels, considering that the fault points generate heat and the temperature is higher than the surrounding area, I adopted the method of detecting infrared images of solar panel hot spots. By comparing several infrared shooting devices, I chose the infrared thermal imager to capture solar panel hot spot images, and proposed a set of algorithms to detect hot spots based on the characteristics of infrared images. Through histogram correction, image filtering, threshold segmentation, and other steps, the hot spot position in the infrared image was detected, verifying the feasibility of the algorithm.
(4) The positioning of defects on solar panels was realized. After determining the robot’s walking path, the distance of the robot’s wheels relative to the ground was calculated through the pulse changes of the photoelectric encoder, obtaining the robot’s position on the solar panel. Combined with the camera internal parameters and distortion parameters obtained from camera calibration, the spatial coordinate transformation was performed on the defect coordinates in the solar panel image, and the single-frame static image model was used to obtain the actual distance from the defect center to the robot.
There are still many areas for improvement in this research:
(1) For the pixel-based defect detection method, the proportion of missed detection and false detection is still relatively large, and the algorithm can be further improved to increase the detection accuracy.
(2) For the improved YOLOv3 network, although the detection effect has improved compared with the ordinary YOLOv3 network, there is still room for improvement. Especially for images with irregularly shaped defects in the dataset, missed detection sometimes occurs. Further optimization of the network structure from aspects such as model pruning and loss function improvement can be considered.
(3) Currently, the infrared image detection can only detect the position of hot spots. The specific fault types, namely distinguishing the four different faults of hidden cracks, fragments, broken grid lines, and black cells, have not been studied yet. Further research will be conducted on this, and deep learning methods can be considered for detecting hot spots on solar panels.
(4) The odometry-based positioning method used in this paper has a relatively large possibility of error. Since the surface of solar panels is very smooth, the robot may deviate from the straight path due to slipping, causing a large difference between the positioning result and the true value. Therefore, further considerations are needed on how to ensure the robot moves along a straight line.