Technical Field

The invention relates to the ES-Hybrid (Edge preserving sorting-hybrid) method in daytime/thermal images applied in real time on the FPGA to provide noise removal.

In particular, the invention relates to the ES-Hybrid method in daytime/thermal images that provides noise removal by preserving edges and structure in which results of median filtering and convolutional filtering are used as sorting-based.

Prior Art

The importance of noise removal is increasing day by day due to the common use of sensors in real-time systems in recent years. Common causes of noise can be classified as noise caused by analog circuits, production errors, bit errors and insufficient photon amount when obtaining images from the sensor. AWGN (Additive White Gaussian Noise) is frequently encountered in real-time applications. This type of noise, mainly caused by analog circuits, is widely used in noise reduction research topics. Structural defects should not be caused while removing noise in images. Noise removal methods are the methods in which continuous research / development studies continue from past to present and that minimize the noise without the structural deterioration of the image.

Other noise types studied within the scope of noise reduction can be classified as IN (Impulse Noise/Salt & Pepper), QN (Quantisation noise), PN (Poisson Noise) and SN (Speckle Noise). AWGN is basically noise caused by analog circuits during image acquisition or image transmission. Other types of noise are caused by errors during production, bit errors during image acquisition, or insufficient photon number.

Noise removal methods in the prior art can cause structural distortions, especially in thermal images, while removing noise. It is important not to lose the structure as well as to protect the edges as much as possible and reduce the noise during noise removal.

The following problems are experienced in the state of the art;

Excessive DSP consumption in real-time applications. • Distortions and overlapping of details resulting from noise removal in thermal images, detail losses.

• High amount of memory used.

• Obtaining successful results depending on the noise types.

As a result of the research made in the prior art, document no KR100927458B1 is encountered. The application relates to a method of processing a filter in a spatial domain with a combination of alignment and averaging. It is used in the application to reduce effects such as distortion and overvaluation caused by DCT and quantization in video compression. Pixels are weighted in a certain area and elimination, alignment and averaging processes are performed. Said method primarily evaluates IN (impulse noise) based noise as a noise type, and it is alignment and average based. However, the use of sorting-based median filtering method and convolutional filters, which provide noise removal by preserving the edges and structure is not contained in the application.

As a result, due to the abovementioned disadvantages and the insufficiency of the current solutions regarding the subject matter, a development is required to be made in the relevant technical field.

Aim of the Invention

The invention aims to solve the abovementioned disadvantages by being inspired from the current conditions.

The main aim of the invention is to provide noise removal by preserving the edges and structure on the image. In addition to the ability to protect the edges while removing noise, noise removal can be performed without focusing on the edges.

The advantages of the inventive method are as follows;

• No DSP consumption in real-time application or keeping the same to a minimum.

• Removing noise, protecting details, preventing overlapping by preserving the structure in thermal images.

• Noise removal by preserving edges in images. Optimizing the image with X-Model/Plus-Model filters.

Achieving successful results in most noise types.

• Low power consumption

• Low resource consumption

• Using low amount of memory.

• High operating frequency.

In order to fulfil the above aims, the present invention is a hybrid method applied to daytime or thermal images to provide noise removal, comprising the following process steps,

• taking daytime or thermal images,

• calculating the median value of all the pixels in a certain area of the input images, based on the central pixel, the median value of the 90° neighbourhoods and the median values of the 45°neighbourhoods,

• calculating X-Model and Plus-Model filtering in a certain area of input images,

• sorting the calculated results and taking the average according to the median value neighbourhoods,

• sorting using the calculated mean values and the center pixel,

• obtaining the output pixel by taking the average of the determined positions in the sorted result.

Description of the Figures

Figure 1 is the flow chart of the inventive method.

Figure 2 is a detailed flow chart of the inventive method.

Figure 3 is a 5x5 median flow chart. Figure 4 is a 5x5 median 45°flow chart.

Figure 5 is a 5x5 median 90°flow chart.

Figure 6 is a 3x3 X-Model flow chart.

Figure 7 is a 3x3 Plus-Model flow chart.

Figure 8 is the view of the 5 pixels sorting.

Figure 9 is the view of the center (median) pixel and average operations as a result of the sorting.

Figure 10 is a view of the output pixel by 3 pixels sorting and averaging.

Detailed Description of the Invention

In this detailed description, the preferred embodiments of the invention are described only for clarifying the subject matter in a manner such that no limiting effect is created.

Median value results in different directions and X-Model/Plus-Model filter results are used in the inventive method. After applying operations such as sorting and averaging on the obtained results, the output is obtained. The method is implemented on FPGA in real time. In addition to the high operating frequency, low power consumption, low computational load, noise removal is provided by protecting the edges and without creating effects such as structural defects/overlapping, especially in thermal images. The superiority of the method has been revealed by comparing the same with the prior methods according to the noise types as a result of the experimental studies and analyses.

The X-Model/Plus-Model filter is roughly obtained by combining SSF (Sharpening Spatial Filter) and CF (Clamp Filter).

SSF is a type of high pass filter. It enhances the details in the image. It increases the center pixel density by using neighbouring pixels to increase the brightness in a specified area. It is used to increase the description between brightness and darkness in the present invention. -i -i -r

SSF = -1 s -1

-1 -1 -1.

CF (Clamp Filter) is a type of low pass filter. It is known as a convolutional filter, which removes the distortions and unwanted spaced edges in the image. It is seen in frequent use of the filter that the perimeter of the center pixel is completely surrounded by ones.

1 1 T

CF= 1 C 1

.1 1 1.

High resolution images are usually created using large convolutional filters. However, the increase in filter size increases the memory and thus the hardware cost. 3x3 filters have low computational complexity. Combined filters are the application of successively applied filters at once.

Obtaining the X-Model/Plus-Model filter:

1 . As a result of the convolution of 3x3 SSF and 3x3 CF, a combined filter of 5x5 is obtained.

2. The 3x3 area of the 5x5 filter is cut from the position (2,2). If in the positions (1 ,2), (2,1), (2,3) and (3,2) of the 3x3 filter is called -4 - C + S = 0 to obtain the X-Model filter, the coefficients in these positions are reset and the C = S - 4 equality is obtained. If in the positions (1 ,1), (1 ,3), (3,1) and (3,3) of the 3x3 filter is called -2 - C + S = 0 to obtain the Plus-Model filter, the For X-Model, S-4 is written instead of C. For Plus-Model, S-2 is written instead of C. For both filters, a quadratic equation is obtained at position (2,2). For the X-Model filter, the variable change in the S ² - 4S - 8 = -2HL equation is applied and necessary simplifications are made. For the Plus- Model filter, the variable change in the S ² - 2S - 8 = 2HL equation is applied and necessary simplifications are made. The state of the filters after variable substitution and simplifications is given below.

6. Finally, the combined filter becomes amplified by the gains of the SSF and CF filter since it is obtained by the convolution operation. In order to balance these gain values, the gain value is determined as (H-3)*(L+3).

X-Model/Plus-Model filters are as follows:

X-Model filter is as follows;

Plus-Model filter is as follows; H and L parameters are obtained by mathematical inferences after convolution of SSF and CF filters. In other words, H and L parameters are derived from S and C parameters. Finally, it is used to determine the characteristics of X-Model/Plus-Model filters.

While S and C parameters will be obtained as complex roots after integers greater than 6 in the X-Model filter in the increasing values of the H*L value as positive integers, on the other hand, this effect will be seen at values less than -4 in the decreasing values of the H*L value as negative integers in the Plus-Model filter. Namely, the reactions of the two filters in the same operating ranges are different from each other. In addition, while it is S — C = 4 in the acquisition of the X-Model filter, it is taken as S — C = 2 in the Plus- Model filter.

5x5 windowing was preferred for the modified hybrid median filtering method applied. Different approaches such as 3x3, 7x7 etc. can be used as well. Pixels and their positions obtained in 5x5 windowing;

(1,1) (1.2) (1.3) (1.4) (1.5)

(2.1) (2.2) (2.3) (2.4) (2.5)

Ml= (3.1) (3.2) (3.3) (3.4) (3.5)

(4.1) (4.2) (4.3) (4.4) (4.5)

(5.1) (5.2) (5.3) (5.4) (5.5)

Diagram between neighbouring pixels used in hybrid median filter;

'H' represents horizontal pixels and 'V' represents vertical pixels. The diagonal pixels represent 45 degree neighbourhoods, while the vertical and horizontal pixels represent 90 degree neighbourhoods.

The general flow definitions of the inventive method are as follows;

1. Median operations are applied for three conditions. The first is the classical median filtering (s1 ) in Figure 3, the second is the median (s2) of the 45 degree neighbourhoods in Figure 4, third one is the median of 90 degree neighbourhoods in Figure 5(s3). s1 = median(M1 ). s2 = median((1 ,1 ), (1 ,5), (2,2), (2,4), (3,3), (4,2), (4,4), (5,1 ), (5,5)).

S3 = median((1 ,3), (2,3), (3,1 ), (3,2), (3,3), (3,4), (3,5), (4,3), (5,3)).

2. The 3x3 X-Model filtering in Figure 6 and the Plus-Model filtering process in Figure 7 are also applied over the same center pixel and two different values, s4 and s5, are obtained from there. The 5 different pixel values obtained are sorted from smallest to largest, as in Figure 8. (a = sort(s1 ,s2,s3,s4,s5))

3. Let a vector elements be a1 , a2, a3, a4 and a5, from smallest to largest. As in Figure 9, the average of the 1 st and 5th elements and the 2nd and 4th elements is taken. Then, sorting is performed again from smallest to largest using the obtained values and the a3 value. (Other than the above mentioned elements, the process can be continued by taking the average.)

4. Let's call the vector obtained as a result of sorting the vector b. Let the vector elements be b1 , b2 and b3. As in Figure 10, the average of b1 and b3 is taken and output as the output pixel. (It is not necessary to take the average of b1 and b3. In this step, necessary analyses can be made and different approaches can be applied.)

If the inventive method is to be detailed; It should be (3,3) = (3,3)' after the noise removal applied on the noisy image. Let the value obtained from the 5x5 median result in figure 3 be ml , Let the median value found from 45° neighbourhoods in Figure 4 be m2, Let the median value found from 90° neighbourhoods in Figure 5 be m3. Namely; ml = median (GLG). m2 = median (GLG (1 ,1), GLG (1 ,5), GLG (2,2), GLG (2,4), GLG (3,3), GLG (4,2), GLG

(4.4), GLG (5,1 ), GLG (5,5)). m3 = median (GLG (1 ,3), GLG (2,3), GLG (3,1), GLG (3,2), GLG (3,3), GLG (3,4), GLG

(3.5), GLG (4,3), GLG (5,3)).

Different conditions occur when the median of three different values obtained as in Figure 3, Figure 4 and Figure 5 and/or when the values are considered one by one.

1. The median of the ml , m2, m3 values is less than the desired (3,3) value (median(ml,m2, m3) < (3,3)).

2. The median of the ml , m2, m3 values is very close to the desired (3,3) value (median(ml, m2, m3) = (3,3)).

3. The median of ml , m2, m3 values is greater than the desired (3,3) value (median(ml,m2, m3) > (3,3)).

4. The largest of the ml , m2, m3 values is less than the desired (3,3) value (maximum(ml, m2, m3) < (3,3)).

5. The largest of the ml , m2, m3 values is very close to the desired (3,3) value (maximum(ml, m2, m3) = (3,3)).

6. The largest of the ml , m2, m3 values is greater than the desired (3,3) value (maximum(ml, m2, m3) > (3,3)).

7. The smallest of the ml , m2, m3 values is less than the desired (3,3) value (minimum(ml,m2,m3) < (3,3)). In this case, sub-conditions will occur. In other words, values other than the smallest value can be smaller or larger than (3,3). 8. The smallest of the ml , m2, m3 values is very close to the desired (3,3) value (minimum(ml, m2, m3) = (3,3)). In this case, sub-conditions will occur. In other words, values other than the smallest value can be smaller or larger than (3,3).

9. The smallest of the ml , m2, m3 values is greater than the desired (3,3) value (minimum(ml, m2, m3) > (3,3)).

If the first condition is considered, it means that a value smaller than the actual value to be obtained will be output. In such cases, a smoothing effect is observed in the images. The smoothing effect can cause overlapping of images as well as loss of edge information. The X-Model/Plus-Model results are used to eliminate all possible effects of the conditions. X-Model/Plus-Model filtering process has adjustable coefficients. These results are adjusted for effects, thus provides maximum convergence to the noise-free image.

Let the X-Model/Plus-Model results obtained as in Figure 6 and Figure 7 be x1 and p1 , respectively. x1 , p1 >median(m1 ,m2,m3) should be for the first condition.

Filter coefficients should be chosen to meet this condition. Here it can be x1 >maximum(m1 ,m2,m3),p1 >maximum(m1 ,m2,m3); x1 <maximum(m1 ,m2,m3),p1 >maximum(m1 ,m2,m3); x1 <maximum(m1 ,m2,m3),p1 <maximum(m1 ,m2,m3) or x1 >maximum(m1 ,m2,m3),p1 <maximum(m1 ,m2,m3).

When the 5 pixel values available in total are ordered from smallest to largest as in Figure 8, vector S5 is obtained as S5 = [a,b, c, d, e]. Here, the median value 'c' should converge to the desired (3,3) value and x1 and p1 values should help value 'c' to converge this value. To increase the precision, as represented in Figure 9, and c values are taken and sorted from smallest to largest. Let be the newly obtained vector S3 S3 = [fg h] as shown in Figure 10. The average of the 1 st and 3rd elements of the S3 vector is output pixel. So, it is considered as = (3,3) . The implications of this acceptance are as follows: a) If it is g = c then it will be obtained. ci+b

There will be — < c in the vector S5, ordered from smallest to largest. In this case, it should be — > c to converge to the desired value. In addition, the value d+e of — must ensure that the mean of the four elements is greater than 'c' and that it is the value at which the maximum convergence to (3,3) is obtained. In other words, it can be figured out that the pixel results from the X-Model and Plus- Model filters are greater than the median results from the convergence of the mean of the four elements other than the median value of the S5 vector to the value (3,3). b) If it is g = known that a < b < c < d < e is precisely in the vector S5. It should also be noted that there is a < b < c < (3,3). In this case, when ^b+d+2c is considered, the value of 'd' provides b < c < d equality and ^b+d+2c = (3,3) convergence. The convergence of the value obtained using the median value of the S5 vector and its two closest neighbours to the value of (3,3) requires that the pixel results from the X- Model and Plus-Model filters be greater than the median results. c) then it will known that a < b < c < d < e is

’

3 42c precisely in the vector S5. In this case, when — - — is considered, the value of ‘e' n4 e42c provides a < c < e equality and - s (3,3) convergence. It is known that f < g < h is in the vector S3. It is not possible to deduce h= c from the conditions. Because if such a condition occurs, it will diverge to (3,3). The convergence of the value obtained using the median value of the S5 vector and its two neighbours to the value of (3,3) requires that the pixel results from the X-Model and Plus- Model filters be greater than the median results (The same can be valid for condition 4 and condition 5).

It has been shown that the values from the X-Model/Plus-Model filters should be greater than the median results considering all possibilities for the relevant condition. It is possible to converge noisy pixels to the original pixels using the X-Model/Plus- Model filters.

The flow of the design on the FPGA is as follows: 1 . The incoming video image is taken with the control block. The received image is sent to the Fifo controller.

2. The Fifo controller is responsible for the task of writing and reading video lines.

3. The lines are read from the Fifos with the incoming third video line.

4. Mirroring is performed in the image by using the registers suitable for the video data width together with the incoming lines. This operation is used in the boundary conditions (corners and edges) of the video frame. X-Model/Plus- Model calculators and median calculators calculate on all pixels along the active area of the video frame and sends the calculation results to the sorting blocks.

5. After the sorting and averaging operations are performed, the calculations are sent to the output controller along with the synchronous video signals.

6. The output controller synchronously transmits the video frame signals from the sorting blocks and average calculators. The calculations can be made or the incoming video can be output without being calculated with the bypass signal control in the output controller.

In experimental studies, the results were evaluated using PSNR (Peak Signal-To- Noise Ratio), RMSE (Root Mean Square Error) and SSIM (Structural Similarity Index for measuring image quality) image quality metrics. “Salt & pepper” noise was applied to the daytime image and the results of the study performed in the noisy image are evaluated in Table 1.

Table 1

“Salt & pepper” noise was applied to the thermal image and the results of the studies performed on the noisy image are evaluated in Table 2 below.

Table 2

Gaussian noise was applied to the thermal image with a variance value of 10, and then the results of the studies performed on the noisy image are shared in Table 3.

Table 3 Gaussian noise was applied to the thermal image with a variance value of 20, and then the results of the studies performed on the noisy image are shared in Table 4.

Table 4

Gaussian noise was applied to the thermal image with a variance value of 30, and then the results of the studies performed on the noisy image are shared in Table 5.

Table 5 “Poisson” noise was applied to the thermal image, and then the results of the studies performed on the noisy image are shared in Table 6.

Table 6

“Poisson” noise was applied to the daytime image, and then the results of the studies performed on the noisy image are shared in Table 7.

Table 7