FILTER ON FPGA Technical Field

The invention relates to methods of obtaining X-Model/Plus-Model filter by combining SSF (Sharpening Spatial Filter) and CF (Clamp Filter) and scaling high quality video with this filter in real time on FPGA. State of the Art

In recent years, video scaling methods have become increasingly important as a result of the increase in video resolutions (2K, 4K, 8K, etc.) and video quality. The methods proposed in recent years mainly include deep learning and machine learning methods. Implementation of these methods in FPGA and/or real-time implementation in embedded systems causes computational complexity and significant amount of resource consumption. When it is desired to obtain a high-resolution video from a low-resolution video, bilinear interpolation appears as one of the most frequently used methods. After the application of bilinear interpolation, blur effect and aliasing artifact are seen in the video. In the state of art, an image processing method used in aerial refueling aircraft is disclosed in the application numbered US2021042894A1. The method is based on the application of contrast limited adaptive histogram equalization technique (CLAHE), which is a local image processing technique, on FPGA. When applying CLAHE, the image is divided into tiles of different sizes. After the required operations, these tiles are combined to obtain the final image with the same resolution. When these tiles are considered as pieces of a puzzle and brought together, interpolation is applied to smooth the transition of the image between neighboring tiles. Filtering operations are applied horizontally and vertically. The computational complexity is high, and the filter dimensions are high and there are multiple different filters. The delay calculations of the filter used are as follows. Table 1 : Filter Delay Calculations in Application Number US2021042894A1

The reason for x8 overcalculation is that 3x9 and 9x3 filtering will be applied. If CLAHE is applied with read and write to DDR, +1 frame delay time should be added on it. Also for an application like CLAHE the maximum operating frequencies can be maximum 200- 225MHz.

As a result, it was deemed necessary to make an improvement in the relevant technical field due to the disadvantages mentioned above and the inadequacy of the existing solutions on the subject due to differences in the purpose, tool, functionality and application of the methods. Object of the Invention

With the invention, the developed X-Model/Plus-Model filter is applied before the application of interpolation to increase the quality of the scaled video and eliminate the mentioned problems. In addition to providing the elimination of these effects, the filter of the invention is implemented in real time with advantages such as low computational complexity, low memory requirement, low power and low resource consumption and high operating frequency.

X-Model/Plus-Model filters are combined filters based on the convolution of two different filters. In the high-quality video scaling implemented in the FPGA in real time with the X- Model/Plus-Model filter, the main idea is based on increasing the video resolution. For example, if the incoming image has a resolution of 640x480, this is a method used to increase the quality of the video by scaling the image to 1920x1080 resolution. The incoming image is filtered through the X-Model/Plus-Model filter and then sampled to the desired higher resolution. Significant improvements have been achieved in signal-noise ratios and structural similarities of videos that are filtered through the X-Model/Plus-Model filter and then scaled to high resolution, compared to real videos.

Compared to the prior art, the invention is used when scaling video from a low resolution to a high resolution and it increases the signal noise-ratio and the structural similarity of the image to real life. The interpolation application is used to scale the video to higher resolutions. The recommended filtering process uses only 5 out of 9 pixels in a 3x3 area with single filtering. X-Model/Plus-Model filter, which is mathematically obtained by convolutional combining Sharpening Spatial Filter and Clamp Filter and then reduction of the result is made and optimized values that work best

The structural and characteristic features of the invention and all its advantages will be understood more clearly by means of the figures given below and the detailed explanation written with reference to these figures.

Description of the Figures

Figure 1 shows the combined filter obtained by the convolution of 3x3 SSF and 3x3 CF.

Figure 2 shows the cut off 3x3 area which obtained by cutting 5x5 convolutional filter starting from (2,2) position.

Figure 3 shows the X-Model filter created by setting the coefficients (1 ,2), (2,1 ), (2,3) and (3.2) to zero of the cut off 3x3 area and the Plus-Model filter created by setting the coefficients (1 ,1 ), (1 ,3), (3,1 ) and (3,3) to zero of the cut off 3x3 area.

Figure 4 shows the filters obtained when S-4 is written instead of C for the X-Model and S-2 is written instead of C for the Plus-Model and the graphs of the quadratic equations in the range [-60 60].

Figure 5 is the view after the mathematical variable change and simplification applied in the filters

Figure 6 shows images obtained by doubling the resolution of video frames of different resolutions.

Figure 7 shows the differences of results between 1080p (FHD) video frame to 1440p (2K) scaling and background details. Figure 8 shows the differences of results between 1080p (FFID) video frame to 2160p (4K) scaling and background details.

The figures are not necessarily to scale and details not necessary for understanding the present invention may have been omitted.

Detailed Description of the Invention

In this detailed description, the preferred embodiments of the invention are described only for a better understanding of the subject and without causing any limiting effect.

The developed X-Model/Plus-Model filter is basically obtained by combining SSF (Sharpening Spatial Filter) and CF (Clamp Filter). To summarize these two filters briefly: SSF is a type of high-pass filter. It enhances the details in the image. It increases the center pixel density by using neighboring pixels to increase the brightness in a specified area. The reason for using it here is to increase the definition between brightness and darkness.

-1 -l -r

SSF -l s -1 -1 -1 1

CF is a type of low pass filter. It is known as a convolutional filter, which removes the distorting effects and unwanted gapping edges in the image. In frequent use of the filter, it is seen that the perimeter of the center pixel is completely surrounded by ones. l l r

CF 1 c 1 1 1 1

High-resolution images are often created using large-size convolutional filters. Flowever, the increase in filter size increases the memory and thus the hardware cost. 3x3 filters have low computational complexity. Combined filters can be considered as one-time application of filters that are applied in succession.

The steps for obtaining the X-Model/Plus-Model filter are as follows:

The 5x5 combined filter in Figure 1 is obtained by the convolution of 3x3 SSF and 3x3 CF.

The 3x3 area of the 5x5 filter is cut out starting from the position (2, 2). • To obtain the X-Model filter, setting -4 - C + S = 0 at (1 , 2), (2, 1 ), (2, 3) and (3, 2) positions of the 3x3 filter, these coefficients are set to zero and the equation C = S - 4 is obtained. The filter becomes as shown at the left in Figure 3. To obtain the Plus- Model filter, setting -2 - C + S = 0 at (1 , 1 ), (1 , 3), (3, 1 ) and (3, 3) positions of the 3x3 filter, these coefficients are set to zero and the equation C = S - 2 is obtained.

• For X-Model, S-4 is written instead of C. For Plus-Model, S-2 is written instead of C. For both filters, a quadratic equation is obtained at position (2, 2). In Figure 4, the graphs of the obtained filters and the quadratic equations in the range [-60 60] are given, with the upper ones being for the X-Model filter and the lower ones being for the Plus-Model filter.

• For X-Model filter, variable change is made, and necessary simplification is carried out in the equation S ² - 4S - 8 = -2HL. For Plus-Model filter the equation S ² - 2S - 8 = 2HL is used. Figure 5 shows the filters after variable change and simplification.

• Finally, because the combined filter is obtained by convolution operation, it will be amplified by the gains of the SSF and CF filters. To compensate for this gain value, the gain value is set as (FI-3) ^* (L+3). Another method may be used to compensate for the gain values.

The developed X-Model filter is in the form given below:

-1 0 -11

XF 0 H * L 0 / ((H - 3) ^* (L+3)) L— 1 0 -1J

The developed Plus-Model filter is in the form given below: In the increasing values of the H ^* L value as positive integers, the S and C parameters will be obtained as complex roots after integers greater than 6 in the X-Model filter; in the Plus-Model filter, this same effect will be seen at values smaller than -4 in the decreasing values of the H ^* L value as negative integers. In other words, the responses of the two filters in the same operating ranges are different from each other. In addition, it should be noted that while S - C = 4 in obtaining the X-Model filter, S - C = 2 in the Plus-Model filter.

In order to apply a 3x3 filter on real-time video, 3x3x3 = 27 multiplication/division operations and 3x3x3 = 27 addition/subtraction operations are required in the calculation to be made for each pixel in the RGB color space. When the same operation is done in the YCbCr color space, it is sufficient to apply the filter only on the Y(luma) values. Therefore, 9 multiplication/division and 9 addition/subtraction operations will be required for each pixel. Thus, the computational complexityis reduced by 1/3.

In a 3x3 filter, a total of 9 pixels will be processed in the time domain. In an X-Model/Plus- Model filtering, a total of 5 pixels are processed. In other words, the total workload is reduced to 5 multiplication/division and 5 addition/subtraction operations. If the multiplication/division operations to be applied on FPGA are 2 and/or multiples of 2, these operations can be handled by shifting operation without multiplication/division. The design we made here is designed to be applied both with and without multiplication/division. In the application where we do not do multiplication/division, the computational complexityconsists of only 5 addition/subtraction operations.

The design flow is as follows:

• The video image that comes as YCbCr is taken with the control block. YCbCr here can be 4:4:4 or 4:2:2. The received image is sent to the fifo controller. The fifo controller is responsible for writing and reading video lines.

• As the arrival of the second video line, the first line is read from the fifo. The second video line and the first video line read from the fifo go to the X-Model/Plus-Model calculator.

• Mirroring is done on the image by using the registers suitable for the video data width with the two incoming lines. This operation is used in the boundary conditions (vertices and edges) of the video frame. The X-Model/Plus-Model calculator makes calculation on all pixels along the active area of the video frame and sends the calculation results together with the synchronous video signals to the output controller.

• The output controller outputs the buffered CbCr values synchronously together with the video frame signals coming from the X-Model/Plus-Model calculator. With the bypass signal control in the output controller, X-Model/Plus-Model filter calculation can be made or the incoming video can be output without making calculation.

• Bilinear interpolation is applied to scale video passing through the X-Model/Plus- Model filter.

In the experimental studies, both PSNR values and the image were evaluated from visual point of view. There are images obtained by doubling the resolution of video frames of different resolutions at Figure 6. The images on the left were obtained by applying bidirectional linear interpolation only, while the images on the right were obtained by applying bilinear interpolation after passing through the X-Model/Plus-Model filter. Table 2 shows the PSNR values of cases where different samples are scaled to two times resolution. On the left are the results of bilinear interpolation, in the middle are the results obtained with the FPGA-friendly X-Model filter coefficients (where multiplication and division operations are handled by the shift operator), and on the far right are the PSNR values of the results obtained with the FPGA-friendly Plus-Model filter coefficients for which the filter provides maximum results. Table 2: PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index for

Measuring Image Quality) Results by Data Sets

In Figure 7, 1080p (1920 x 1080) is scaled to 1440p (2560 x 1440) resolution using an FFID (Full FID) image. That is, a FFID video frame is scaled to a 2K video frame. In the obtained results, sections were taken by zooming into the recorded 2K video frames to show the effect of the applied X-Model filter on the background image. In the left frame, there is the result with only bilinear interpolation, while in the right frame, there is the result that is passed through the X-Model filter and then bilinear interpolation is applied.

In Figure 8, 1080p (1920x1080) is scaled to 2160p (3840 x 2160) resolution using an FHD (Full FID) image. That is, a FFID video frame is scaled to a 4K video frame. In the obtained results, sections were taken by zooming in on the recorded 4K video frames to show the effect of the applied X-Model filter on the background image. In the left frame there is the result only with bilinear interpolation, while in the right frame there is the result that is filtered through the X-Model filter and then bilinear interpolation is applied. Delay calculations of X-Model/Plus-Model Filter are as given below.

Table 3: Delay calculations of X-Model/Plus-Model Filter