SPECIFICATION

BACKGROUND

RELATED APPLICATIONS

[0001] The present application claims the priority of U.S. Provisional Application

Serial No. 62/685,460 filed on June 15, 2018, the entire disclosure of which is expressly incorporated by reference.

TECHNICAL FIELD

[0002] The present disclosure relates generally to the field of computer vision technology. More specifically, the present disclosure relates to computer vision systems and methods for optimized computer vision using deep neural networks and Lipschitz analysis.

RELATED ART

[0003] Convolutional neural network (“CNNs”) are widely used in machine learning and are an effective tool in various image processing tasks, such as classification of objects. In particular, CNNs can be used as feature extractors to extract different details from images to identify objects in the images. As a feature extractor, CNNs are stable with respect to small variations in the input data, and therefore, perform well in a variety of classification, detection and segmentation problems. As such, similar features are expected when inputs are from the same class.

[0004] The stability to deformation of certain CNNs can be attributed to sets of filters that form semi-discrete frames which have an upper bound equal to one. This deformation stability is a consequence of the Lipschitz property of the CNN or of the feature extractor. As such, the upper bound can be referred to as a Lipschitz bound.

[0005] However, current CNNs can be fooled by changing a small number of pixels, thus leading to an incorrect classification. This can be the result of an instability of the CNN due to a large Lipschitz bound or a lack of one. Therefore, there is a need for computer vision systems and methods which can determine the Lipschitz bound for different types of CNNs, thereby improving the ability of computer vision systems to tolerate variations in input data. These and other needs are addressed by the computer vision systems and methods of the present disclosure.

SUMMARY

[0006] The present disclosure relates to computer vision systems and methods for optimized computer vision using deep neural networks and Lipschitz analysis. A neural network, such as a CNN, is a multiple layer network with learnable weights and biases that can be used for, among other things, analyzing visual imagery. The system of the present disclosure receives signals or data related to the visual imagery, such as data from a camera, and feed-forwards the signals/data through the multiple layers of the CNN. At one or more layers of the CNN, the system determines at least one Bessel bound of that layer. The system then determines a Lipschitz bound based on the one or more Bessel bounds. The system then applies the Lipschitz bound to the signals. Once the Lipschitz bound is applied, the system can feed-forward the signals to other processes of the layer or to a further layer.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The foregoing features of the invention will be apparent from the following

Detailed Description of the Invention, taken in connection with the accompanying drawings, in which:

[0008] FIG. 1 is a block diagram illustrating a structure of a layer of a convolution neural network;

[0009] FIG. 2 is a block diagram illustrating three types of merging operations of the present disclosure;

[0010] FIG. 3 is a block diagram illustrating a multiple layer convolutional neural network;

[0011] FIGS. 4A-4B are block diagrams illustrating a pooling filter;

[0012] FIGS. 5A-5B are block diagrams illustrating a multiple layer convolutional neural network;

[0013] FIG. 6 is a flowchart illustrating processing steps carried out by the computer vision system of the present disclosure;

[0014] FIG. 7 is a diagram illustrating a convolutional neural network with multiple layers, in accordance with the present disclosure; and

[0015] FIG. 8 is a diagram illustrating hardware and software component capable of implementing the system of the present disclosure.

DETAILED DESCRIPTION

[0016] The present disclosure relates to computer vision systems and methods for optimized computer vision using deep neural networks and Lipschitz analysis as described in detail below in connection with FIGS. 1-8.

[0017] By way of background and before describing the systems and methods of the present disclose in detail, the structure and properties of convolutional neural networks (“CNNs”) will be discussed first. It should be noted that the CNNs discussed below relate to a generic CNN. However, those skilled in the art would understand that the method and exemplary embodiments in this disclosure can pertain to any CNN, including but not limited to, scattering CNNs, fully connected CNNs, sparsely connected CNNs, etc.

[0018] A CNN can contain multiple layers, where each layer can consist of different or similar features. FIG. 1 illustrates a layer 100 of a CNN. The layer 100 of FIG. 1 includes an input node 102, a convolution filter 104, a dilation operation 106, a detection operation 108, a merge filter 110, an output node 112, a pooling filter 114 and output generation 116. It should be understood that although the nodes, filters and the operations will be discussed in the singular context, any layer of the CNN can contain any number of nodes, filters and operations.

[0019] The input node 102 can process one or more signal(s) and/or data, such as image data (e.g., pixels) or audio data. The input node 102 can be derived from an output node of a previous layer of the CNN or, when a layer of the CNN is a first layer, the input node 102 can be an initial input or signal. For example, a camera can be positioned to record traffic patterns in a particular area. The data from the camera can be fed into the CNN, where each image can be converted into an input node 102 that is fed into a first layer of the CNN. The first layer can then apply its filters and operations to the input signal of the input node 102 and produce an output node 112, which can then be fed into a second layer of the CNN.

[0020] The convolution filter 104 is a filter that can apply a convolution operation to the signal from the input node 102. For example, the operation can apply one or more convolution filters to different sections of the input signal. The result of the operation produces an output signal that is fed-forward to a next filter or process in the next layer of the CNN, or the result can be transformed into an output node. It should be noted that the input signal can be described with the symbol (“y”) and the convolution filter 104 can be described with the symbol (“g”) throughout the figures and description of the present disclosure.

[0021] The dilation operation 106 is an operation that can dilate an element of the output signal and/or data by a predetermined factor. For example, if the signal and/or data is represented in a 3x3 matrix, the dilation operation can dilate the 3x3 matrix into a 7x7 matrix. It should be noted that the dilation operation 106 can be described with the symbol hroughout the figures and description of the present disclosure.

[0022] The detection operation 108 is a nonlinear operation(s) that is applied pointwise to the output signal from the convolution filter 104. For example, the nonlinear operation can be a Lipschitz constant, a rectified linear unit (“ReLU”), a sigmoid, etc. The application of the nonlinear operation can improve the robustness of the CNN by preventing instability. Specifically, the nonlinear operation can prevent a value(s) from the input signals from uncontrollably increasing and becoming unmanageable when the values are processed through, for example, the merge filter 110 and the pooling filter 114. It should be noted that the detection operation 108 can be described with the symbol (“s”).

[0023] To optimize the performance of computer vision systems which rely on

CNNs, the present disclosure determines an optimal Lipschitz bound for the CNN. This determination can occur during the detection operation 108. By determining the optimal Lipschitz bound, they can be a significant reduction in identification errors, which would enable the CNN to conduct a more accurate analysis. Thus, the methods and embodiments discussed herein produce a significant improvement in the functioning of computer vision systems. Processes for determining the optimal Lipschitz bound are discussed in more detail below. [0024] The merge filter 110 is a filter that merges two or more outputs from the detection operation 108 by a pointwise operation to produce a single output. FIG. 2 shows three examples of possible merging operations. The first example is sum merging 202, which can, for example, take input signals from a filter (e.g., the convolution filter), apply a nonlinearity function respectively (e.g., a Lipschitz bound), and

produce a sum. The output of the sum merging 202 can be defined by the following formula, where k is a total number of inputs:

[0025] The second example is p-norm aggregation 204, which can, for example, take input signals y ₁ , y ₂ , y _k from a filter, apply a nonlinearity function d ₁ , d ₂ , d _k , respectively, and apply a pointwise p-norm aggregation. The output of the p-norm aggregation 204 can be defined by the formula:

[0026] The third example is pointwise multiplication merging 206, which can, for example, take input signals yi, y ₂ , y _k from a filter, apply a nonlinearity function si, s ₂ , O _k , respectively, and apply a pointwise multiplication. The output of the pointwise multiplication merging 206 can be defined by the formula:

[0027] It should be noted that the merge filter 110 of FIG. 1 can be described with the symbol (“0”) for sum merging 202 and with the symbol (“0”) for pointwise multiplication merging 206 throughout the figures and description of the present disclosure. It should further be noted that the output value“z” can be used to determine the Lipschitz bound. [0028] The output node 112 of FIG. 1 can be a node that includes one or more output signals produced in a layer of the CNN. As noted above, the output node 112 can transition into an input node for a further layer. For example, as seen in FIG. 3, input node l02a can feed another layer 302. Within the layer 302, one or more filters and/or operations can be applied to the input signals of the input nodes l02a to generate output signals. The output signals can exit the layer 302 as output nodes H2a, and/or are generated by output generation process H6a. The output nodes H2a can enter layer 304 as input nodes l02b and the input signals from input nodes l02b can be subjected to one or more filters and/or operations. It should be understood by those skilled in the art that the layer 302 and the layer 304 can contain the same or different filters and/or operations. The above described functions can repeat until layer M 306 is reached.

[0029] Returning to FIG. 1, the pooling filter 114 is a filter that can lower the dimensionality or bandwidth of the input signals for output generation 116. A first example of the pooling filter 114 is“max” pooling, which is shown in FIG. 4A. Max pooling is an operation that includes taking the maximal element among the elements in a sub-region. Translation and dilation can be used to separate elements in a sub-region to distinct channels and an L ^¥ aggregation process selects the largest element.

[0030] A second example of the pooling filter 114 is average pooling, which is shown in FIG. 4B. Average pooling is an operation of determining an average value among the elements in a sub-region. The values of three elements are combined and divided by the number of elements to determine an average pooling value.

[0031] It should be understood that the structure of the layer in FIG. 1 is only exemplary. Those skilled in the art would understand that any combination of the filters and operations discussed in FIG. 1 can be included in any layer of the CNN and the filters and operations can be in different positions along the flow of the layer. For example, FIGS. 5A and 5B are different layers within a scattering CNN. In particular, FIG. 5A shows a scattering CNN 52 with pointwise p-norms. FIG. 5B shows a scattering CNN 54 with multiplication. It should be understood that a scattering CNN is only one type of CNN and, as discussed above, other CNNs can be used (e.g., fully connected CNNs, sparsely connected CNNs, etc.)

[0032] It should further be noted that the merging operations of the merge filter

110 and the pooling operations of the pooling filter 114 aggregate input signals from the input nodes 102 filters and/or signals. As noted discussed above, a nonlinear operation, such as an operation which imposes a Lipschitz bound, can prevent the values from the input signals from uncontrollably increasing and becoming unmanageable when the values are processed through the merge filter 110 and the pooling filter 114. Additionally, a Lipschitz bound can be imposed on a signal, after which the signal can proceed to a next layer.

[0033] FIG. 6 is a flowchart illustrating processing steps 600, according to the present disclosure, for determining an optimal Lipschitz bound of a layer in the CNN. As noted above, by determining the optimal Lipschitz bounds for the CNN, the system of the present disclosure greatly improves the functionality of computer vision systems.

[0034] In step 602, a first layer of the CNN receives a first input node. The first input node includes a first input signal. For example, the input signal can be a matrix representative of an image. In step 604, the input signal can pass through a first filter. The first filter can be the convolution filter 104. The convolution filter 104 applies the convolution operation to the input signal. The result of the operation produces an output signal that is fed-forward to the detection operation 108.

[0035] In step 606, the detection operation 108 receives the output signal and determine at least one type of Bessel bound for the first layer. Two set of formulas for determining the different types of Bessel bounds will be discussed. The first set of Bessel bound formulas, seen below, can be used to determine three types of Bessel bounds if the first layer does not contain the merge filter 110:

where: type of Bessel bound for an m-th layer with no merge

filter; type of Bessel bound an for m-th layer with no merge

filter; type of Bessel bound an for m-th layer with no merge filter;

for each stands for a matrix that contains

the Fourier transform

stands for the

diagonal matrix that has the Fourier transform

at w for its entry; and

is the the diagonal matrix with

as its

[0036] The second set of Bessel bound formulas, seen below, can be used to determine three types of Bessel bounds if the first layer contains the merge filter 110.

where: type of Bessel bound for an m-th layer with a merge filter; type of Bessel bound an for m-th layer with a merge

filter; type of Bessel bound an for m-th layer with a merge filter;

[0037] It should be understood by those skilled in the art that the amount and type of Bessel bounds determined in 606, as well as the formulas for determining each type of Bessel bound, can be different depending on the filters and operations in a layer (e.g., the first layer). Thus, the two sets of formulas used in this disclosure to determine three types of Bessel bounds based on whether the first layer contains the merge filter 110 is only exemplary.

[0038] In step 608, the detection operation 108 can determine the Lipschitz bound for the first layer. The Lipschitz bound can be determined based on the type of the Bessel bound(s) determined in step 606. Bessel bounds B _m ⁽¹⁾ , B _m ⁽²⁾ , and B _m ⁽³⁾ can be determined in step 606 and, in step 608, the Lipschitz bound can be determined by the following first Lipschitz calculation:

[0039] As discussed above, the value“z” relates to the output value of the merge operation used by the merge filter 110.

[0040] The Lipschitz bound can also be determined in step 608 by the following second Lipschitz calculation using only Bessel bound

[0041] Additionally, the Lipschitz bound can also be determined in step 608 by the following third Lipschitz calculation using only

[0042] In step 610, the determined Lipschitz bound is applied to the output signal.

It should be understood that in step 608, the first Lipschitz calculation, the second Lipschitz calculation and the third Lipschitz calculation can produce different Lipschitz bound values. As such, in step 610, the Lipschitz bound value that is closest to optimality can be selected. Alternatively, a different Lipschitz bound value can be selected based on a predetermined parameter. In step 612, the output signal is fed-forward to the next process or filter in the first layer or to a next layer. For a first example, the output signal can be fed-forward to a merge filter 110 or a pooling filter 114. Alternatively, the output signal can be fed-forward to a next layer of the CNN.

[0043] FIG. 7 illustrates a CNN with multiple (four) layers 700. Using the steps described in FIG. 6, the following Fourier formula will produce the Bessel bounds shown below in Table 1.

[0044] The function on the Fourier domain supported on (-1,1) is defined as:

[0045] The Fourier transforms of the filters to be C” gate function are defined as:

[0046] Applying the first Lipschitz calculation to the determined Bessel bounds in

Table 1 produces a Lipschitz bound of 2.866. Applying the second Lipschitz calculation to the determined Bessel bounds in Table 1 produces a Lipschitz bound of 4.102. Applying the third Lipschitz calculation to the determined Bessel bounds in Table 1 produces a Lipschitz bound of 5. As such, the Lipschitz bound value of 2.866, as determined by the first Lipschitz calculation, is the closest to optimality. Thus, for example, the Lipschitz bound value of 2.866 can be selected in step 610 of FIG. 6 and applied to the output signal.

[0047] FIG. 8 is a diagram showing a hardware and software components of a computer system 802 on which the system of the present disclosure can be implemented. The computer system 802 can include a storage device 804, computer vision software code 806, a network interface 808, a communications bus 810, a central processing unit (CPU) (microprocessor) 812, a random access memory (RAM) 814, and one or more input devices 816, such as a keyboard, mouse, etc. The server 802 could also include a display (e.g., liquid crystal display (LCD), cathode ray tube (CRT), etc.). The storage device 804 could comprise any suitable, computer-readable storage medium such as disk, non-volatile memory (e.g., read-only memory (ROM), eraseable programmable ROM (EPROM), electrically-eraseable programmable ROM (EEPROM), flash memory, field-programmable gate array (FPGA), etc.). The computer system 802 could be a networked computer system, a personal computer, a server, a smart phone, tablet computer etc. It is noted that the server 802 need not be a networked server, and indeed, could be a stand-alone computer system.

[0048] The functionality provided by the present disclosure could be provided by computer vision software code 806, which could be embodied as computer-readable program code stored on the storage device 804 and executed by the CPU 812 using any suitable, high or low level computing language, such as Python, Java, C, C++, C#, .NET, MATLAB, etc. The network interface 808 could include an Ethernet network interface device, a wireless network interface device, or any other suitable device which permits the server 802 to communicate via the network. The CPU 812 could include any suitable single-core or multiple-core microprocessor of any suitable architecture that is capable of implementing and running the computer vision software code 806 (e.g., Intel processor). The random access memory 814 could include any suitable, high-speed, random access memory typical of most modern computers, such as dynamic RAM (DRAM), etc.

[0049] Having thus described the system and method in detail, it is to be understood that the foregoing description is not intended to limit the spirit or scope thereof. It will be understood that the embodiments of the present disclosure described herein are merely exemplary and that a person skilled in the art can make any variations and modification without departing from the spirit and scope of the disclosure. All such variations and modifications, including those discussed above, are intended to be included within the scope of the disclosure. What is intended to be protected by Letters Patent is set forth in the following claims.