Methods and devices for processing input signals The present application claims benefit to and the priority of U.S. Provisional Application No. 62/287,222, filed January 26, 2016, the entire disclosure of which is hereby expressly incorporated herein by reference. Background of the invention

The invention relates to the field of signal processing.

Objective of the present invention

An objective of the present invention is to provide methods and devices for processing input signals in an efficient manner .

Brief summary of the invention

An embodiment of the present invention relates to a method for processing an input signal and generating an output signal based on the input signal, said method comprising the steps of:

- in an analysis stage, extracting a plurality of kernels from the input signal, wherein each kernel is described by a parameter vector that is defined by a given number of extracted kernel parameters, and

- forming the output signal based on the extracted kernel parameters . The input signal may be an N (N=l , 2 ,...) -dimensional signal.

The extracted kernel parameters are preferably quantized. The quantized kernel parameters are preferably coded, and said step of forming the output signal is preferably based on the coded kernel parameters.

The kernels may be extracted using a kernel density mixture model algorithm.

The kernel parameters may further be derived using an

Expectation-Maximization, EM, algorithm. The kernels are preferably Gaussian kernels that are defined by a one-dimensional or multi-dimensional Gaussian function.

The extracted kernel parameters are preferably transformed to another domain prior to quantizing and coding, preferably based on FFT (fast Fourier transform) , KLT (Karhunen-Loeve transform) , DCT (discrete cosine transform) or predictive coding operators .

The step of quantizing is preferably an adaptive quantizing step wherein the kernel parameters are quantized differently.

The step of coding may comprise a predictive coding.

Before quantizing the kernel parameters, a sparsification approach may be used to reduce the number of kernels.

According to a further embodiment, one or more iteration steps may be carried out before coding the quantized kernel parameters wherein each iteration step comprises:

- generating a reconstructed signal based on the quantized kernel parameters;

- comparing the input signal and the reconstructed signal, - determining the deviation between the reconstructed signal and the input signal,

- if the deviation exceeds a given threshold, modifying the analysis stage, preferably with respect to the number of kernels, and/or the quantizing, and repeating the

iteration until the deviation reaches an accepted range, and

- continuing with the formation of the coded signal based on the current analysis stage and/or the current quantizing if the deviation has reached the accepted range.

The coded signal may also be decoded. A reconstructed signal may be generated based on the decoded signal. Further, signal features may be derived from the decoded signal .

Furthermore, an enhanced signal with increased or decreased sample resolution may be derived from the decoded continuous kernel signal equation.

The N-dimensional input signal may describe a two-dimensional picture (e.g. N=2) . In this case , each parameter vector preferably comprises:

- at least two kernel parameters that define a position of an ellipse in the two-dimensional picture,

- at least one kernel parameter which defines the

orientation of the ellipse,

- at least one kernel parameter which defines the size of the ellipse, and

- at least one kernel parameter which defines the color

and/or the average intensity of the ellipse. Alternatively, the N-dimensional input signal may describe a video sequence (e.g. N=3) . The N-dimensional input signal may further be a one- or multi-dimensional audio or speech signal.

A further embodiment of the present invention relates to an encoder configured to process an input signal and generate a coded signal based on the input signal, the encoder carrying out the steps of:

- in an analysis stage, extracting a plurality of kernels from the input signal, each kernel being defined by a given number of extracted kernel parameters,

- quantizing the extracted kernel parameters,

- coding the kernel parameters, and

- forming said coded signal based on the coded kernel

parameters . A further embodiment of the present invention relates to a decoder configured to decode a coded signal that has been encoded by an encoder, said decoder being configured to carry out a signal reconstruction method for reconstruction of the encoder's input signal, said reconstruction method comprising the steps of:

- deriving from each decoded kernel parameter set an tri ^¬ dimensional interpolation function,

- deriving from each decoded kernel parameter set an tri ^¬ dimensional window function, and

- reconstructing each sample of the encoder input tri ^¬ dimensional signal based on a weighted combination of all interpolation and window functions, or a subset thereof. A further embodiment of the present invention relates to a datastream comprising coded kernel parameters that have been extracted from an N-dimensional input signal.

The kernel approach as described above allows the processing of signals in an efficient manner.

Brief description of the drawings

Hereinafter, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments thereof which are illustrated in the appended figures. Understanding that these figures depict only typical embodiments of the invention and are therefore not to be considered to be limiting of its scope, the invention will be described and explained with additional specificity and detail by the use of the accompanying drawings in which: Figure 1 shows a system comprising an encoder and a

decoder;

Fig. 2 7 show exemplary embodiments of the encoder and the decoder of Figure 1 according to the present invention; and

Figure 8 sualises embodiments of the kernel approach

used by the devices shown in Figures

Detailed description of the preferred embodiments

The preferred embodiments of the present invention will best understood by reference to the drawings, wherein identical or comparable parts are designated by the same reference signs throughout.

It will be readily understood that the present invention, as generally described herein, could vary in a wide range. Thus, the following more detailed description of the exemplary embodiments of the present invention, is not intended to limit the scope of the invention, as claimed, but is merely representative of presently preferred embodiments of the invention.

Figure 1 shows an encoder 100 and a decoder 200. The encoder 100 receives an input signal S which can be a one-dimensional signal (e.g. sound or speech), a two-dimensional signal (e.g. an image) or a multi-dimensional signal (e.g. plenoptic field or multispectral images) . The encoder 100 processes the input signal S and generates a corresponding output signal Sout. The decoder 200 receives the output signal Sout from the encoder 100 and generates a reconstructed signal Sr.

Figure 2 shows exemplary embodiments of the encoder 100 and the decoder 200 of Figure 1 in further detail. The encoder 100 of Figure 2 comprises an analysis stage 110 which uses a kernel approach to analyze the input signal S.

Figure 8 shows the kernel approach in an exemplary fashion. More specifically, Figure 8 depicts an example of a 3-D signal with 1000 kernels. The 3-D signal may be a grey-level video signal with 128x128 pixel per frame and 64 frames. It is apparent that each kernel steers in spatial as into temporal direction. Not shown in Figure 8 is the 4 ^th

dimension of each kernel that relates to the average grey value of each kernel. In temporal direction the intensity flow relates to the motion of a segment of pixels. Also seen in Figure 8 is the segmentation that can be performed in spatio-temporal dimensions. Each kernel is described by a parameter vector Θ- with elements that relate to location, steering (orientation), width and average grey-level value. Also a kernel weighting factor may be included.

The analysis stage 110 of Figure 2 extracts K kernels with respective kernel parameters Θ = [© _J , Θ ₂ ,..., (¾] from the input signal S. The kernel parameters may include center values, correlation matrices as well as texture parameters. The estimation of the parameters may be done using GMM estimation and/or Expectation-Maximization (EM) algorithms. The encoder 100 further comprises a quantizing and coding unit 120. The quantizing and coding unit 120 quantizes and codes the kernel parameters Θ = [© _J , Θ ₂ ,..., (¾] and generates an output signal Sout=0 which consists of or comprises the quantized and coded kernel parameters Θ = [Θι,Θ ₂ ,..,(¾] .

Alternatively, the kernel parameters Θ = [Θ _1; Θ ₂ (¾] may be transformed into another domain prior to quantization and coding as indicated in Figure 2. ^{Θ} designates the

transformed parameters.

The transform operator Τ{ } can be linear or non-linear and vary between individual kernels in order to make the approach adaptive. Possible operators include the FFT (fast Fourier transform) , KLT (Karhunen-Loeve transform) , DCT (discrete cosine transform) or predictive coding operators. It is also possible to perform adaptive quantization to different kernels and kernel's parameters, as well as predictive coding between kernel parameters and kernel ' s parameters .

In case of transformed parameters ^{Θ}, the quantizing and coding unit 120 generates an output signal which consists of or comprises the transformed, quantized and coded kernel parameters {θ] as indicated in Figure 2.

The decoder 200 of Figure 2 comprises a decoding stage 210 which decodes the received signal Θ or {θ} and determines the kernel parameters Θ =[Θ ₁ ,0 ₂ ,...,¾]. The decoder 200 of Figure 2 further comprises a synthesis stage 220 which is configured to generate the reconstructed

Signal Sr based on the kernel parameters Θ = [Θ _1; Θ ₂ (¾] .

Figure 3 shows another exemplary embodiment of the encoder 100 of Figure 1 in further detail. The encoder 100 of Figure 3 comprises the analysis stage 110, and the quantizing and coding unit 120 as described above with respect to Figure 2. The decoder 100 of Figure 3 further comprises a reduction unit 130 which reduces the number of kernels prior to quantization and coding. In Figure 3, the original number M of kernels & _M = [Θ _1; 0 ₂ ,...,0 _M ] is reduced to a smaller number K, K<M. The resulting kernels are designated by = [Θ _1; Θ ₂ ,...,Θ^] in Figure 3. The reduction unit 130 may be based on a sparsification approach. Figure 4 shows another exemplary embodiment of the encoder 100 of Figure 1 in further detail. The encoder 100 of Figure 4 comprises the analysis stage 110, and the quantizing and coding unit 120 as described above with respect to Figures 2 and 3. In contrary to the embodiment of Figure 3, a reduction unit 140 is arranged upstream of the analysis stage 110 and generates a reduced input signal S' for the analysis stage 110. The reduction unit 140 may be based on irregular

subsampling .

Figure 5 shows another exemplary embodiment of the encoder 100 of Figure 1 in further detail. The embodiment of Figure 5 comprises an analysis stage 110, a quantizing unit 121, a coding unit 122 and a synthesis stage 150. The analysis stage 110 may be identical with the analysis stages 110 described above with respect to Figures 2-4. Alternatively, the

analysis stage 110 of Figure 5 may also include an internal sparsification stage corresponding to the external

sparsification stages 130 and 140 as described above with respect to Figures 3 and 4.

The loop comprising the analysis stage 110, the quantizing unit 121 and the synthesis stage 150 is carried out in an iterative manner (i.e. in an Analysis-by-Synthesis approach) in order to optimize the number of kernels K and their parameters as well as the quantization prior to coding. In Figure 5, the quantized but not yet coded kernel parameters are designated by reference sign Θ . The loop depicted in Figure 5 can also be added to the non- coding embodiment of the encoder shown in Figure 7. Figure 6 shows another exemplary embodiment of the decoder 200 of Figure 1 in further detail. The decoder 200 comprises a decoding stage 210 and a synthesis stage 220 which may be identical or comparable to the decoding stage 210 and the synthesis stage 220 of Figure 2. In addition, the decoder 200 comprises a post-processing unit 230 and a sample rate conversion unit 240.

The post-processing unit 230 may be configured to extract features F, for instance MPEG-7-like features such as N-D (N- D: multi-dimensional) segments, N-D intensity flows and N-D gradients etc. Such features F may be used for segmentation, classification and similarity operations, for example with subsequent visualization of the result.

The rate conversion unit unit 240 may be configured to generate a N-D signal Sisr with reduced or increased sample resolution (i.e. for super-resolution). The results from the post-processing unit 230 and/or the rate conversion unit 240 may be used to generate reconstructed signals with reduced noise or reduced textures. Alternatively or additionally, a rate-conversion may be carried out to any precision based on the kernel parameters.

Figure 7 shows an embodiment of a system comprising an analysis stage 110, a synthesis stage 220, a post-processing unit 230 and a rate conversion unit 240. The system of Figure 7 corresponds to a system that comprises the decoder 200 of Figure 6 and an upstream encoder, with the exception that the coding and decoding steps are not used. The analysis stage 110 may include an internal sparsification stage. The kernel approach and the extraction of kernels with respective kernel parameters Θ = [Θ^Θ^.,.,Θ^ ] from an input signal S as well as the method of operation of the above mentioned stages and units, in particular the method of operation of the analysis stage 110 and the synthesis stage 220, will be explained hereinafter in further detail.

With the above-described steered kernel approach a N-D discrete signal is modeled by a weighted combination of M kernels. Each kernel acts as an expert in its respective region of the signal and steers along direction of highest similarity. An example of such a model p(x,y) is a linear combination of kernels Φ , such that

M

In the particular embodiment in Figure 8 the N-D signal is a 3-D pixel grid resembling 64 consecutive images, each of size 128x128 pixels (a stack of 64x128x128 samples) . Each pixel has a grey-level value. Each kernel in Figure 8 is thus of dimension N=4 (even though only 3-D kernels are shown for illustration purposes) . Vector x in the above equation is of dimension 3 (positions in the 3-D sample stack) and y of dimension 1 (the grey-level values). p(x,y) = p(x,y) may take the form of a joint probability density function. The invention is, however, not restricted to kernels that relate to

probability density functions, nor is the model restricted to linear combinations of kernels. In the above specific embodiment each kernel Φ. is a

weighted 4-D Gaussian kernel density function Φ, = ΤΓ · Ν(μ ,∑ ) with parameters Θ and

Here d e kernel

in the 3-D sample grid.

Parameter vector μ of size d is the center vector of the j th

Gaussian kernel and describes the location of the kernel in d-dimensional space. ∑■ is the covariance matrix of size dxd and describes the variances and steering (orientation) of the kernel in d-dimensional space.

In the above embodiment (Figure 8) each kernel has a center vector μ . of dimension d=4,

such that _j ^. , μ _{χ ]} and μ _{χ ]} are the kernel's location in the 3-D sample space and μ _γ is the average grey level value represented by the j th kernel. Parameter matrix ∑. is of dimension 4x4 and contains the variances of the kernel in each dimension as well as the covariance between the dimensions,

While the center vector μ _} describes the location of the j th kernel the covariance matrix describes the width and steering (orientation) of the kernel.

In the analysis stage 110 in Figure 2 the parameter vector

—j of each kernel is estimated based on the

—J

complete or a subset of the samples of the signal. In the embodiment in Figure 8 this is the 4-D signal (sample stack and sample grey level values) . The parameters may be

optimized using a stochastic framework, i.e. using the well- known iterative Expectation-Maximization (EM) algorithm.

In a preferred embodiment the model kernel parameters are optimized in the analysis step (110) using a sparsification system (140) in Figure 4 prior to optimization. For this purpose the N-D signal sample stack is divided into smaller, possibly overlapping regions. In each of these regions the sample variations are analyzed and an appropriate number of kernels used to initialize the optimization algorithm.

Algorithms such as the N-D Discrete Cosine Transform (N-D- DCT) or related orthogonal or non-orthogonal or even non ^¬ linear Transforms can be used for this purpose. It is

advantageous to analyze the sample variation activity in the Transform "frequency" domain as the normalized squared sum of the lowest N-D Transform AC coefficients. The regions with the higher spatial activity are admitted a higher number of seed kernels. What results after optimization of all model parameters

Θ = [Θ _ι ,Θ^...,©^] is a sparse model of a d-dimensional signal.

The d-dimensional signal is condensed into the location, width and steering parameters of the kernels. If kernels used can be described by space-continuous

analytical functions (such as the above Gaussian kernels) , the model arrives at a space-continuous model "equation"

M

In the embodiment depicted in Figure 2 the model parameters Θ = ® _Ϊ ,® _2> ··· _Μ ] ^are quantized and coded prior to storage and/or transmission. In all embodiments that require

quantization and coding of the model parameters (system (120) in Figures 2, 3 and 4) it is advantageous to employ

redundancy reduction techniques.

The specific model of the N-D signals is a point cloud of neighboring kernels with specific features (parameters) .

Center parameters and covariance are not necessarily

correlated and it is advisable to encode these parameters separately. It is advantageous to employ predictive coding strategies to encode the center vectors of the kernels using linear or non-linear prediction filters. A useful technique for coding the covariance matrices is to perform a Eigen- decomposition on the covariance matrix for each kernel, an example of a transform operator j according to init 120), followed by quantization and coding of the Eigenvectors and Eigenvalues. With such an approach it is possible to reduce the number of bits/kernel significantly.

In yet another embodiment a quantization codebook approach, such as a multi-dimensional vector quantizer, is used to encode the kernel center and/or shape and/or steering

information to arrive at an even further reduced bit rate. In this embodiment as well as in the other above embodiments it may be benefitial to transmit the quantizer code books to the receiver as part of the side information in the bitstream.

The above mentioned Eigendecomposition of the covariance matrices of the kernels using the transform operator

provides a coded representation with rich and powerful embedded MPEG-7 like features. Signal features (i.e. for images) are frequently used for similarity ranking and retrieval between different coded signals as well as for classification of content. Usually signal features (such as color, color distribution, edge information in images) are extracted from the original or the decoded signal samples and then further processed. In a preferred embodiment of the system in Figure 6, the coded parameters for each kernel are already the desired features used by the system 230 for segmentation,

classification and similarity operations. In this embodiment the coding strategy uses essentially MPEG-7 like features for coding and reconstruction of signals. This is in example the case, when coded kernel parameters, i.e. for image signals, include color and color location information (the kernel center vectors) as well as edge and directional gradient information (from the Eigenvalue and Eigenvector parameters of the transform operator ^{Θ,} ) . Other valuable features include the absolute and relative orientation of the number of axes of the kernels.

The purpose of the decoder system in Figure 8 (200) is to decode and reconstruct the d-dimensional signal from the coded parameters. This encompasses the processes of decoding the parameters (210) and synthesizing of the signal from the decoded parameters (220) .

It is well known from information and coding theory that the process of quantization of a signal (such as parameters

Θ = [Θ _1; Θ ₂ ,...,¾]) in an encoder (120) always results in

quantization artifacts introduced to the signal. As a result, the received and decoded model parameters Θ = [Θ _υ Θ^.,.,Θ^] from the decoder system (210) are not identical to the ones outputted from the analysis process (110) at the encoder - they are distorted.

The purpose of the synthesis process in system 220 of Figure 2 is to reconstruct the N-dimensional signal from the decoded and reconstructed model parameters (and thus distorted) model parameters_0 = [Θ _] ,^,...,©^] . This will result in a distorted reconstructed signal Sr in comparison to a signal

reconstructed from the original undistorted parameters (as in the embodiment depicted in Figure 7) . Various methods for synthesis of the signal from its model parameters (i.e. in synthesis system 220) can be envisioned. In a preferred embodiment the signal y(x) is reconstructed using an interpolation function m-fx) and a window function W _j (x) for each kernel, derived from the parameters Θ ₇ of the j th kernel, such that

y(x) = mix) =∑m,(x)- W _j (x)

In a further preferred embodiment each kernel is described by an analytical d-dimensional continuous function and provides an interpolation function with global support. The window function for each kernel defines its local support. This can be achieved i.e. using the above-described Gaussian kernels Ν(μ _, ,∑ _j ) · For the example used in Figure 8 this results in a formula for reconstruction of the grey level values y(x) at arbitrary locations x in the 3-D sample space using

K

y(x) = mix) =∑m .(x)' W _j (x)

7 =1

Here m.(x) is a 3-D linear interpolation function and w .(x) a

3D window function. An important feature of this specific embodiment is, that the regression function y(x) = m(x) is now a closed form analytical equation for the 3-D sample stack. As a consequence, pixels of the video signal can now be reconstructed at any sub-pixel location. This enables a rate- conversion of the video signal to any scale, based directly on the model parameters Θ = [Θ ₁₅ Θ ₂ ,...,<¾,-] . A particular feature of the steering kernel approach is, that the rate-conversion implicitly incorporates knowledge about direction of motion in the scene. For the above Gaussian kernels Ν(μ ,∑ ₇ ) , the linear interpolation functions m.(x) and window functions w .(x can be derived based on the model parameters directly, i.e. using

= μ _γ ,+∑ _YXi ^■ Σ ^"1 · (x - u

In the embodiment in Figure 7 the synthesis system 220 reconstructs the N-D signal Sr based on the non-quantized model parameters Θ = [Θ ₁₅ Θ ₂ ,...,<¾,-] . Since the sparse

representation of the signals using steered kernels is expected to be "edge-aware" and with reduced noise level compared to the original signal S, a high quality, edge- preserving noise reduced signal Sr is obtained. This is of advantage for many application. In addition, the now closed form analytical equation for the signal

K

allows highest quality rate conversion to any scale in system (240) directly from the equation.