TECHNICAL FIELD

The invention relates to methods, encoders, decoders, computer programs, and computer program products relating to compressing and decompressing haptic data for a plurality of channels.

BACKGROUND

Haptic internet is seen as the next step on mobile networking. Users are currently able to efficiently communicate over voice and video, but in the future networked society, it is envisioned that people will be able to communicate the sense of touch via haptic devices. There has been a large amount of research on devices, which allow such communication to take place. Several haptic interfaces are now appearing in the market, which deliver haptic feedback using different actuation technologies such as ultrasound, vibrotactile, electrostatic and piezoelectric.

Haptic perception consists of both kinesthetic and tactile sense and relates to the sensation of the size, shape, mass, texture and stiffness of physical objects, surfaces, etc. Kinesthetic information refers to the information perceived when moving joints, muscles and tendons, while tactile

information refers to information retrieved via the skin. In a haptic system, the amount of data representing all the sensors quickly becomes large.

Hence, data compression is needed to keep the data transfer requirements at a reasonable level. However, the data compression and decompression can introduce delays. Since any delay can seriously reduce the experience of the user, any compression and decompression which is used should be designed such that the amount of delay which is introduced is as small as possible.

R. Chaudhariu, C. Schuwerk, M. Danaei, and E. Steinbach, "Perceptual and Bitrate-scalable Coding of Haptic Surface Texture Signals," IEEE J. Selected Topics in Signal Processing , vol. 9, no. 3, pp. 462-473, 2015, discloses a bitrate scalable haptic texture codec, which incorporates a masking model based on weaker signals close in frequency to a powerful signal being masked.

SUMMARY

It is an object to further reduce bitrate requirements for haptic data. According to a first aspect, it is presented a method for compressing haptic data for a plurality of channels. The method is performed by an encoder and comprises the steps of: receiving respective samples, representing haptic data, in parallel from a plurality of channels; performing a lossy encoding, wherein, for each channel, compression of data is achieved at least partly by encoding a sample, which differs less from a preceding sample for the same channel than a perceptual threshold as a no change codeword, the lossy encoding yielding respective codewords for each channel; grouping the codewords in groups; wherein each group contains codewords from a plurality of channels; generating a group index indicating whether each group contains only no change codewords or at least one codeword indicating a change; and outputting the group index along with codewords only for groups where at least one codeword indicates a change.

The no change codeword may consists of one or more zeros.

The group index may comprise an indicator for each group, each indicator indicating whether the respective group contains only no change codewords or at least one intermediate codeword indicating a change.

In the step of performing a lossy encoding, the lossy encoding, for each channel, may be based on a current sample and a most recent previous sample. The perceptual threshold may be based on what change in haptic data a person is expected to be able to perceive.

Each sample may be a scalar quantisation of sensor data. According to a second aspect, it is presented an encoder for compressing haptic data for a plurality of channels. The encoder comprises: a processor; and a memory storing instructions that, when executed by the processor, causes the encoder to: receive respective samples, representing haptic data, in parallel from a plurality of channels; perform a lossy encoding, wherein, for each channel, compression of data is achieved at least partly by encoding a sample, which differs less from a preceding sample for the same channel than a perceptual threshold as a no change codeword, the lossy encoding yielding respective codewords for each channel; group the codewords in groups; wherein each group contains codewords from a plurality of channels; generate a group index indicating whether each group contains only no change codewords or at least one codeword indicating a change; and output the group index along with codewords only for groups where at least one codeword indicates a change. The no change codeword may consist of one or more zeros.

The group index may comprise an indicator for each group, each indicator indicating whether the respective group contains only no change codewords or at least one intermediate codeword indicating a change.

The instructions to perform lossy encoding may comprise instructions that, when executed by the processor, causes the encoder to perform lossy encoding, for each channel, based on a current sample and a most recent previous sample.

The perceptual threshold may be based on what change in haptic data a person is expected to be able to perceive. Each sample may be a scalar quantisation of sensor data.

According to a third aspect, it is presented an encoder comprising: means for receiving respective samples, representing haptic data, in parallel from a plurality of channels; means for performing a lossy encoding, wherein, for each channel, compression of data is achieved at least partly by encoding a sample, which differs less from a preceding sample for the same channel than a perceptual threshold as a no change codeword, the lossy encoding yielding respective codewords for each channel; means for grouping the codewords in groups; wherein each group contains codewords from a plurality of channels; means for generating a group index indicating whether each group contains only no change codewords or at least one codeword indicating a change; and outputting the group index along with codewords only for groups where at least one codeword indicates a change.

According to a fourth aspect, it is presented a computer program for compressing haptic data for a plurality of channels. The computer program comprises computer program code which, when run on an encoder causes the encoder to: receive respective samples, representing haptic data, in parallel from a plurality of channels; perform a lossy encoding, wherein, for each channel, compression of data is achieved at least partly by encoding a sample, which differs less from a preceding sample for the same channel than a perceptual threshold as a no change codeword, the lossy encoding yielding respective codewords for each channel; group the codewords in groups;

wherein each group contains codewords from a plurality of channels;

generate a group index indicating whether each group contains only no change codewords or at least one codeword indicating a change; and output the group index along with codewords only for groups where at least one codeword indicates a change.

According to a fifth aspect, it is presented a computer program product comprising a computer program according to the fourth aspect and a computer readable means on which the computer program is stored.

According to a sixth aspect, it is presented a method for decompressing haptic data for a plurality of channels. The method is performed by a decoder and comprises the steps of: receiving a group index and codewords for groups of channels of the plurality of channels, wherein codewords are only received for groups where at least one codeword indicates a change, wherein the group index indicates whether each group contains only no change codewords or at least one codeword indicating a change, wherein each codeword represents one of the plurality of channels; and wherein a no change codeword indicates a sample which differs less from a preceding sample for the same channel than a perceptual threshold; generating no change codewords for channels in any group of channels containing only no change codewords; combining the generated codewords with the received codewords, wherein the order of the codewords is based on the group index; and converting each codeword to a reconstructed sample value for a respective channel.

The no change codeword may consist of one or more zeros. The group index may comprise an indicator for each group, each indicator indicating whether the respective group contains only no change codewords or at least one intermediate codeword indicating a change.

The perceptual threshold may be based on what change in haptic data a person is expected to be able to perceive. Each reconstructed sample may be a scalar quantisation representing sensor data.

According to a seventh aspect, it is presented a decoder for decompressing haptic data for a plurality of channels. The decoder comprises: a processor; and a memory storing instructions that, when executed by the processor, causes the decoder to: receive a group index and codewords for groups of channels of the plurality of channels, wherein codewords are only received for groups where at least one codeword indicates a change, wherein the group index indicates whether each group contains only no change codewords or at least one codeword indicating a change, wherein each codeword represents one of the plurality of channels; and wherein a no change codeword indicates a sample which differs less from a preceding sample for the same channel than a perceptual threshold; generate no change codewords for channels in any group of channels containing only no change codewords; combine the generated codewords with the received codewords, wherein the order of the codewords is based on the group index; and convert each codeword to a reconstructed sample value for a respective channel.

The no change codeword may consist of one or more zeros.

The group index may comprise an indicator for each group, each indicator indicating whether the respective group contains only no change codewords or at least one intermediate codeword indicating a change.

The perceptual threshold may be based on what change in haptic data a person is expected to be able to perceive.

Each reconstructed sample may be a scalar quantisation representing sensor data.

According to an eighth aspect, it is presented a decoder comprising: means for receiving a group index and codewords for groups of channels of the plurality of channels, wherein codewords are only received for groups where at least one codeword indicates a change, wherein the group index indicates whether each group contains only no change codewords or at least one codeword indicating a change, wherein each codeword represents one of the plurality of channels; and wherein a no change codeword a sample which differs less from a preceding sample for the same channel than a perceptual threshold; means for generating no change codewords for channels in any group of channels containing only no change codewords; means for combining the generated codewords with the received codewords, wherein the order of the codewords is based on the group index; and means for converting each codeword to a reconstructed sample value for a respective channel. According to a ninth aspect, it is presented a computer program for decompressing haptic data for a plurality of channels. The computer program comprises computer program code which, when run on a decoder causes the decoder to: receive a group index and codewords for groups of channels of the plurality of channels, wherein codewords are only received for groups where at least one codeword indicates a change, wherein the group index indicates whether each group contains only no change codewords or at least one codeword indicating a change, wherein each codeword represents one of the plurality of channels; and wherein a no change codeword a sample which differs less from a preceding sample for the same channel than a perceptual threshold; generate no change codewords for channels in any group of channels containing only no change codewords; combine the generated codewords with the received codewords, wherein the order of the codewords is based on the group index; and convert each codeword to a reconstructed sample value for a respective channel.

According to a tenth aspect, it is presented a computer program product comprising a computer program according to the ninth aspect and a computer readable means on which the computer program is stored.

Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a/an/the element, apparatus, component, means, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is now described, by way of example, with reference to the accompanying drawings, in which:

Fig 1 is a schematic diagram illustrating an environment in which

embodiments presented herein can be applied;

Fig 2 is a schematic diagram illustrating details of the encoder and decoder of Fig 1 according to one embodiment;

Fig 3 is a flow chart illustrating embodiments of methods for compressing haptic data for a plurality of channels performed in an encoder; Fig 4 is a flow chart illustrating embodiments of methods for decompressing haptic data for a plurality of channels performed in a decoder;

Fig 5 is a schematic diagram illustrating components of the encoder and decoder of Fig l, according to one embodiment; Fig 6 is a schematic diagram showing functional modules of the encoder of Fig l according to one embodiment;

Fig 7 is a schematic diagram showing functional modules of the decoder of Fig l according to one embodiment; and

Fig 8 shows one example of a computer program product comprising computer readable means.

DETAILED DESCRIPTION

The invention will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Like numbers refer to like elements throughout the description. Fig 1 is a schematic diagram illustrating an environment in which

embodiments presented herein can be applied. A user 9 is here connected to interact with a remote device 6. The remote device 6 comprises a number of sensors 5. Although not shown, the remote device 6 also comprises one or more actuators. The sensors 5 generate data which is input to an encoder 1. Since there are multiple sensors 5, the data input to the encoder 1 is data for a plurality of channels, i.e. multichannel data, where each channel is associated with a separate set of one or more sensors. The encoder 1 compresses the multichannel data to encoded data 4 and outputs the data for transmission to a decoder 2. The transmission can occur using any communication technology known in the art per se, e.g. over a Local Area Network (LAN), such as Ethernet and/or using a Wide Area Network (WAN), e.g. using Internet Protocol (IP). In any case, the encoded data 4 is received by the decoder 2, which decompresses the encoded data 4 to signals which are suitable for provision to actuators 10 of a user device 8. It is to be noted that the term 'remote' in this context does not need to be far away. 'Remote' only implies that there is communication between the user device 8 and the remote device 6.

Hence, the actuators 10 of the user device 8 provide haptic sensations reflecting what is detected by the sensors 5. This allows a user 9 of the user device 9 to haptically experience a physical environment which can be located remotely, which can be in another room or in another country from the user 9·

Optionally, the equivalent communication occurs in the other direction from the master (the user device 8) to the slave (the remote device 6). In other words, there may also be an encoder 1 by the user device 8 and a decoder 2 by the remote device 6, to thereby communicate haptic data from sensors (not shown) in the user device 8 to control actuators (not shown) in the remote device 6.

While the user device 8 is here shown as a glove where the user can insert a hand in the user device 8, the user device 8 can be provided in any suitable way that allows haptic perception for the user 9. For instance, the user device 8 could be an exoskeleton, etc. The remote device 6 does in any case provide sensors in a way that makes sense in terms of the physical appearance of the user device 8.

To illustrate the requirements for such a multichannel system, one can consider a teleoperation scenario where an operator uses an exoskeleton (master device), which is capable of providing both tactile and kinesthetic feedback to his/her hands and his/her forearm and upper arms, to control a slave device (e.g. a robot).

In order to perform the teleoperation, the measured values (e.g. position, acceleration, and/force force) at the master are communicated to the control unit at the slave side, while the measured values at the slave side are communicated to the control unit at the master side. With such a setup, the operator is able to perform the precise control of the robotic arms, while he/she is also provided with tactile feedback.

This setup possesses three data channels (position, acceleration and force) per sensor measurement per location. As a possible implementation, one may consider that there exists one acquisition point for each finger, one for the wrist, and one for the forearm and one for the upper arm, resulting in a total of 13 acquisition points.

This would result in a total number of sensors of 39 (13*3 data channels (position, acceleration and force)), where each sensor has three dimensions, resulting in 117 (13*3*3) data channels. If each data channel requires 32 Kbps (kilobits per second), the total bandwidth needed can be calculated as 32 Kbps * 117 = 3.744 Mbps. Hence, it is evident that an efficient compression system is of great benefit to enable this type of haptic application with a reasonable data rate requirement.

Teleoperation systems with haptic feedback may become unstable if the closed-loop system experiences latency larger than tens of milliseconds. Additionally, research has shown than for latency above 10 ms, the

teleoperation transparency is greatly reduced, affecting both the perception quality as well as the task performance. Hence, the compression system should not introduce further algorithmic delay. This implies that block processing is not allowed, preventing conventional multichannel

compression schemes from being applied.

Fig 2 is a schematic diagram illustrating details of the encoder and decoder of Fig 1 according to one embodiment. There are here k channels, resulting in k samples Xi-xk representing haptic data. The samples Xi-xk are received in parallel from the different sensors connected to the encoder. The samples Xi- xk are here in digital form, whereby the sensor data has previously been quantized, which can occur prior to the encoder l or as part of the encoder l (not shown). The encoder ι comprises a source encoder 13 and a sparse encoder 14. The source encoder 13 encodes, using lossy encoding, the samples Xi-xk into an equivalent number of codewords Ci-Ck. Lossy encoding is to be construed as an encoding from source data to encoded data where a decoding process is unable to (at least always) regenerate the source data from the encoded data. The sparse encoder 14 then converts the codewords Ci-Ck in a lossless way to an encoded data stream 4 which is communicated to the decoder as explained above with reference to Fig 1. Lossless encoding is to be construed as an encoding from source data to encoded data where a decoding process is able to reliably regenerate the source data from the encoded data.

The decoder 2, on its side, comprises a sparse decoder 15 and a source decoder 16. The sparse decoder converts the data stream 4 to the codewords Ci-Ck, which are the same as the codewords Ci-Ck in the encoder 1 since the sparse encoder 14 employs a lossless algorithm. It is to be noted that transmission errors between the encoder 1 and decoder 2 are not considered here and are dealt with as known in the art per se. The source decoder 16 then converts the codewords Ci-Ck to reconstructed sample values yi-yk which can be provided to actuators of the user device 8 to achieve a haptic sensation for the user. The details of how the encoding and decoding is performed will be explained below with reference to Fig 3 (for the encoder) and Fig 4 (for the decoder).

Fig 3 is a flow chart illustrating embodiments of methods for compressing haptic data for a plurality of channels performed in an encoder. In a receive samples step 40, samples Xi-xk are received in parallel from a plurality of channels. The sample respectively represent haptic data. Each sample can be a scalar quantisation of sensor data.

In a perform lossy encoding step 42, a lossy encoding is performed. For each channel, compression of data is achieved at least partly by encoding a sample, which differs less from a preceding sample for the same channel than a perceptual threshold as a no change codeword. This lossy encoding yields respective codewords Ci-Ck for each channel. The perceptual threshold can be based on what change in haptic data a person is expected to be able to perceive. In other words, the perceptual threshold represents the range in which a change of signal is unnoticeable by human perception. In one embodiment, the lossy compression could be based on a Differential Pulse Code Modulation (DPCM) scheme.

The no change codeword can consist of one or more zeros. In one embodiment, the lossy encoding, for each channel, is based on a current sample and a most recent previous sample.

In a group step 44, the codewords are grouped in groups. Each group contains codewords from a plurality of channels.

In a generate group index step 46, a group index is generated, which indicates whether each group contains only no change codewords or at least one codeword indicating a change. In one embodiment, the group index comprises an indicator for each group, each indicator indicating whether the respective group contains only no change codewords or at least one

intermediate codeword indicating a change. In one embodiment, the group index consists of identifiers of those groups which contain at least one intermediate codeword indicating a change. All other groups are then groups containing only no change codewords. The complement is also possible; the group index then consists of identifiers of those groups which only contain no change codewords. In an output step 48, the group index is output along with codewords only for groups where at least one codeword indicates a change. In other words, no codewords for groups containing only no change codewords are output.

Fig 4 is a flow chart illustrating embodiments of methods for decompressing haptic data for a plurality of channels performed in a decoder.

In a receive data step 50, a group index and codewords for groups of channels of the plurality of channels are received. The codewords are only received for groups where at least one codeword indicates a change. As explained above, the group index indicates whether each group contains only no change codewords or at least one codeword indicating a change. Each codeword represents one of the plurality of channels. A no change codeword indicates a sample which differs less from a preceding sample for the same channel than a perceptual threshold. For instance, the no change codeword can consist of one or more zeros. In one embodiment, the group index comprises an indicator for each group, wherein each indicator indicates whether the respective group contains only no change codewords or at least one intermediate codeword indicating a change. In one embodiment, the group index consists of identifiers of those groups which contain at least one intermediate codeword indicating a change. All other groups are then groups containing only no change codewords. The complement is also possible; the group index then consists of identifiers of those groups which only contain no change codewords.

The perceptual threshold is based on what change in haptic data a person is expected to be able to perceive. In a generate no change codewords step 52, no change codewords are generated for channels in any group of channels containing only no change codewords. In a combine step 54, the generated codewords are combined with the received codewords, wherein the order of the codewords Ci-Ck is based on the group index.

In a convert step 56, each codeword is converted to a reconstructed sample value yi-yk for a respective channel. Each sample is a scalar quantisation representing sensor data.

Using the methods of Figs 3 and 4, the encoding and decoding can occur without the need to hold buffers on either end. In other words, the encoding and decoding can occur without introducing any significant delay, which would be the case e.g. if block processing is utilised.

An embodiment of an encoding and decoding, employed by the methods of Fig 3 and 4, will now be described using a more detailed scenario.

In step 42, the source encoder calculates a set of codewords Ci-Ck which will eventually determine the reconstructed samples yi-yk at the decoder. The codewords are designed such that reconstructed samples less than the perceptual threshold are signaled by no-change codewords, in this case by '0' bits only. As explained above, these codewords are denoted no change codewords. This holds for both variable and fixed length codewords, as illustrated in Tables 1 and 2. In this example, both cases are presented with four reconstruction levels ai, where a ₀ corresponds to reconstruction lower than the perceptual threshold.

Table 1 shows codewords which are generated using variable length codewords. More specifically, Huffman coded amplitude levels are applied here. reconstruction level of yk codeword Ck a ₀ 0

ai 10 a ₂ 110 a ₃ 111

Table 1: Variable length codeword: Huffman coded amplitude levels

Table 2 shows codewords which are generated using fixed length codewords, in this case PCM (Pulse Code Modulation).

Table levels Due to the nature of haptic data, where many sensors, as a proportion of total time, rarely change, many of the channels will be within the perceptual threshold. This is exploited by the sparse encoder in steps 44, 46 and 48 to efficiently compress the bitstream.

In step 44, the sparse encoder receives the K codewords Ci-Ck and groups the codewords. In this example, we have 24 codewords Ci-c ₂₄ at the input of the sparse encoder in step 44. The fixed length codewords of Table 2 are employed here. Each group here consists of three codewords. The bitstream after the grouping then appears as follows (commas are used here to denote group boundaries): {00 10 01, 00 00 00, 00 00 00, 00 00 00, 00 00 00, 00 00 00, 00 00 00, 11 00 00}

We thus have eight (24 codewords/3 codewords per group) groups. In step 46, the sparse encoder generates the group index containing indicators to be able to work out, for each group, whether it contains only no change codewords or not. In this example, only the first and last groups contain at least one codewords indicating a change. The indicators are here calculated by performing a logical OR operation of the bits in each group, which results in a group index (containing the indicators) of {10000001}.

In step 48, the sparse encoder outputs the group index and only the codewords for groups where at least one codeword indicates a change, in this case the codewords for the first group and the last group. Hence, the bitstream that is output from the encoder for transmission to the decoder takes the form of: {10000001, 00 10 01, 11 00 00}.

Since the bitstream of codewords Ci-c ₂₄ consists of 48 bits (24 codewords x 2 bits each), while the bitstream sent to the decoder consists of 20 bits

(8+6+6), in this example we achieve compression ratio of 42 %. Moving now to the decoder, in step 50 the decoder receives the group index and the subset of codewords, i.e. {10000001, 00 10 01, 11 00 00}.

In step 52, the decoder deduces from the group index that no change codewords need to be generated for the six groups in the middle and that the received codewords should form part of the first and the last groups. In step 54, the decoder combines the generated six groups in the middle with the received codewords for the first and the last groups.

In this way, the decoder is able to regenerate the original set of codewords: {00 10 01, 00 00 00, 00 00 00, 00 00 00, 00 00 00, 00 00 00, 00 00 00, 11 00 00}. Fig 5 is a schematic diagram illustrating components of the encoder 1 and decoder 2 of Fig 1, according to one embodiment. Each of the encoder 1 and the decoder 2 comprises the components described here. A processor 60 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), application specific integrated circuit etc., capable of executing software instructions 67 stored in a memory 64, which can thus be a computer program product. The processor 60 can be configured to execute the method described with reference to Fig 3 and 4 above for the encoder 1 and the decoder 2, respectively.

The memory 64 can be any combination of read and write memory (RAM) and read only memory (ROM). The memory 64 also comprises persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory.

A data memory 66 is also provided for reading and/or storing data during execution of software instructions in the processor 60. The data memory 66 can be any combination of read and write memory (RAM) and read only memory (ROM). The encoder i/decoder 2 further comprises an I/O interface 62 for

communicating with other external entities. Optionally, the I/O interface 62 also includes a user interface.

Fig 6 is a schematic diagram showing functional modules of the encoder 1 of Fig 1 according to one embodiment. The modules are implemented using software instructions such as a computer program executing in the encoder 1. Alternatively or additionally, the modules are implemented using hardware, such as any one or more of an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or discrete logical circuits. The modules correspond to the steps in the methods illustrated in Fig 3. A receiver 70 corresponds to step 40. A lossy encoder 72 corresponds to step 42. A grouper 74 corresponds to step 44. A generator 76 corresponds to step 46. An outputter 78 corresponds to step 48.

Fig 7 is a schematic diagram showing functional modules of the decoder 2 of Fig 1 according to one embodiment. The modules are implemented using l8 software instructions such as a computer program executing in the decoder 2. Alternatively or additionally, the modules are implemented using hardware, such as any one or more of an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or discrete logical circuits. The modules correspond to the steps in the methods illustrated in Fig 4.

A receiver 80 corresponds to step 50. A generator 82 corresponds to step 52. A combiner 84 corresponds to step 54. A converter 86 corresponds to step 56.

Fig 8 shows one example of a computer program product comprising computer readable means. On this computer readable means a computer program 91 can be stored, which computer program can cause a processor to execute a method according to embodiments described herein. In this example, the computer program product is an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. As explained above, the computer program product could also be embodied in a memory of a device, such as the computer program product 64 of Fig 5. While the computer program 91 is here schematically shown as a track on the depicted optical disk, the computer program can be stored in any way which is suitable for the computer program product, such as a removable solid state memory, e.g. a Universal Serial Bus (USB) drive.

The invention has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims.