DEVICE AND METHOD FOR PROVIDING AN APPLICATION USER INTERFACE WITH HAPTIC FEEDBACK

TECH N ICAL FI E LD

[0001] The present disclosure relates to a device and method for providing an application user interface with haptic feedback for a passenger in an automotive setting.

BACKG ROU N D

[0002] CN213092557U discloses a sense of touch that can speak turns over voiced book, including the book body, the internal page or leaf in a plurality of books that is provided with of book, all printed the characters pattern on the page or leaf in every book, the book body is including left book body and right book body, left side book body and right book body connect and be the books form, and left book body can be followed left book body and right book body junction and opened and closed, right side book body is equipped with sound generating mechanism by right side edge, sound generating mechanism includes the sound generating mechanism shell and sets up loudspeaker and the PCB board in the sound generating mechanism shell, loudspeaker and PCB board electric connection, be provided with volume on the sound generating mechanism shell and keep off the position switch, be provided with book page or leaf switch in this volume below that keeps off the position switch, page or leaf switch and volume keep off the position switch and all are connected with the PCB board in the book. The utility model discloses simple structure, space utilization is high, and is small, conveniently carries, and page or leaf and sound generating mechanism separately set up in the book simultaneously, cannot have the condition emergence of sound production confusion or non-sound production, and user experience feels good.

[0003] KR20160010843A discloses a method for playing an audio book provided with a vibration function, comprising the steps of: loading sound source data to play the audio book and obtaining total play time of the sound source data loaded; playing the sound source data; loading one or more from vibration data and illustration data corresponding to the sound source data; obtaining one or more from the vibration information and the illustration information corresponding to the play time of the sound source data from one or more from the vibration data and the illustration data; and outputting vibration or illustration based on one or more from the vibration data or the illustration data obtained.

[0004] US2016240102A1 discloses a system for enhanced learning that combines tactile surfaces and audio recordings, allowing the user to explore a 3-dimensional object through touch while also hearing pre-recorded audio explaining regions or features of interest. One of a collection of 3-dimensional tactile modules is used with the system at a given time. When a tactile module is inserted into the system (or otherwise connected) by the user, the system automatically recognizes the inserted module, locates the set of audio recordings and region maps associated with that tactile module, and then plays module-specific audio recordings.

[0005] Document EP2972687A1 discloses a method of receiving user input includes operating a software program in a first operating mode and modifying a graphical interface of the software program in response to receiving gesture. The method also includes receiving a haptic input and operating the software program in a second operating mode in response to the haptic input. In the second operating mode, the gesture produces a different modification to the graphical interface than in the first operating mode.

[0006] These facts are disclosed in order to illustrate the technical problem addressed by the present disclosure.

GENERAL DESCRIPTION

[0007] The present disclosure relates to a device and method for providing an application user interface with haptic feedback for a passenger in an automotive setting. [0008] One of the main ways in which children explore the world around them is through touch. When human physicality is used, especially the sense of touch, the brain receives a higher amount of stimulation. Thus, by physically interacting with "things", children acquire learning experiences that provide important contributions for their development, with a positive impact on their comprehension of the world around them (Shibasaki et al., 2020). Due to this, many museums (e.g., Boston Children's Museum, n.d.; Please Touch Museum, n.d.) have created hands-on exhibition activities, aimed at children, with the goal of creating a greater learning environment regarding the content of their exhibitions. Due to the role of touch on a child's development, research on how haptic feedback sensations can be used to help improve children's education has been performed.

[0009] Studies, such as Yannier et al. (2015), Zhao et al. (2015), and (Shibasaki et al., 2020), indicate that the introduction of haptic feedback sensations on children's reading and listening activities has a positive impact on several aspects, such as the children's ability to comprehend and recall parts of the story, while also enhancing the children's enjoyment of these activities, although these benefits might only be truly effective for children above 4 years old. Other activities, such as drawing and painting, also contribute to a child's ability to recollect events and information, although these effects also seem to be more effective with children above the age of 4 years (Butler et al., 1995). By introducing haptic feedback modalities into drawing activities, richer, more immersive, and more entertaining experiences can be created and presented to children, which can have positive impacts on their development. All these activities, among others, that are enhanced by the presence of haptic feedback sensations, can be either partially or fully implemented as applications (apps) in modern entertainment systems, such as tablets and computers. With current display and surface technologies, devices can be made so that their touchscreen surfaces have the ability to provide users with different tactile feedback sensations beyond those that are normally felt when touching the surface, such as vibrotactile (KHOSHKAVA et al., 2019) and frictional (Olley et al., 2020) stimuli. [0010] Although many edutainment apps, aimed at children, are available on several app markets, the haptic feedback they provide, if any, is strictly vibrotactile, thus existing a need for fully generating the benefits to children's development that arise from the stimulation of the sense of touch.

[0011] In an embodiment, taking into consideration the technological advancements that have recently occurred in the field of surface haptics, it is currently possible to create and develop edutainment apps, aimed at children, that make use of the haptic feedback capabilities with which touchscreen displays and surfaces can now be enhanced with, to provide children with fun, engaging, and educational activities, enhanced by the sensations that haptic feedback can provide. By introducing touchscreen devices and surfaces with haptic feedback capabilities on the interior of vehicles, children can enjoy the activities provided by these apps while riding with adults on their commutes. In the near future, it is expected that level 5 autonomous vehicles will be widely adopted by general consumers. Thanks to these vehicles, adults will have to dedicate less time and attention to the task of driving the vehicle and can instead spend said time and attention bonding with their children, joining in on these edutainment activities, be it as simple play partners, or as tutors, roles that are both helpful and appreciated by the children.

[0012] An aspect of the present disclosure relates to making use of a smart surface, composed of an insulative layer, fitted atop a transparent and conductive electrode sheet layer, which is in turn fitted atop a touch sensitive layer. This touch-sensitive layer is fitted atop a layer of LED lights, which are in turn fitted atop a layer of vibrotactile actuators. The layer of LED lights is optional, and when not presented, the touch-sensitive layer is fitted atop the layer of vibrotactile actuators. This smart surface can be employed either as: a) the interactive surface interface of a display device to which it is physically connect (e.g., the smart surface is part of the device's display); or b) as the interactive surface interface of a display device to which it is not physically connected (e.g., touch surface interface communicating wirelessly with a display device in close proximity). The capacitive-based touch sensitive layer can be made of a variety of materials, depending on the intended use case and aesthetic appeal (e.g., transparent glass - display device; translucent plastic - interactive surface interface for an external device with LED lights underneath it). This smart surface can generate both frictional and vibrotactile haptic sensations. Frictional haptic sensations are generated thanks to the insulative and conductive layers, which allow for friction between the smart surface and the user's skin to be modulated through electrostatic actuation, while the skin is in motion. Vibrotactile haptic sensations, in turn, are generated thanks to the vibrotactile actuators, which can generate vibrations strong enough to be felt through the user's skin, either stationary or in motion, when in direct contact with the smart surface. When the smart surface is used as the interactive interface for an external device, haptic feedback sensations, caused by the external device's software, can be generated on it, as long as the haptic actuation technology that is present on the smart surface is both capable of interpreting the software's information and of generating such sensations.

[0013] Since the present disclosure is mainly aimed at being used by children while they are passengers in a vehicle, and keeping in mind that several health and safety organizations, as well as several legislations worldwide, recommend keeping child passengers on the back seats as much as possible until they are at least 12 years old. Therefore, for the embodiments being discussed in this document, this smart surface is to be fitted on at least one region inside the vehicle: the display surface of an entertainment display on the back of vehicle, where it will serve as said device's interactive surface interface. This device can be attacked to an individual adjustable gooseneck holder, which can be extended and manipulated so that it is placed at a comfortable location for the user to interact with. Other entertainment displays inside the vehicle can also be fitted with said smart surface, such as other entertainment display devices available for rear passengers, but the use of the disclosure should be preferably limited according to the above-mentioned restrictions regarding child passengers.

[0014] It is disclosed a device for providing an application user interface with haptic feedback for a passenger in an automotive setting, comprising: a haptic surface comprising actuators for exhibiting tactile textures, comprising: a touch sensitive layer, a frictional haptic layer comprising an insulative sheet for user touch and a transparent conductive electrode film for providing frictional haptic sensations, and a vibrotactile haptic layer for providing vibrotactile haptic sensations; a display for displaying images of objects of said application user surface; an electronic data processor configured to: drive the haptic layers to exhibit a tactile texture when the user touches the image of an object in said display; to provide haptic feedback as the user touches the device.

[0015] In an embodiment, the touch sensitive layer is a capacitive-based touch sensitive layer.

[0016] In an embodiment, the electronic data processor is further configured to drive the haptic layers to exhibit a tactile texture whose roughness is defined dependent of a colour of the touched object in said display.

[0017] In an embodiment, the electronic data processor is further configured to drive the haptic layers to exhibit a tactile texture whose roughness is defined dependent of the touched object being within or outside a predetermined boundary in said display.

[0018] In an embodiment, the electronic data processor is further configured to drive the haptic layers to exhibit a tactile texture whose roughness is defined dependent of the touched object being along or outside a predetermined path in said display.

[0019] In an embodiment, the electronic data processor is further configured to play an audio file or an audio streaming when the object is touched.

[0020] In an embodiment, the touch sensitive layer is made of translucent plastic comprising LEDs underneath it.

[0021] In an embodiment, the automotive setting is an autonomous vehicle and the display is an entertainment display arranged to be mounted on the back of the vehicle. [0022] In an embodiment comprises an adjustable gooseneck holder or a headrest tablet holder for supporting the device.

[0023] It is also disclosed a method of operating a device according to any of the embodiments, comprising using said electronic data processor for: driving haptic layers to exhibit a tactile texture to provide haptic feedback as the user touches the touchpad device interface.

BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The following figures provide preferred embodiments for illustrating the disclosure and should not be seen as limiting the scope of invention.

[0025] Figure 1 illustrates a schematic illustration of a child user on back seat using app on haptic display device according to an embodiment.

[0026] Figure 2 illustrates a schematic illustration of an application home screen (different waveform indicate different texture sensations) according to an embodiment.

[0027] Figure 3 illustrates a schematic illustration of a drawing with different colours (different waveform indicate different texture sensations) according to an embodiment.

[0028] Figure 4 illustrates a schematic illustration of a drawing inside and outside the lines (different waveform indicate different texture sensations) according to an embodiment.

[0029] Figure 5 illustrates a schematic illustration of a copying a item to learn how to write it (different waveform indicate different texture sensations) according to an embodiment.

[0030] Figure 6 shows a schematic illustration of an animal picture dictionary, exemplary page (different waveform indicate different texture sensations) according to an embodiment. [0031] Figure 7 shows a schematic illustration of a haptic sensations which then triggers from and up to certain "trigger" words (different waveform indicate different texture sensations) according to an embodiment.

DETAILED DESCRIPTION

[0032] The present disclosure relates to a device for providing an application user interface with haptic feedback for a passenger in an automotive setting, comprising: a haptic surface comprising actuators for exhibiting tactile textures, comprising: a touch sensitive layer, a frictional haptic layer comprising an insulative sheet for user touch and a transparent conductive electrode film for providing frictional haptic sensations, and a vibrotactile haptic layer for providing vibrotactile haptic sensations; a display for displaying images of objects of said application user surface; an electronic data processor configured to: drive the haptic layers to exhibit a tactile texture when the user touches the image of an object in said display; to provide haptic feedback as the user touches the device.

[0033] Figure 1 shows a schematic representation of an embodiment of a child user on back seat using app on haptic display device 1 according to an embodiment, for providing a user interface 2.

[0034] In an embodiment, the haptic feedback children edutainment app, on which this document focuses on, can be installed and accessed through several different devices, such as the vehicle's infotainment system or the user's personal smartphone, as long as the software permits it. However, the full benefits of the haptic feedback sensations described below can only be felt when the interaction between user and app is done through a device or surface which possesses actuators capable of generating the intended sensations. Therefore, the interaction between user and app is more fruitful when done through a smart surface as , especially when said interaction occurs inside the vehicle, during the user's commute. The haptic feedback sensations described in this section make use of frictional and/vibrotactile stimuli, which can be generated through electrostatic and vibrotactile actuation, respectively. Although the main innovation of this app is the employment of haptic feedback sensations during the interactions between it and the user, there are no impediments regarding installing and using this app on devices without actuators that can produce the stimuli types from which this app benefits, as long as users keep in mind that the full intended experience cannot be achieved in said devices.

[0035] Regarding activity content of the app itself, these will be described hereafter. These descriptions will focus on use cases of child passengers sitting on the back seats of a vehicle, with access to an individual haptic feedback touch display device, while alongside and not alongside an adult. This application has 5 main activities, accessible from its main menu, Figure 2:

- A free drawing mode in which users are presented with an empty, unrestricted canvas, where they can freely draw and paint with their fingers, using a variety of colours, with each colour giving their own distinct haptic feedback sensation (e.g., haptic feedback ranging from rougher to smoother, depending on the colour's RGB values) (Figure 3). At any time during or after finishing the drawing/painting, the brush tool can be disengaged so that users can freely explore their creations through touch controls and by sliding their fingers across the display surface. At any time during, or after finishing the drawing/painting, the image can be saved in the device's memory to be loaded later or exported through a USB or wireless connection to another device (if available), as not to restrict the user's creation to only being available on the vehicle's displays. This mode is mainly aimed at users around or above 4 years old;

- A figure drawing and painting mode, in which the outline of different items (e.g., animals, objects, fruits) will be displayed, with the user being tasked with painting inside the outlines. In this mode, two extra modes are available, one, "Original", in which the correct colours should be used for the corresponding portion, and another, "Freestyle", in which the user is free to use whatever colour they wish. As in the free drawing mode, each colour will give a corresponding haptic feedback sensation. When painting outside the outline, a rougher haptic sensation is felt to indicate to the user that they are painting out of bounds (Figure 4). As in the free drawing mode, at any time during, or after finishing the drawing/painting, the brush tool can be disengaged so that users can freely explore their creations through touching or sliding their fingers. At any time during, or after finishing the drawing/painting, the image can be saved in the device's memory to be loaded later, or exported through a USB or wireless connection to another device, to not restrict the user's creation to only the in-vehicle display. This mode is mainly aimed at users around or above 4 years old;

- A follow-along, in which "flashcards" containing the visualization of different elements (e.g., roman numerals, Arabic numerals, Latin alphabet letters, Hiragana, Kanjis) are displayed on either the right or the left side of the screen, one at a time, alongside an audio file containing the correct vocalization of said element, which is played when the item is loaded in. The user's task, in this mode, is to draw the visualization of the element present on the screen on the other portion of the screen not displaying the element (which side is the visualization and the drawing portion of the screen can be adjusted in settings by the parent or supervising adult, depending on if the user is left-handed, righthanded, or ambidextrous). While following along correctly, a smoother haptic sensation is felt. When following along incorrectly, a rougher haptic sensation is felt instead. The user can also slide their finger on the visualization portion of the screen, feeling a smooth texture sensation when sliding on top of the line of the element, and a rougher texture sensation when sliding outside the lines of the element (Figure 5). This mode is aimed more towards educational purposes. This mode is mainly aimed at users above 4 years old, with different settings available depending on the user's age (e.g., 4-year-olds - all Latin alphabet letters are displayed in their uppercase form, by themselves, and Arabic numbers from 1 to 10 are presented; 6-year-olds - Latin alphabet letters can be displaying in uppercase and lowercase form, alongside other letters to form words, and Arabic numbers from 1 to 100 are presented). The type of elements that will be displayed, as well as the age settings, are controlled by the parent or supervising adult;

- A Picture Dictionary mode (e.g., Animal Picture Dictionary, Art Gallery Picture Dictionary) which presents individual items, alongside pictures and a written description of said item. A "speaker" button at the side of the description can be clicked to play an audio file containing a verbalized version of the description. When sliding their finger across the picture, users can feel the sensation of the different textures present in it (e.g., Animal Picture Dictionary - sliding finger provides sensation of specific animals' skin texture). Whenever possible, 3D models of the subject item will also be available to be explored on a "3D model" viewer (Figure 6). In this viewer, users can explore the item through normal touch controls, such as rotating two fingers to rotate, or pinching out to zoom in, but they can also slide their fingers on top of the object to feel its textures. This mode is aimed towards both educational and entertainment purposes. This mode is mainly aimed at users above 4 years old;

- Haptic story time mode. This mode has two submodes, one in which the user is meant to read a story by themselves (Submode A), and another in which a story is to be read to the user, either by another person at their side (e.g., parent) or by the device's text-to-speech function, with audio being transmitted from the device's speakers (Submode B). At certain, specific parts of each story, the user will receive different haptic feedback sensations, prompted by the story (e.g., "an earthquake could be felt" is accompanied by vibrotactile stimuli representing the phenomenon). A pre-selection of stories is available on the device, with content aimed at different ages. This mode is aimed towards both educational and entertainment purposes. This mode is mainly aimed at users above 5 years old;

Submode A - User reading the story themselves: user slides finger below and along the sentence they are reading. When the finger hits a certain "trigger" word (e.g., "the ground shook"; "its hair was like silk") the corresponding haptic feedback sensation is generated (e.g., the device vibrates; texture sensation of silky hair is generated until another "trigger' word, indicating the end of this feedback, is reached) (Figure 7);

Submode B - User being read the story aloud by another person or the device's text-to-speech function: as with when reading the story themselves, the user should slide their finger below and alongside the sentence that is being read. If another person is reading the story aloud to the user, they should try to either match the user's pace, or lead the user's finger as they are reading. If text-to-speech is being used to read the story aloud, the user's finger sliding speed dictates the pace of the audio, with each word only being read when the finger is slid under it.

[0036] The term "comprising" whenever used in this document is intended to indicate the presence of stated features, integers, steps, components, but not to preclude the presence or addition of one or more other features, integers, steps, components, or groups thereof.

[0037] The disclosure should not be seen in any way restricted to the embodiments described and a person with ordinary skill in the art will foresee many possibilities to modifications thereof. The above-described embodiments are combinable. The following claims further set out particular embodiments of the disclosure.

[0038] References

1. Boston Children's Museum, (n.d.). Boston Children's Museum Home Page. Retrieved January 12, 2021.

2. Butler, S., Gross, J., & Hayne, H. (1995). The effect of drawing on memory performance in young children. Developmental Psychology, 31(4), 597-608.

3. KHOSHKAVA, V., CRUZ-HERNANDEZ, J. M., & SHAH, K. (2019). HAPTIC ACTUATOR ASSEMBLY WITH A SPRING PRE-LOAD DEVICE (Patent No. EP3582074A1).

4. Olley, M. F. D., Peshkin, M. A., & Colgate, J. E. (2020). Electronic controller haptic display with simultaneous sensing and actuation (Patent No. US10768749B2). Please Touch Museum, (n.d.). Please Touch Museum Home Page. Retrieved January 12, 2021, from https://www.pleasetouchmuseum.org Shibasaki, M., Kamiyama, Y., Czech, E., Obata, K., Wakamoto, Y., Kishi, K., Hasegawa, T., Tsuchiya, S., Matsuda, S., & Minamizawa, K. (2020). Interest Arousal by Haptic Feedback During a Storytelling for Kindergarten Children. In I. Nisky, J. Hartcher-O'Brien, M. Wiertlewski, & J. Smeets (Eds.), Haptics: Science, Technology, Applications (pp. 518-526). Springer International Publishing. Yannier, N., Israr, A., Lehman, J. F., & Klatzky, R. L. (2015). FeelSleeve: Haptic Feedback to Enhance Early Reading. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 2015- April, 1015- 1024. Zhao, S., Lehman, J., Israr, A., & Klatzky, R. (2015). Using haptic inputs to enrich story listening for young children. Proceedings of the 14th International Conference on Interaction Design and Children, 239-242.