ADAPTATION TECHNIQUES

Cross-Reference to Related Application

[0001] This application claims priority from U.S. Provisional Serial No. 62/057,990, filed on September 30, 2014, entitled SPEAKER IDENTIFICATION AND

UNSUPERVISED SPEAKER ADAPTATION TECHNIQUES, and U.S. Non-Provisional Serial No. 14/835,169, filed on August 25, 2015, entitled SPEAKER IDENTIFICATION AND UNSUPERVISED SPEAKER ADAPTATION TECHNIQUES, which are hereby incorporated by reference in their entirety for all purposes.

Field

[0002] This relates generally to natural language processing and, more specifically, to performing speaker identification for natural language processing.

Background

[0003] Intelligent automated assistants (or virtual assistants) provide an intuitive interface between users and electronic devices. These assistants can allow users to interact with devices or systems using natural language in spoken and/or text forms. For example, a user can access the services of an electronic device by providing a spoken user input in natural language form to a virtual assistant associated with the electronic device. The virtual assistant can perform natural language processing on the spoken user input to infer the user's intent and operationalize the user's intent into tasks. The tasks can then be performed by executing one or more functions of the electronic device, and a relevant output can be returned to the user in natural language form.

[0004] Some natural language processing systems can perform speaker identification to verify the identity of a user. These systems typically require the user to perform an enrollment process during which the user speaks a series of predetermined words or phrases to allow the natural language processing system to model the user's voice. While this process can be used to effectively model the user' s voice, it can produce unreliable results if the user speaks in an unnatural manner during the enrollment process and/or if the user performs the enrollment process in an acoustic environment that is different than those in which the speaker identification is later performed. Thus, improved processes for modeling a user's voice are desired.

Summary

[0005] Systems and processes for generating a speaker profile for use in performing speaker identification for a virtual assistant are provided. One example process can include receiving an audio input including user speech and determining whether a speaker of the user speech is a predetermined user based on a speaker profile for the predetermined user. In response to determining that the speaker of the user speech is the predetermined user, the user speech can be added to the speaker profile and operation of the virtual assistant can be triggered. In response to determining that the speaker of the user speech is not the predetermined user, the user speech can be added to an alternate speaker profile and operation of the virtual assistant may not be triggered. In some examples, contextual information can be used to verify results produced by the speaker identification process.

Brief Description of the Drawings

[0006] FIG. 1 illustrates an exemplary environment in which a virtual assistant can operate according to various examples.

[0007] FIG. 2 illustrates an exemplary user device according to various examples.

[0008] FIG. 3 illustrates an exemplary process for operating a virtual assistant according to various examples.

[0009] FIG. 4 illustrates an exemplary process for operating a virtual assistant in a speaker profile generating mode according to various examples.

[0010] FIG. 5 illustrates an exemplary process for operating a virtual assistant in a speaker profile modifying mode according to various examples.

[0011] FIG. 6 illustrates an exemplary process for operating a virtual assistant in a static speaker profile mode according to various examples.

[0012] FIG. 7 illustrates an exemplary process for performing speaker identification according to various examples.

[0013] FIG. 8 illustrates a functional block diagram of an electronic device configured to generate a speaker profile based on a spoken user input according to various examples. Detailed Description

[0014] In the following description of examples, reference is made to the accompanying drawings in which it is shown by way of illustration specific examples that can be practiced. It is to be understood that other examples can be used and structural changes can be made without departing from the scope of the various examples.

[0015] This relates to generating a speaker profile for use in performing speaker identification for a virtual assistant. One example process can include receiving an audio input including user speech and determining whether a speaker of the user speech is a predetermined user based on a speaker profile for the predetermined user. In response to determining that the speaker of the user speech is the predetermined user, the user speech can be added to the speaker profile and operation of the virtual assistant can be triggered. In response to determining that the speaker of the user speech is not the predetermined user, the user speech can be added to an alternate speaker profile and operation of the virtual assistant may not be triggered. In some examples, contextual information can be used to verify results produced by the speaker identification process.

SYSTEM OVERVIEW

[0016] FIG. 1 illustrates exemplary system 100 for implementing a virtual assistant according to various examples. The terms "virtual assistant," "digital assistant," "intelligent automated assistant," or "automatic digital assistant" can refer to any information processing system that interprets natural language input in spoken and/or textual form to infer user intent, and performs actions based on the inferred user intent. For example, to act on an inferred user intent, the system can perform one or more of the following: identifying a task flow with steps and parameters designed to accomplish the inferred user intent; inputting specific requirements from the inferred user intent into the task flow; executing the task flow by invoking programs, methods, services, APIs, or the like; and generating output responses to the user in an audible (e.g., speech) and/or visual form.

[0017] A virtual assistant can be capable of accepting a user request at least partially in the form of a natural language command, request, statement, narrative, and/or inquiry.

Typically, the user request seeks either an informational answer or performance of a task by the virtual assistant. A satisfactory response to the user request can include provision of the requested informational answer, performance of the requested task, or a combination of the two. For example, a user can ask the virtual assistant a question, such as "Where am I right now?" Based on the user's current location, the virtual assistant can answer, "You are in Central Park." The user can also request the performance of a task, for example, "Please remind me to call Mom at 4 p.m. today." In response, the virtual assistant can acknowledge the request and then create an appropriate reminder item in the user's electronic schedule. During the performance of a requested task, the virtual assistant can sometimes interact with the user in a continuous dialogue involving multiple exchanges of information over an extended period of time. There are numerous other ways of interacting with a virtual assistant to request information or performance of various tasks. In addition to providing verbal responses and taking programmed actions, the virtual assistant can also provide responses in other visual or audio forms (e.g., as text, alerts, music, videos, animations, etc.).

[0018] An example of a virtual assistant is described in Applicants' U.S. Utility

Application Serial No. 12/987,982 for "Intelligent Automated Assistant," filed January 10, 2011, the entire disclosure of which is incorporated herein by reference.

[0019] As shown in FIG. 1, in some examples, a virtual assistant can be implemented according to a client-server model. The virtual assistant can include a client-side portion executed on a user device 102, and a server- side portion executed on a server system 110. User device 102 can include any electronic device, such as a mobile phone, tablet computer, portable media player, desktop computer, laptop computer, PDA, television, television set- top box, wearable electronic device, or the like, and can communicate with server system 110 through one or more networks 108, which can include the Internet, an intranet, or any other wired or wireless public or private network. The client- side portion executed on user device 102 can provide client-side functionalities, such as user-facing input and output processing and communications with server system 110. Server system 110 can provide server-side functionalities for any number of clients residing on a respective user device 102.

[0020] Server system 110 can include one or more virtual assistant servers 114 that can include a client-facing 170 interface 122, one or more processing modules 118, data and model storage 120, and an I/O interface to external services 116. The client-facing I/O interface 122 can facilitate the client-facing input and output processing for virtual assistant server 114. The one or more processing modules 118 can utilize data and model storage 120 to determine the user' s intent based on natural language input, and perform task execution based on inferred user intent. In some examples, virtual assistant server 114 can

communicate with external services 124, such as telephony services, calendar services, information services, messaging services, navigation services, and the like, through network(s) 108 for task completion or information acquisition. The I/O interface to external services 116 can facilitate such communications.

[0021] Server system 110 can be implemented on one or more standalone data processing devices or a distributed network of computers. In some examples, server system 110 can employ various virtual devices and/or services of third party service providers (e.g., third-party cloud service providers) to provide the underlying computing resources and/or infrastructure resources of server system 110.

[0022] Although the functionality of the virtual assistant is shown in FIG. 1 as including both a client-side portion and a server-side portion, in some examples, the functions of the assistant can be implemented as a standalone application installed on a user device. In addition, the division of functionalities between the client and server portions of the virtual assistant can vary in different examples. For instance, in some examples, the client executed on user device 102 can be a thin-client that provides only user-facing input and output processing functions, and delegates all other functionalities of the virtual assistant to a backend server.

USER DEVICE

[0023] FIG. 2 is a block diagram of a user-device 102 according to various examples. As shown, user device 102 can include a memory interface 202, one or more processors 204, and a peripherals interface 206. The various components in user device 102 can be coupled together by one or more communication buses or signal lines. User device 102 can further include various sensors, subsystems, and peripheral devices that are coupled to the peripherals interface 206. The sensors, subsystems, and peripheral devices gather information and/or facilitate various functionalities of user device 102.

[0024] For example, user device 102 can include a motion sensor 210, a light sensor 212, and a proximity sensor 214 coupled to peripherals interface 206 to facilitate orientation, light, and proximity sensing functions. One or more other sensors 216, such as a positioning system (e.g., a GPS receiver), a temperature sensor, a biometric sensor, a gyroscope, a compass, an accelerometer, and the like, are also connected to peripherals interface 206, to facilitate related functionalities.

[0025] In some examples, a camera subsystem 220 and an optical sensor 222 can be utilized to facilitate camera functions, such as taking photographs and recording video clips. Communication functions can be facilitated through one or more wired and/or wireless communication subsystems 224, which can include various communication ports, radio frequency receivers and transmitters, and/or optical (e.g., infrared) receivers and transmitters. An audio subsystem 226 can be coupled to speakers 228 and a microphone 230 to facilitate voice-enabled functions, such as voice recognition, voice replication, digital recording, and telephony functions.

[0026] In some examples, user device 102 can further include an I/O subsystem 240 coupled to peripherals interface 206. I/O subsystem 240 can include a touch screen controller 242 and/or other input controller(s) 244. Touch-screen controller 242 can be coupled to a touch screen 246. Touch screen 246 and the touch screen controller 242 can, for example, detect contact and movement or break thereof using any of a plurality of touch sensitivity technologies, such as capacitive, resistive, infrared, and surface acoustic wave technologies, proximity sensor arrays, and the like. Other input controller(s) 244 can be coupled to other input/control devices 248, such as one or more buttons, rocker switches, a thumb- wheel, an infrared port, a USB port, and/or a pointer device such as a stylus.

[0027] In some examples, user device 102 can further include a memory interface 202 coupled to memory 250. Memory 250 can include any electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, a portable computer diskette (magnetic), a random access memory (RAM) (magnetic), a read-only memory (ROM) (magnetic), an erasable programmable read-only memory (EPROM) (magnetic), a portable optical disc such as CD, CD-R, CD-RW, DVD, DVD-R, or DVD-RW, or flash memory such as compact flash cards, secured digital cards, USB memory devices, memory sticks, and the like. In some examples, a non-transitory computer-readable storage medium of memory 250 can be used to store instructions (e.g., for performing some or all of process 300, 400, 500, 600, or 700, described below) for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device, and execute the instructions. In other examples, the instructions (e.g., for performing process 300, 400, 500, 600, or 700, described below) can be stored on a non-transitory computer-readable storage medium of server system 110, or can be divided between the non-transitory computer-readable storage medium of memory 250 and the non-transitory computer-readable storage medium of server system 110. In the context of this document, a "non-transitory computer readable storage medium" can be any medium that can contain or store the program for use by or in connection with the instruction execution system, apparatus, or device.

[0028] In some examples, the memory 250 can store an operating system 252, a communication module 254, a graphical user interface module 256, a sensor processing module 258, a phone module 260, and applications 262. Operating system 252 can include instructions for handling basic system services and for performing hardware dependent tasks. Communication module 254 can facilitate communicating with one or more additional devices, one or more computers, and/or one or more servers. Graphical user interface module 256 can facilitate graphic user interface processing. Sensor processing module 258 can facilitate sensor related processing and functions. Phone module 260 can facilitate phone- related processes and functions. Application module 262 can facilitate various functionalities of user applications, such as electronic-messaging, web browsing, media processing, navigation, imaging, and/or other processes and functions.

[0029] As described herein, memory 250 can also store client-side virtual assistant instructions (e.g., in a virtual assistant client module 264) and various user data 266 (e.g., user-specific vocabulary data, preference data, and/or other data, such as the user's electronic address book, to-do lists, shopping lists, etc.) to provide the client-side functionalities of the virtual assistant.

[0030] In various examples, virtual assistant client module 264 can be capable of accepting voice input (e.g., speech input), text input, touch input, and/or gestural input through various user interfaces (e.g., I/O subsystem 240, audio subsystem 226, or the like) of user device 102. Virtual assistant client module 264 can also be capable of providing output in audio (e.g., speech output), visual, and/or tactile forms. For example, output can be provided as voice, sound, alerts, text messages, menus, graphics, videos, animations, vibrations, and/or combinations of two or more of the above. During operation, virtual assistant client module 264 can communicate with the virtual assistant server using communication subsystem 224.

[0031] In some examples, virtual assistant client module 264 can utilize the various sensors, subsystems, and peripheral devices to gather additional information from the surrounding environment of user device 102 to establish a context associated with a user, the current user interaction, and/or the current user input. In some examples, virtual assistant client module 264 can provide the contextual information or a subset thereof with the user input to the virtual assistant server to help infer the user's intent. The virtual assistant can also use the contextual information to determine how to prepare and deliver outputs to the user.

[0032] In some examples, the contextual information that accompanies the user input can include sensor information, such as lighting, ambient noise, ambient temperature, images or videos of the surrounding environment, distance to another object, and the like. The contextual information can further include information associated with the physical state of user device 102 (e.g., device orientation, device location, device temperature, power level, speed, acceleration, motion patterns, cellular signal strength, etc.) or the software state of user device 102 (e.g., running processes, installed programs, past and present network activities, background services, error logs, resources usage, etc.). Any of these types of contextual information can be provided to the virtual assistant server 114 as contextual information associated with a user input.

[0033] In some examples, virtual assistant client module 264 can selectively provide information (e.g., user data 266) stored on user device 102 in response to requests from the virtual assistant server 114. Virtual assistant client module 264 can also elicit additional input from the user via a natural language dialogue or other user interfaces upon request by virtual assistant server 114. Virtual assistant client module 264 can pass the additional input to virtual assistant server 114 to help virtual assistant server 114 in intent inference and/or fulfillment of the user's intent expressed in the user request.

[0034] In various examples, memory 250 can include additional instructions or fewer instructions. Furthermore, various functions of user device 102 can be implemented in hardware and/or in firmware, including in one or more signal processing and/or application specific integrated circuits.

SPEAKER IDENTIFICATION

[0035] FIG. 3 illustrates an exemplary process 300 for operating a virtual assistant using speaker identification according to various examples. In some examples, process 300 can be performed by a system similar or identical to system 100 having a user device similar or identical to user device 102 configured to implement a virtual assistant capable of

continuously (or intermittently over an extended period of time) monitoring an audio input for a receipt of a trigger phrase that initiates activation of the virtual assistant. For example, a user device implementing the virtual assistant can continuously or intermittently monitor sounds, speech, and the like detected by a microphone of the user device without performing an action, such as performing a task flow, generating an output response in an audible (e.g., speech) and/or visual form, or the like, in response to the monitored sounds and speech. However, in response to detecting the trigger phrase, the virtual assistant can perform a speaker identification process to ensure that the speaker of the trigger phrase is the intended operator of the virtual assistant. Upon verification of the identity of the speaker, the virtual assistant can be activated, causing the virtual assistant to process a subsequently received word or phrase and to respond accordingly. A more detailed description of process 300 is provided below.

[0036] At block 302 of process 300, an audio input including user speech can be received at a user device. In some examples, a user device (e.g., user device 102) can receive the audio input including user speech via a microphone (e.g., microphone 230). The microphone can convert the audio input into an analog or digital representation, and provide audio data representing the audio input to one or more processors (e.g., processor(s) 204) of the user device.

[0037] At block 304, speech-to-text conversion can be performed on the audio input received at block 302 to determine whether the audio input includes user speech containing a predetermined trigger phrase. The trigger phrase can include any desired set of one or more predetermined words, such as "Hey Siri." The trigger phrase can be used to activate the virtual assistant and signal to the virtual assistant that a user input, such as a request, command, or the like, will be subsequently provided. For example, a user may utter the trigger phrase "Hey Siri," followed by the command "Call Mom," to activate the virtual assistant and request that the virtual assistant initiate a phone call to the phone number associated with "Mom" in the user's contact list. In some examples, the speech-to-text conversion performed at block 304 can be performed locally on the user device (e.g., user device 102). In other examples, the speech-to-text conversion can be performed at a remote sever (e.g., server system 110). In these examples, the user device can transmit the audio input received at block 302 to the remote server through one or more networks (e.g., network 108). The remote server can perform the speech-to-text conversion on the audio input and return a textual representation of the audio input to the user device. If it is determined at block 304 that the audio input does not include the trigger phrase, process 300 can return to block 302. Blocks 302 and 304 can be repeatedly performed at any desired interval of time to monitor the audio input for the trigger phrase. If it is instead determined at block 304 that the audio input received at block 302 includes the trigger phrase, process 300 can proceed to block 305.

[0038] At block 305, the user device can generate a speaker profile, selectively perform speaker recognition using the speaker profile, and selectively activate the virtual assistant in response to positively identifying the speaker using speaker recognition. In some examples, the speaker profile can generally include one or more voice prints generated from an audio recording of a speaker's voice. The voice prints can be generated using any desired speech recognition technique, such as by generating i- vectors to represent speaker utterances.

Speaker recognition can be performed using the voice prints of a speaker profile by comparing an audio input containing user speech with the voice prints in the speaker profile. As discussed in greater detail below, block 305 can include blocks 306, 308, 310, and 312 for allowing the user device to operate in multiple modes of operation based on a status of the speaker profile.

[0039] Specifically, at block 306, the user device can select one of multiple modes in which to operate. In some examples, the multiple modes can include a speaker profile building mode (represented by block 308) in which a speaker's voice can be modeled to generate a speaker profile, a speaker profile modifying mode (represented by block 310) in which a speaker profile can be used to verify the identity of a user and in which the speaker profile can be updated based on newly received user speech, and a static speaker profile mode in which an existing speaker profile can be used to verify the identity of a user and in which the speaker profile may not be changed based on newly received user speech. A more detailed description of each of these modes is provided below with respect to FIGs. 4-7.

[0040] In some examples, the speaker profile building mode (represented by block 308) can be selected at block 306 if a speaker profile for a user's voice does not exist or if the speaker profile includes less than a lower threshold number of voice prints (e.g.., 1, 5, 10, or other number of voice prints). In some examples, the speaker profile modifying mode (represented by block 310) can be selected at block 306 if the speaker profile for the user's voice includes a number of voice prints between the lower threshold number of voice prints and an upper threshold number of voice prints (e.g., 15, 20, 25, 30, or other number of voice prints). In some examples, the static speaker profile mode (represented by block 312) can be selected at block 306 if the speaker profile for the user's voice includes more than the upper threshold number of voice prints. [0041] If it is determined at block 306 that the appropriate mode of operation is the speaker profile building mode (e.g., because no speaker profile exists or because the speaker profile includes an insufficient number of voice prints), process 300 can proceed to block 308 where the virtual assistant can operate in the speaker profile building mode. FIG. 4 illustrates an exemplary process 400 for operating the virtual assistant in this mode.

[0042] At block 402, the user device can add the audio input that was received at block 302 and determined to include the trigger phrase at block 304 to an existing speaker profile (if such a profile exists), or can be used create a new speaker profile (if no speaker profile currently exists). Adding the audio input to the speaker profile can generally include generating a voice print from the audio input (e.g., by generating an i- vector representation of the audio input) and storing the voice print in association with the speaker profile. In some examples, the stored voice print can be annotated with contextual data, such as a location where the audio input used to generate the voice print was received (e.g., based on location data generated by the user device), a time that the audio input used to generate the voice print was received, or the like.

[0043] At block 404, the user device can activate the virtual assistant by processing audio input received subsequent to the audio input containing the trigger phrase. For example, block 404 can include receiving the subsequent audio input, performing speech-to- text conversion on the subsequently received audio input to generate a textual representation of user speech contained in the subsequently received audio input, determining a user intent based on the textual representation, an acting on the determined user intent by performing one or more of the following: identifying a task flow with steps and parameters designed to accomplish the determined user intent; inputting specific requirements from the determined user intent into the task flow; executing the task flow by invoking programs, methods, services, APIs, or the like; and generating output responses to the user in an audible (e.g., speech) and/or visual form.

[0044] After completing block 404, process 400 can end and the operation of the user device can return to block 302 of process 300. Blocks 302 and 304 can then be repeatedly performed until detecting another occurrence of the trigger phrase in the audio input. In response, block 306 can again be performed to determine the appropriate mode in which to operate. In some examples, the profile building mode represented by block 308 and process 400 can continue to be selected until a sufficient number of voice prints are generated for the speaker profile. As mentioned above, this can be 1, 5, 10, or any other desired number of voice prints. Thus, blocks 302, 304, 306, and 308 (e.g., process 400) can repeatedly be performed until the speaker profile includes this number of voice prints.

[0045] Using process 400, the virtual assistant can advantageously use utterances of the trigger phrase received at block 302 to build a speaker profile without forcing the user to perform a separate enrollment process. Instead, the virtual assistant can use speech from the user's natural interaction with the virtual assistant to generate this speaker profile. This can result in a speaker profile that more accurately represents the user's voice. Additionally, while using process 400 in the speaker profile building mode, the virtual assistant can process subsequently received audio input at block 404 without previously performing speaker identification on the user's utterance of the trigger phrase. This can be done because either a speaker profile does not exist or because an existing speaker profile contains an insufficient number of voice prints to reliably identify the speaker.

[0046] Once the speaker profile generated using process 400 includes the desired number of voice prints, receipt of an audio input containing the trigger phrase at block 302 can cause the user device to determine at block 306 that the appropriate mode of operation is the speaker profile modifying mode represented by block 310. As mentioned above, in some examples, the speaker profile modifying mode can be selected if the speaker profile for the user includes a number of voice prints between a lower threshold number of voice prints (e.g.., 1, 5, 10, or other number of voice prints) and an upper threshold number of voice prints (e.g., 15, 20, 25, 30, or other number of voice prints). FIG. 5 illustrates an exemplary process 500 for operating the virtual assistant in this mode.

[0047] At block 502, the user device can perform a speaker identification process on the audio input received at block 302 of process 300 to determine whether the speaker is a predetermined user (e.g., an authorized user of the device). Any desired speaker

identification process can be used, such as an i-vector speaker identification process. In some examples, block 502 can include comparing the audio input received at block 302 with some or all of the voice prints of the speaker profile generated using process 400. For instance, in some examples, block 502 can include determining whether the audio input received at block 302 matches (e.g., matches within an accepted tolerance) each of the voice prints of the speaker profile. If it is determined that the audio input matches a threshold number of the voice prints of the speaker profile, it can be determined that the speaker of the utterance contained in the audio input is the user represented by the speaker profile. The threshold number of matching voice prints can be any desired number, such as 1 voice print, 50% of the voice prints, 75% of the voice prints, all of the voice prints, or any other number depending on the desired confidence that the speaker is actually the user represented by the speaker profile. In some examples, the threshold number of matching voice prints can depend on the number of voice prints contained in the speaker profile. For example, if the speaker profile includes a large number of voice prints, block 502 can require a larger threshold number or percentage to require a closer match to the speaker profile. Conversely, if the speaker profile includes a small number of voice prints, block 502 can require a lower threshold number or percentage to require less stringent matching to the speaker profile

[0048] In other examples where the voice prints are annotated with contextual information, block 502 can include comparing the audio input received at block 302 with a subset of the voice prints of the speaker profile that match contextual information associated with the audio input. For example, if the audio input received at block 302 was received at a location corresponding to the user's office, block 502 can include determining whether the audio input received at block 302 matches the voice prints generated from audio data received at a location corresponding to the user's office, but can exclude determining whether the audio input received at block 302 matches the voice prints generated from audio data received at a location corresponding to the user's home or in the user's vehicle. Comparing the audio input to the speaker profile in this way can advantageously result in greater speaker identification accuracy since the conditions in which the audio input and the audio input used to generate the voice prints are more closely related.

[0049] In yet other examples, the user device can include a speaker profile for more than one user. In these examples, block 502 can include comparing the audio input received at block 302 with some or all of the voice prints of some or all of the speaker profiles. The determined identity of the speaker can be the user associated with the speaker profile that most closely matches the audio input.

[0050] Using any of the speaker identification processes described above, if it is determined at block 502 that the speaker of the audio input is the predetermined user represented by the speaker profile, process 500 can proceed to block 504.

[0051] At block 504, the audio input can be added to the speaker profile in a manner similar or identical to block 402 of process 400. In some examples, the audio input can be added to the speaker profile to increase the total number of voice prints contained in the speaker profile. In other examples, the speaker profile can include a maximum number of voice prints and block 504 can include adding the most recently received audio input to the speaker profile and removing the voice print associated with the oldest received audio input from the speaker profile. At block 506, the virtual assistant can be activated and

subsequently received audio input can be processed in a manner similar or identical to block 404 of process 400. After completing block 506, the user device can return to block 302 of process 300.

[0052] Referring back to block 502 of process 500, if it is instead determined that the speaker of the audio input is not the predetermined user represented by the speaker profile, process 500 can proceed from block 502 to block 508. At block 508, the user device can add the audio input that was received at block 302 of process 300 and determined to not be made by the predetermined user represented by the speaker profile to an alternate speaker profile (if such a profile exists), or can be used create a new alternate speaker profile (if no alternate speaker profile currently exists). The alternate speaker profile can represent the voice of one or more users that are different than the predetermined user represented by the speaker profile. Similar to adding the audio input to the speaker profile at block 402 or 504, adding the audio input to the alternate speaker profile can include generating a voice print from the audio input (e.g., by generating an i-vector representation of the audio input) and storing the voice print in association with the alternate speaker profile. In some examples, the stored voice print can be annotated with contextual data, such as a location where the audio input used to generate the voice print was received (e.g., based on location data generated by the user device), a time that the audio input used to generate the voice print was received., or the like.

[0053] In some examples, a single alternate speaker profile can be generated at block 508. In these examples, all audio inputs determined to not be spoken by the predetermined user at block 502 can be added to this alternate speaker profile at block 508. In other examples, separate alternate speaker profiles can be generated for each uniquely identified speaker. For example, block 508 can include comparing the audio input received at block 302 with all alternate speaker profiles to determine if the speaker of the audio input matches an existing speaker profile. If it is determined at block 508 that the speaker of the audio input matches one of the existing alternate speaker profiles, the audio input can be added to that alternate speaker profile. If it is instead determined at block 508 that the speaker of the audio input does not match one of the existing alternate speaker profiles, a new alternate speaker profile can be generated using the audio input. [0054] In some examples, after one or more alternate speaker profiles are generated at block 508, subsequent performances of block 502 can include comparing the audio input received at block 302 with the voice prints of both the speaker profile and alternate speaker profile(s). For example, it can be determined whether the audio input received at block 302 matches (e.g., within an accepted tolerance) each of the voice prints of the speaker profile and each of the voice prints of the alternate speaker profile(s). In some examples, if it is determined that the audio input matches a greater number or percentage of voice prints from the speaker profile than a number or percentage of voice prints from the alternate speaker profile(s), it can be determined that the speaker of the utterance contained in the audio input is the predetermined user represented by the speaker profile. If it is instead determined that the audio input matches a greater number or percentage of voice prints from the alternate speaker profile(s) than a number or percentage of voice prints from the speaker profile, it can be determined that the speaker of the utterance contained in the audio input is not the predetermined user represented by the speaker profile.

[0055] After completing block 508, the user device can return to block 302 of process 300 without activating the virtual assistant and without processing subsequently received audio inputs. Using process 500, the virtual assistant can perform speaker identification using the speaker profile generated using process 400 to verify that the speaker of the trigger phrase is the authorized or intended user of the virtual assistant. This can advantageously be performed to prevent unauthorized or unintended triggering of the virtual assistant. For example, performing of speaker identification can be useful in situations where the user device is located in a room with similar devices configured to activate a virtual assistant in response to the same trigger phrase. In this situation, it can be undesirable for one user's virtual assistant to activate in response to an utterance of the trigger phrase by another user. Additionally, process 500 can be used to modify or update the speaker profile with newly received speech from the user' s natural interaction with the virtual assistant to update the speaker profile. Updating the speaker profile can advantageously be performed to allow the speaker profile to adapt to changes in the user's voice over time and/or account for different conditions in which the user' s speech is received.

[0056] After completing block 506 or 508, the operation of the user device can return to block 302 of process 300. Blocks 302 and 304 can repeatedly be performed until detecting another occurrence of the trigger phrase in the audio input. In response, block 306 can be performed to determine the appropriate mode in which to operate. In some examples, the speaker profile modifying mode represented by block 310 and process 500 can continue to be selected until a sufficient number of voice prints are generated for the speaker profile. As mentioned above, this can be 15, 20, 25, 30, or any other desired number of voice prints. Thus, blocks 302, 304, 306, and 310 can repeatedly be performed until the speaker profile includes this number of voice prints.

[0057] Once the speaker profile includes the desired number of voice prints, receipt of an audio input containing the trigger phrase at block 302 can cause the user device to determine at block 306 that the appropriate mode of operation is the static speaker profile mode represented by block 312. As mentioned above, in some examples, the static speaker profile mode can be selected if the speaker profile for the user' s voice includes a number of voice prints equal to or greater than an upper threshold number of voice prints (e.g., 15, 20, 25, 30, or other desired number of voice prints). FIG. 6 illustrates an exemplary process 600 for operating the virtual assistant in this mode.

[0058] At block 602, the user device can perform a speaker identification process on the audio input received at block 302 of process 300 in a manner similar or identical to that of block 502 of process 500. If it is determined that the speaker of the audio input is the predetermined user represented by the speaker profile, then process 600 can proceed to block 604 without adding the audio input to a speaker profile in a manner similar or identical to block 402 of process 400 or block 504 of process 500. At block 604, the virtual assistant can be activated and subsequently received audio input can be processed in a manner similar or identical to block 404 of process 400 or block 506 of process 500. If it was instead determined at block 602 that the speaker of the audio input was not the predetermined user represented by the speaker profile, process 600 can end without adding the audio input to a speaker profile and without processing subsequently received audio input. Instead, the operation of the user device can return to block 302 of process 300.

[0059] Using process 600, the virtual assistant can perform speaker identification using the speaker profile generated using process 400 and/or 500 to verify that the speaker of the trigger phrase is the authorized or intended user of the virtual assistant. This can

advantageously be performed to prevent unauthorized or unintended triggering of the virtual assistant. For example, performing of speaker identification can be useful in situations where the user device is located in a room with similar devices configured to activate a virtual assistant in response to the same trigger phrase. In this situation, it can be undesirable for one user's virtual assistant to activate in response to an utterance of the trigger phrase by another user. Additionally, process 600 can forgo the adding of audio input to a speaker profile. This can be performed in situations where the speaker profile has a sufficient number of voice prints and additional voice prints may not be necessary. This can advantageously reduce the amount of storage required to store the speaker profile(s) and reduce the processing power required to add voice prints to existing speaker profile(s).

[0060] In some examples, the speaker identification performed at block 502 and/or 602 can include determining whether the speaker identification made using the speaker profile and/or alternate speaker profile(s) was erroneous based on contextual information. FIG. 7 illustrates an exemplary process 700 for performing speaker identification using contextual information that can be performed at block 502 or 602. At block 702, a speaker identification process can be performed on the audio input received at block 302 of process 300 in a manner similar or identical to block 502 of process 500 or block 602 of process 600. If it is determined that the speaker of the audio input is the predetermined user represented by the speaker profile, process 700 can proceed to block 704.

[0061] At block 704, it can be determined based on contextual information whether the determination made at block 702 was likely erroneous. In some examples, this can include determining whether a request to cancel an operation is received (e.g., via a verbal request from the user, via a button or other input mechanism of the user device, or the like). The occurrence of the request to cancel can indicate that the determination made at block 702 was likely incorrect because the user is now canceling the operation that an utterance of the trigger phrase was intended to initiate. This can occur, for example, in a situation where multiple users are located in the same room and an utterance from one user triggers activation of the virtual assistant on another user's device. In other examples, block 704 can include determining whether subsequent audio input including user speech is not received within a threshold length of time. The threshold length of time can be any desired length of time, but, in some examples, can represent the amount of time during which a user is likely to provide a request or command to the virtual assistant after speaking the trigger phrase. The absence of an audio input including user speech within this threshold length of time can indicate that the determination made at block 702 was likely incorrect because the user is not providing the type of input expected after utterance of a trigger phrase, suggesting that the user did not intend to trigger the virtual assistant. In other examples, other types of contextual information that can be indicative of the determination made at block 702 being correct or incorrect can be used. If it is determined at block 704 that the determination made at block 702 was not erroneous, process 700 can proceed to block 708 where a final determination that the speaker of the audio input is the predetermined user represented by the speaker profile can be made. Alternatively, if it is instead determined at block 704 that the determination made at block 702 was likely erroneous, process 700 can proceed to block 710 where a final determination that the speaker of the audio input is not the predetermined user represented by the speaker profile can be made.

[0062] Referring back to block 702, if it is instead determined that the speaker of the audio input is not the predetermined user represented by the speaker profile, process 700 can proceed to block 706. At block 706, it can be determined based on contextual information whether the determination made at block 702 was likely erroneous. In some examples, this can include determining whether another audio input including the trigger phrase is received within a threshold length of time. The threshold length of time can be any desired length of time. The receipt of another trigger phrase can indicate that the determination made at block 702 was likely incorrect because the user is again attempting to trigger the virtual assistant. In other examples, block 706 can include determining whether another form of a request to trigger the virtual assistant (e.g., via a selection of a mechanical button or other input mechanism) is received within a threshold length of time. The threshold length of time can be any desired length of time. The receipt of another trigger phrase can indicate that the determination made at block 702 was likely incorrect because the user is continuing to attempt to trigger the virtual assistant using means other than the trigger phrase. In other examples, other types of contextual information that can be indicative of the determination made at block 702 being correct or incorrect can be used. If it is determined at block 706 that the determination made at block 702 was not erroneous, the process can proceed to block 710 where a final determination that the speaker of the audio input is not the predetermined user represented by the speaker profile can be made. Alternatively, if it is instead determined at block 706 that the determination made at block 702 was likely erroneous, process 700 can proceed to block 708 where a final determination that the speaker of the audio input is the predetermined user represented by the speaker profile can be made.

[0063] Processes 300, 400, 500, 600, and 700 can continue to be repeated any number of times in accordance with the description above to process user inputs and to generate/modify speaker profiles for performing speaker identification. In some examples, the speaker profiles can be reset or deleted in response to a request from the user, a request to change a language of the virtual assistant, or other input. In these examples, process 300 can again operate in the speaker profile generating mode (represented by block 308) and continue as described above.

[0064] While specific modes of operation and criteria for selecting each mode (e.g., at block 306) are described above, it should be appreciated that any other criteria can be used depending on the desired operation of the virtual assistant. Moreover, it should be appreciated that process 300 can include alternate, additional, or fewer modes of operation. For instance, in some examples, process 300 can include only the speaker profile modifying mode. In these examples, blocks 306, 308, and 312 can be omitted. In other examples, process 300 can include the speaker profile building mode and the speaker profile modifying mode, but can exclude the static speaker profile mode. In these examples, process 300 can include block 306, 308, and 310, but can exclude block 312. It should be appreciated that other combinations of these and other modes are possible.

[0065] It should be appreciated that the blocks of processes 300, 400, 500, 600, and 700 can be performed on user device 102, server system 110, or a combination of user device 102 and server system 110. For instance, in some examples, all blocks of processes 300, 400, 500, 600, and 700 can be performed on user device 102. In other examples, some blocks of processes 300, 400, 500, 600, and 700 can be performed at user device 102, while other blocks of processes 300, 400, 500, 600, and 700 can be performed at server system 110. In yet other examples, all blocks of processes 300, 400, 500, 600, and 700 can be performed at server system 110.

ELECTRONIC DEVICE

[0066] In accordance with some examples, FIG. 8 shows a functional block diagram of an electronic device 800 configured in accordance with the principles of the various described examples. The functional blocks of the device can be implemented by hardware, software, or a combination of hardware and software to carry out the principles of the various described examples. It is understood by persons of skill in the art that the functional blocks described in FIG. 8 can be combined or separated into sub-blocks to implement the principles of the various described examples. Therefore, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein.

[0067] As shown in FIG. 8, electronic device 800 can include a touch screen display unit 802 configured to display a user interface and to receive touch input, and a sound receiving unit 804 configured to receive sound input. In some examples, electronic device 800 can include a speaker unit 806 configured to generate sound. Electronic device 800 can further include a processing unit 808 coupled to touch screen display unit 802 and sound receiving unit 804 (and, optionally, coupled to speaker unit 806). In some examples, processing unit 808 can include receiving unit 810, determining unit 812, adding unit 814, speech-to-text unit 816, intent determining unit 818, task determining unit 820, parameter determining unit 822, and task performing unit 824.

[0068] Processing unit 808 can be configured to receive (e.g., from audio receiving unit 804) an audio input comprising user speech. Determining unit 810 can be configured to determine whether a speaker of the user speech is a predetermined user based at least in part on a speaker profile for the predetermined user. Adding unit 812 can be configured to add, in accordance with a determination that the speaker of the user speech is the predetermined user, the audio input comprising user speech to the speaker profile for the predetermined user.

[0069] In some examples, the speaker profile for the predetermined user comprises a plurality of voice prints.

[0070] In some examples, each of the plurality of voice prints of the speaker profile for the predetermined user was generated from previously received audio inputs comprising user speech.

[0071] In some examples, determining whether the speaker of the user speech is the predetermined user based at least in part on the speaker profile for the predetermined user comprises: determining whether the audio input comprising user speech matches at least a threshold number of the plurality of voice prints; in accordance with a determination that the audio input comprising user speech matches at least the threshold number of the plurality of voice prints, determining that the speaker of the user speech is the predetermined user; and in accordance with a determination that the audio input comprising user speech does not match at least the threshold number of the plurality of voice prints, determining that the speaker of the user speech is not the predetermined user.

[0072] In some examples, determining whether the speaker of the user speech is the predetermined user based at least in part on the speaker profile for the predetermined user comprises: determining whether the audio input comprising user speech matches at least a threshold number of the plurality of voice prints; in accordance with a determination that the audio input comprising user speech matches at least the threshold number of the plurality of voice prints: determining whether an erroneous speaker determination was made based on contextual data; in accordance with a determination that an erroneous speaker determination was not made based on contextual data, determining that the speaker of the user speech is the predetermined user; and in accordance with a determination that an erroneous speaker determination was made based on contextual data, determining that the speaker of the user speech is not the predetermined user; and in accordance with a determination that the audio input comprising user speech does not match at least the threshold number of the plurality of voice prints: determining whether an erroneous speaker determination was made based on contextual data; in accordance with a determination that an erroneous speaker determination was not made based on contextual data, determining that the speaker of the user speech is not the predetermined user; and in accordance with a determination that an erroneous speaker determination was made based on contextual data, determining that the speaker of the user speech is the predetermined user.

[0073] In some examples, adding the audio input comprising user speech to the speaker profile for the predetermined user comprises: generating a voice print from the audio input comprising user speech; and storing the voice print in association with the speaker profile for the predetermined user.

[0074] In some examples, adding unit 814 can be further configured to add, in accordance with a determination that the speaker of the user speech is not the predetermined user, the audio input comprising user speech to a speaker profile for an alternate user.

[0075] In some examples, the speaker profile for the alternate user comprises a plurality of voice prints.

[0076] In some examples, each of the plurality of voice prints of the speaker profile for the alternate user was generated from previously received audio inputs comprising user speech.

[0077] In some examples, determining whether the speaker of the user speech is the predetermined user is further based at least in part on the speaker profile for the alternate user.

[0078] In some examples, determining whether the speaker of the user speech is the predetermined user comprises: determining whether the audio input comprising user speech matches a greater number of voice prints of the speaker profile for the predetermined user than a number of voice prints of the speaker profile for the alternate user; in accordance with a determination that the audio input comprising user speech matches a greater number of voice prints of the speaker profile for the predetermined user than a number of voice prints of the speaker profile for the alternate user, determining that the speaker of the user speech is the predetermined user; and in accordance with a determination that the audio input comprising user speech does not match a greater number of voice prints of the speaker profile for the predetermined user than a number of voice prints of the speaker profile for the alternate user, determining that the speaker of the user speech is not the predetermined user.

[0079] In some examples, determining whether the speaker of the user speech is the predetermined user comprises: determining whether the audio input comprising user speech matches a greater number of voice prints of the speaker profile for the predetermined user than a number of voice prints of the speaker profile for the alternate user; in accordance with a determination that the audio input comprising user speech matches a greater number of voice prints of the speaker profile for the predetermined user than a number of voice prints of the speaker profile for the alternate user: determining whether an erroneous speaker determination was made based on contextual data; in accordance with a determination that an erroneous speaker determination was not made based on contextual data, determining that the speaker of the user speech is the predetermined user; and in accordance with a determination that an erroneous speaker determination was made based on contextual data, determining that the speaker of the user speech is not the predetermined user; and in accordance with a determination that the audio input comprising user speech does not match a greater number of voice prints of the speaker profile for the predetermined user than a number of voice prints of the speaker profile for the alternate user: determining whether an erroneous speaker determination was made based on contextual data; in accordance with a determination that an erroneous speaker determination was not made based on contextual data, determining that the speaker of the user speech is not the predetermined user; and in accordance with a

determination that an erroneous speaker determination was made based on contextual data, determining that the speaker of the user speech is the predetermined user.

[0080] In some examples, speech-to-text unit 816 can be configured to perform, in accordance with a determination that the speaker of the user speech is the predetermined user, speech-to-text conversion on a second audio input comprising a second user speech, wherein the second audio input is received after receiving the audio input comprising user speech. Intent determining unit 818 can be configured to determine, in accordance with a

determination that the speaker of the user speech is the predetermined user, a user intent based on the second user speech. Task determining unit 820 can be configured to determine, in accordance with a determination that the speaker of the user speech is the predetermined user, a task to be performed based on the second user speech. Task performing unit 824 can be configured to perform, in accordance with a determination that the speaker of the user speech is the predetermined user, the task to be performed in accordance with the determined parameter.

[0081] As described above, one aspect of the present technology is the gathering and use of data available from various sources to improve the delivery to users of invitational content or any other content that may be of interest to them. The present disclosure contemplates that in some instances, this gathered data can include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data can include demographic data, location-based data, telephone numbers, email addresses, home addresses, or any other identifying information.

[0082] The present disclosure recognizes that the use of such personal information data, in the present technology, can be used to the benefit of users. For example, the personal information data can be used to deliver targeted content that is of greater interest to the user. Accordingly, use of such personal information data enables calculated control of the delivered content. Further, other uses for personal information data that benefit the user are also contemplated by the present disclosure.

[0083] The present disclosure further contemplates that the entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such

personal information data will comply with well-established privacy policies

and/or privacy practices. In particular, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining personal information data private and secure. For example, personal information from users should be collected for legitimate and reasonable uses of the entity and not shared or sold outside of those legitimate uses. Further, such collection should occur only after receiving the informed consent of the

users. Additionally, such entities would take any needed steps for safeguarding and securing access to such personal information data and ensuring that others with access to the personal information data adhere to their privacy policies and procedures. Further, such entities can subject themselves to evaluation by third parties to certify their adherence to

widely accepted privacy policies and practices. [0084] Despite the foregoing, the present disclosure also contemplates examples in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and/or software elements can be provided to prevent or block access to such personal information data. For example, in the case of advertisement delivery services, the present technology can be configured to allow users to select to "opt in" or "opt out" of participation in the collection of personal information data during registration for services. In another example, users can select not to provide location information for targeted content delivery services. In yet another example, users can select to not provide precise location information, but permit the transfer of location zone information.

[0085] Therefore, although the present disclosure broadly covers use of personal information data to implement one or more various disclosed examples, the present disclosure also contemplates that the various examples can also be implemented without the need for accessing such personal information data. That is, the various examples of the present technology are not rendered inoperable due to the lack of all or a portion of such personal information data. For example, content can be selected and delivered to users by inferring preferences based on non-personal information data or a bare minimum amount of personal information, such as the content being requested by the device associated with a user, other non-personal information available to the content delivery services, or publicly available information.

[0086] Although examples have been fully described with reference to the

accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the various examples as defined by the appended claims.