The present application claims the benefit of U.S. Provisional Patent Application Serial No. 62/805,456, which was filed on February 14, 2019, by Madonna et al. for MULTI-ROLE DEVICES FOR AUTOMATION

ENVIRONMENTS, which is hereby incorporated by reference.

Incorporated Patents and Applications

U.S. Patent No. 7,930,644, issued April 19, 2011, and entitled

PROGRAMMING ENVIRONMENT AND METADATA MANAGEMENT FOR PROGRAMMABLE MULTIMEDIA CONTROLLER, is assigned to a common assignee with the present application and is hereby incorporated by reference. U.S. Patent No. 8,213,463, issued July 3, 2012, and entitled LOCATION-BASED SHARING OF MULTIMEDIA CONTROL RESOURCES, is assigned to a common assignee with the present application and is hereby incorporated by reference. U.S. Patent No. 8,296,669, issued October 23, 2012, and entitled VIRTUAL ROOM- BASED LIGHT FIXTURE AND DEVICE CONTROL, is assigned to a common assignee with the present application and is hereby incorporated by reference. U.S Patent No. 8,659,704, issued February 25, 2014, and entitled APPARATUS AND METHOD FOR MIXING GRAPHICS WITH VIDEO IMAGES, is assigned to a common assignee with the present application and is hereby incorporated by reference. U.S. Patent No. 9,153,125, issued October 6, 2015, and entitled

PROGRAMMABLE MULTIMEDIA CONTROLLER WITH PROGRAMMABLE SERVICES, is assigned to a common assignee with the present application and is hereby incorporated by reference. Co-pending application Serial No. 16/123,731, filed September 6, 2018, and entitled SMALL SCREEN VIRTUAL ROOM-BASED USER INTERFACE, is assigned to a common assignee with the present application and is hereby incorporated by reference. Co-pending application Serial No.

16/254,245, filed January 22, 2019, and entitled DEVICE MOUNTING SYSTEM, is assigned to a common assignee with the present application and is hereby incorporated by reference. Field of the Invention

The present invention relates generally to the field of automation and, more specifically, to enhancing the capabilities of various devices to enable them to perform multiple roles in automation environments.

Background Information

Demand for automation of residential and commercial environments has resulted in a dramatically expanding market of“smart” doorbells, thermostats, lighting, security systems, audio, video, and numerous other devices. These devices are offered by a substantial number of manufacturers who, for competitive or performance reasons, have adopted different control protocols that are not generally compatible with each other. Thus, a“smart” thermostat offered by one manufacturer does not operate in accordance with a widely adopted standard, and often will not interoperate with“smart” lighting offered by a different manufacturer. Among other disadvantages, this lack of interoperability limits the choices of an end user in selecting devices for a particular automation environment. Another disadvantage is that an end user is prevented from effectively controlling all of the devices in an environment with a single control device (e.g., remote control, smartphone, or tablet).

As a result, an end user is often forced to either select one manufacturer, even though that manufacturer does not offer all of the“smart” devices of interest to the end user, or select multiple manufacturers at increased cost and lack of

interoperability.

SUMMARY OF THE INVENTION

In accordance with one aspect of the invention, an automation control device, such as a host or hub, is provided with enhanced functionality so that it becomes a multi-role automation control device. The multi-role automation control device is capable of controlling an expanded variety of“smart” or other devices whose native control protocol is not generally compatible. The multi-role automation control device may be implemented using a variety of underlying devices including a host, hub, soundbar,“smart” thermostat,“smart” amplifier, audio speaker, or other device.

In accordance with another aspect of the invention, an automation control hub is provided with enhanced functionality so that it becomes a multi -role automation control hub. Leveraging high performance graphics processing available in the automation control hub, the multi-role automation control hub is provided with an enhanced on screen display. The enhanced on screen display provides a user interface for automation control over an expanded variety of devices than the automation control hub is capable of controlling with its native functionality. The enhanced on screen display may also provide advanced video display management including windowing, tiling, and picture-in-picture among others. The enhanced on screen display may also support configuration, as well as real time control and status of substantially all aspects of an automation environment. More specifically, the enhanced on screen display may provide access to and control over services, favorites, scenes, automations, and scheduling.

In addition, multiple automation control hubs with enhanced on screen displays may be pooled together as a shared resource that distributes such on screen displays to a large number of TVs or video monitors. Such distribution may be effected by transmitting or streaming over IP dedicated to a TV HDMI.

In accordance with another aspect of the invention, a ceiling or wall mountable speaker, which includes wired network connectivity, is provided with additional functionality including wireless network connectivity. As a result, each speaker becomes a multi-role speaker that functions both as an audio output device as well as a wireless network access point. In addition, the multi-role speaker may perform a third role, an automation control device, as referenced above.

In accordance with another aspect of the invention, an automation control hub is provided with AVB network connectivity and becomes a multi -role automation control hub. Through an AVB network, the multi-role automation control hub distributes music to networked speakers, soundbars, or other devices.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention description below refers to the accompanying drawings, of which:

Fig. 1 is a block diagram an automated environment which includes a multi- role automation host in accordance with one aspect of the present invention;

Fig. 2 is a block diagram of a soundbar which may be used as a base for the multi-role automation host shown in Fig. 1 ; Fig. 3 is a message flow diagram illustrating an example of higher level messaging performed by the multi-role automation host and other devices shown in Fig. 1;

Fig. 4 is a message flow diagram illustrating an example of lower level messaging performed by the multi-role automation host and other devices shown in Fig. 1;

Fig. 5 is a block diagram of an automated environment which includes a multi-role automation hub that provides an enhanced on screen display in accordance with one aspect of the present invention;

Figs. 6A-6D are exemplary screenshots of the enhanced on screen display produced by the multi-role automation hub shown in Fig. 5;

Fig. 7 is a message flow diagram illustrating an example of higher level messaging performed by the multi-role automation hub with enhanced on screen display shown in Fig. 5;

Fig. 7A is a block diagram of an automated environment in which multiple are pooled to become a shared resource;

Fig. 7B is a message flow diagram illustrating communication among various devices in Fig. 7 A;

Fig. 8 is a block diagram of a multi -role speaker in accordance with one aspect of the present invention; and

Fig. 9 is a block diagram of a multi -role amplifier in accordance with one aspect of the present invention.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

Fig. 1 shows an automation environment 100 which includes a multi-role automation host 104. A first network cloud 102 is accessible by multi-role automation host 104. Multi-role automation host 104 includes a system configuration (not shown) which, in turn, includes a component or device profile (not shown) associated with each profiled device 108a...108b controlled by multi-role automation host 104. In general, a device profile is a file or other data structure which describes the inputs, outputs, and available set of commands for the associated device. Through either internal storage, network cloud 102, or another external source, multi-role automation host 104 must have access to an associated device profile in order to control a given device.

Examples of a system configuration, component profile, and hardware and software suitable for multi-role automation host 104 may be found in U.S. Patent No. 9,153,125 incorporated by reference above.

A user may interact with, and effectively control, multi -role automation host 104 by operating a user device 116a running a host control application 120 which is compatible with multi-role automation host 104. User device 116a may be implemented with a smartphone, tablet, or other device capable of running host control application 120 and having appropriate network connectivity to communicate with multi-role automation host 104. Alternatively, a user may interact with and effectively control multi-role automation host 104 using a user remote 118a which is compatible with multi-role automation host 104. Multi-role automation host 104 may, for example, be based upon a Smart Host Model SHC-S2-00, a surround sound soundbar, a“smart” Multistat thermostat, a“smart” audio speaker (as described below in connection with Fig. 8), a“smart” amplifier (as described below in connection with Fig. 9), or other devices offered by Savant Systems, LLC of Hyannis, MA.

A second network cloud 110 is accessible by an automation hub 112. Unlike multi-role automation host 104 which is capable of controlling any profiled device, automation hub 112 is conventionally capable of controlling only devices 114a-l 14c which have been certified as control protocol compliant (/. _<? ., devices which have been certified as compliant with the control protocol specified by the manufacturer of automation hub 112).

A user may interact with and effectively control automation hub 112 by operating a user device 116b running a hub control application 122 which is compatible with automation hub 112. User device 116b may be implemented with a smartphone, tablet, or other device capable of running hub control application 122 and having appropriate network connectivity to communicate with automation hub 112. Alternatively, a user may interact with and effectively control automation hub 112 using a user remote 118b which is compatible with automation hub 112. Automation hub 112 may, for example, be based on an Apple TV®, HomePod™, or iPad® running HomeKit®, all of which are offered by Apple Inc. of Cupertino, CA. Multi-role automation host 104 includes an automation bridge 106. In one embodiment, an SDK available from Apple Inc. may be used to create automation bridge 106 and embed it in multi-role automation host 104. As described in greater detail below, automation bridge 106 functions, in part, to communicate

(bidirectionally) with automation hub 112 using the same manufacturer- specified control protocol as used by certified control protocol compliant devices 114a-114c. Automation bridge 106 also functions to convert control protocol messages received from automation hub 112 to an appropriate form for processing by multi-role automation host 104 (and vice versa). Advantageously, through such communication and protocol translation, and depending upon the type of device, profiled devices

108a...108b as well as user remote 118a are exposed to automation hub 112. That is, profiled devices 108a...108b and user remote 118a, even though not certified as compliant with the control protocol used by automation hub 112, become visible to and controllable by automation hub 112 by virtue of multi -role automation host 104.

Fig. 2 shows a block diagram of a powered surround sound soundbar 200 which may be used to implement multi-role automation host 104 (Fig. 1). A microprocessor 202 is coupled in bidirectional communication with an audio digital signal processor (DSP) 204. Audio DSP 204 is also coupled to a TosLink 214.

Audio DSP 204 is coupled to digital to analog converters (not shown) whose outputs are coupled to an amplifier 206 whose outputs are coupled to various audio speakers 208.

Microprocessor 202 is also coupled in bidirectional communication with a WISA module 210 as well as a WiFi dual band module 212. In addition, microprocessor 202 is coupled to an LAN/A VB port 216, a microphone 218, control ports 220 which may include RS232 control portsinfrared (IR) control ports, or other control ports. An AC/DC power supply 224 provides power to soundbar 200.

Fig. 3 shows a message flow diagram illustrating an example of higher level messaging needed to make profiled devices 108a...108b and user remote 118a (Fig. 1) visible to and controllable by automation hub 112. Host control application 120, running on user device 116a, initiates a Discovery protocol 300 with multi-role automation host 104. Host control application 120 next sends a Host Provision message 302. In response, multi-role automation host 104 proceeds to Query 304 its configuration to identify any (profiled) device(s) that is of an appropriate type to be supported/controlled by automation hub 112. Assuming at least one profiled device of an appropriate type is identified, multi-role automation host 104, through bridge 106, transmits an Add Bridged Devices to Hub message 306 to automation hub 112.

Subsequently, host control application 120 issues a Control Device message 308 to multi-role automation host 104 which responds by exerting control 310 over the profiled device to which message 308 is directed. In contrast, when hub control application 122 issues a Control Device message 312 that is directed to a profiled device, automation hub 112 responds by forwarding a Control Request message 314 to multi-role automation host 104. In turn, multi-role automation host 104 responds by exerting control 316 over the profiled device to which message 312 is directed.

Fig. 4 shows a message flow diagram illustrating an example of lower level messaging that occurs when hub control application 122 and automation hub 112 are exerting control over profiled device 108a that has become visible through multi-role automation host 104 as described above. In this example, automation hub 112 represents an Apple TV®, HomePod™, or iPad® running HomeKit®. Hub control application 122 issues a HMActionSet execute message 400 to automation hub 112 which, in turn, issues a Device Characteristic Write message 402 to multi-role automation host 104. Device Characteristic Write message 402 could operate, for example, to dim a lamp.

In response, multi-role automation host 104 locates the profile corresponding with the device 404 to which message 402 is directed, and then sends a profiled command 406 to device 108a. Profiled command 406 is in substance comparable to message 402 but is prepared by multi-role automation host 104 in a form that is expected and understood by device 108a. Following receipt and execution of profiled command 406, device 108a sends a command response 408 to multi-role automation host 104 which, in turns, sends a Characteristic Update message 410 to automation hub 112.

Turning now to Fig. 5, an automation environment 500 is shown. In general, each component or device shown in Fig. 5 may include comparable structure and function to the corresponding component or device shown in Fig. 1 and described above with one exception. A multi-role automation hub 512, in its role as a hub, functions in a substantially comparable manner to automation hub 112 (Fig. 1) described above. However, multi-role automation hub 512, in an additional role, produces an enhanced on screen display (EOSD) for a television 524.

In a preferred embodiment, as referenced above, automation hub 112 may be based on Apple TV® running HomeKit®. While Apple TV® includes a limited on screen display which allows a user to select audio/video to play, or power on/off an attached TV, that limited on screen display is not capable of controlling other devices. However, Apple TV® does include a high performance graphics processing capability.

In accordance with one aspect of the present invention, an application is provided which runs on Apple Inc.’s tvOS®, and leverages the high performance graphics processing built into Apple TV® to produce an EOSD 600 as shown in Figs. 6A-6D. By placing an Apple TV® into a lockdown mode, the Apple TV® will automatically boot up into the application which produces EOSD 600.

As shown in Fig. 6A, EOSD 600 may be formatted as three horizontal bands of rotating icons, graphics, images, video or other content. For example, band 602 displays a series of“Scenes” corresponding to various rooms or areas of a home. When a user selects a particular“Scene” (by interacting with EOSD 600 using user devices 516a or 516b, or user remotes 518a or 518b (Fig. 5)) to center and select the corresponding image, previously defined conditions or states of lighting, devices, shades, media, or any other resource that is under the control of multi-role automation host 504 or multi -role automation hub 512, will be set in the selected room or area.

Band 604 of EOSD 600 displays icons corresponding to various systems and services. When a user selects a centered“Security” icon, EOSD 600 transitions to a screen shown in Fig. 6B in which band 608 displays live feeds from several closed circuit security cameras. Similarly, as shown in Fig. 6C, when a user selects a centered“Lighting” icon, EOSD 600 transitions to a screen shown in Fig. 6D in which band 610 displays true images of the current lighting conditions of individual rooms or lamps along with a vertical slider bar control. Exemplary techniques for generating and displaying such true images of current lighting conditions are described in U.S. Patent No. 8,296,669, and co-pending application Serial No.

16/123,731, both of which are incorporated by reference above.

EOSD 600 may also provide advanced video display management including windowing, tiling, and picture-in-picture among others. Details regarding hardware and software for generating EOSD 600 may be found in the patents and pending patent applications incorporated by reference above.

Fig. 7 shows a message flow diagram illustrating an example of lower level messaging that occurs when multi-role automation hub 512 with EOSD 600 interacts with multi-role automation host 504. Host control application 520 issues a Control Profiled Device message 700 to multi-role automation host 504. Because message 700 is directed to a profiled device, multi-role automation host 504 proceeds to Execute Profiled Device Command 702.

Subsequently, host control application 520 issues a Control Certified Device message 704. Because message 704 is not directed to a profiled device, multi-role automation host 504 issues a Forward Request message 706, via bridge 506, to multi role automation hub 512. Multi-role automation hub 512, in turn, issues a Control Request message 708 to automation hub 112 which proceeds to Execute Certified Device Command 710.

In accordance with one aspect of the invention, an application is provided which runs on Apple Inc.’s tvOS®, and provides the necessary functionality to transform an Apple TV® into multi-role automation host. Other devices which are controllable by a dedicated remote may also be suitable to serve as a multi-role automation host. In general, the application embodies all necessary automation host functionality including the ability to recognize and control profiled devices. Because of the native functionality of an Apple TV®, this type of multi-role automation host is capable of effectively controlling a mix of profiled devices and certified devices without an additional hub or other intermediate device. In addition, the capability of this type of multi -role automation host may be further enhanced by the addition of AVB network connectivity. Such connectivity, in the case of an Apple TV® multi role automation host, permits distribution of music in digital form directly to AVB enabled speakers, soundbars or other devices.

In accordance with another aspect of the invention, multiple Apple TVs may be pooled together to become a shared resource for multiple TVs. As shown in Fig. 7A, an automated environment 700 includes a group of TVs 712a-712c, each of which is coupled to a video switch 714. A multi -role automation host 716 is coupled to video switch 714, as are multiple Apple TVs 718a and 718b, and multiple hubs with enhanced OSD 726a and 726b (which are substantially similar to hub with enhanced OSD 512 (Fig. 5) described above).

In general, in response to commands from multi-role automation host 716, video switch 714 functions to set up and tear down video signal connections or paths between various ones of TVs 712a-712c, and various ones of Apple TVs 718a-718b or hubs with enhanced OSD 726a-726b. Multi-role automation host 716 may also command video switch 714 to set up a video signal connection between a single Apple TV 718a-718b and multiple TVs 712a-712c. Similarly, multi-role automation host 716 may also command video switch 714 to set up a video signal connection between a single hub with enhanced OSD 726a-726b and multiple TVs 712a-712c.

Fig. 7B is a message flow diagram illustrating communication among various devices in Fig. 7A needed to display an enhanced OSD on one of TVs 712a-712c.

The communication begins with a user (not shown) pressing a button on user remote 724 which causes transmission of an OSD Request for Zone X message 728 to multi- role automation host 716. Multi-role automation host 716 responds first by Dequeue Menu Token Y which, in effect, dedicates a particular Apple TV 718a-718b

(associated with the token) to the requested OSD session. Next, multi-role automation host 716 transmits an Activate OSD message 732 to a Menu Resource Y 762 (/. _<? ., a particular Apple TV associated with Token Y). This is followed by a Start Stream 1 message 734 from multi-role automation host 716 to a Menu Resource Y’s Video Transmitter 758 which, in turn, is followed by a Start Stream 1 With Resized Video message 736 to Current Transmitter Used By Zone X 764.

Next, multi-role automation host 716 sends a message 738 to Zone X Video Receiver 760 which sets up picture in picture and assigns the appropriate transmitters· At this point, the enhanced OSD appears in a picture in picture 740. A User OSD Navigation message 742 is then transmitted from user remote 724 to Menu Resource Y 762. This is followed by a Cancel or Select message 744 from user remote 724. If a Cancel message 744 was transmitted, an OSD Deactivated message 746 is transmitted from Menu Resource Y to multi-role automation host 716. This is followed by Menu Token Y being returned to the Queue 748 because the OSD session is over.

Subsequently, a message 750 is transmitted from multi-role host 716 to restore full screen video for the original transmitter. This is followed by a Stop Stream 1 message 752 to Current Transmitter 764, a Stop Stream 1 message to Menu Resource Y’s Video Transmitter 758, and restoration of full screen video 756.

In addition, similar to the enhanced OSD example described above, other services available on an Apple TV 718a-718b may be shared with multiple TVs 712a- 712c. Thus, services such as Netflix, Weather, or many others may be shared with multiple TVs 712a-712c by placing an Apple TV in lockdown mode and transmitting the selected video over IP, taking advantage of video switch 714 to set up the appropriate video connections.

In accordance with another aspect of the invention, a ceiling or wall mountable speaker, such as those disclosed in co-pending application Serial No.

16/254,245 and incorporated by reference above, or bookshelf or other speaker, is provided enhanced functionality and becomes a multi -role speaker. The previously disclosed speakers include a LAN/AVB port through which both digital audio and power over Ethernet (PoE) are received. As shown in Fig. 8, a multi-role speaker 800 includes a LAN/AVB port 802 coupled to a microprocessor 804. A microphone 806 is coupled to microprocessor 804 as is a digital signal processor (DSP) 808. An amplifier 810 is coupled to DSP 808 as well as speaker 812. Multi-role speaker 800 also includes a WISA module 814 and a WiFi dual band module 816, both of which provide wireless connectivity. Through its WISA module 814 or LAN/AVB port 802, multi-role speaker 800 may distribute digital audio in either of two forms: 16-bit, 44.1kHz form (used in Airplay 2), or higher fidelity 24-bit, 96kHz form.

Through its WiFi dual band module 816, multi-role speaker 800 provides support for Airplay 2 as well as serving as a wireless access point. To provide whole house audio, speakers are frequently distributed throughout most rooms or other areas of a house. By providing such speakers with wireless network connectivity so that they become multi-role speakers, robust wireless network coverage is attained without the necessity of separate wireless access points. Through its WISA module 814, multi-role speaker 800 may be paired with a WISA speaker 820 to provide stereo sound. Paired WISA speaker 820 includes a microcontroller 822 which is coupled to a WISA module 830 and a DSP 824. DSP 824 is coupled to an amplifier 826, which in turn is coupled to a speaker 828. In general, paired WISA speaker 820 receives digital audio from multi-role speaker 800 via WISA module 830. Within an automation environment 100 (Fig. 1), multi-role speaker 800 is capable of serving in any combination of four roles: audio speaker; multi-role automation host 104 (Fig. 1); wireless access point; and pairing with WISA speaker 820.

Fig. 9 shows a multi-role amplifier 900 which includes a microprocessor 902 coupled to a DSP 904 which in turn is coupled to an amplifier 906. Microprocessor 902 is coupled, respectively, to a LAN/A VB port 908, an analog input 910, a TosLink input 912, and control ports 914 which may include RS232, IR, or other control ports. An AC/DC power supply 916 provides power to multi-role amplifier 900. Multi-role amplifier 900 provides an RCA pre-amp output 920 and may be used to drive external passive speakers 918.

Within an automation environment 100 (Fig. 1), multi-role amplifier 900 is capable of serving as both an audio amplifier and a multi-role automation host 104 (Fig. 1).

The foregoing description has been directed to specific embodiments of this invention. It will be apparent, however, that other variations and modifications may be made to the described embodiments, with the attainment of some or all of their advantages. For example, it is expressly contemplated that the teachings of this invention can be implemented as software, including a computer-readable medium having program instructions executing on a computer, hardware, firmware, or a combination thereof. Accordingly this description is to be taken only by way of example and not to otherwise limit the scope of the invention. It is thus the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.

What is claimed is: