SPECTATORS EVENT

FIELD OF THE INVENTION

The present invention generally relates to controlling mobile devices and other peripherals via remote devices, and more specifically to a method for controlling mobile devices in a synchronized and coordinated fashion in a mass spectator event.

BACKGROUND OF THE INVENTION

Generally, spectacle events are limited to utilizing the audio and visual effects provided by the artists and/or operator of a venue holding the event. Traditionally, venue speakers, scoreboards, and screens are used to supplement a performer's and/or athlete's performance. Currently, there is a lack of additional multimedia interaction for spectators, and their personal mobile devices, to supplement the traditional audio/visual elements at the venue.

There is a need for providing enhanced experiences for the spectators attending spectacles, such as events having a range of attendees or audience members numbering from a few in a small performance room to stadium-filling crowds.

SUMMARY OF THE INVENTION

It is an object of the present invention to disclose a method of controlling multiple mobile electronic devices using a controller application, a portion of the mobile electronic devices do not support multicast, the method comprising transmitting commands to the mobile electronic devices via UDP, transmitting commands to the mobile electronic devices via mDNS, mobile applications installed in the multiple mobile electronic devices and interacts with the controller application listens to the mDNS and receives commands from the controller application via the mDNS.

In some cases, the method further comprises broadcasting the commands by the controller application. In some cases, the method further comprises executing the commands by the controller application by the mobile applications.

In some cases, the method further comprises downloading content from a content server prior to receiving the commands from the controller application. In some cases, the method further comprises the mobile application filters a command from a plurality of optional commands according to a physical location of the mobile electronic device.

It is another object of the present invention to disclose a method of synchronizing multiple mobile electronic devices using broadcasted messages, comprising multiple mobile electronic device receive multiple messages that contain commands in predefined intervals, said messages contain command code, sequence number, the current time of the controller and the expected time of executing the command; after receiving multiple messages, the mobile electronic device identifies two messages received at time difference closest to the predefined periods.

In some cases, the method further comprises determining a local time offset for each mobile electronic device according to the difference between subsequent messages. In some cases, the method further comprises executing commands from the controller according to the time offset of each mobile electronic device. In some cases, the commands are transmitted from the controller via multicast and mDNS IP address.

It is another object of the present invention to disclose a method for coordinating operation of multiple mobile electronic devices, comprising each of the multiple mobile electronic devices acquires its location; each of the multiple mobile electronic devices receives a map of an area in which the multiple mobile electronic devices are located; each of the multiple mobile electronic devices identifies its location in the map and identifies a command to be executed out of a plurality of optional commands received in the map. In some cases, the location of the multiple mobile electronic devices is acquired from a user input. In some cases, the location of the multiple mobile electronic devices is acquired via GPS. In some cases, the map creates an image by assigning different colors to different mobile electronic devices.

It is an object of the present invention to disclose a method of controlling a plurality of mobile electronic devices using a controller application which sends commands and data to a mobile application running on each of the plurality of mobile electronic devices using Multicast UDP protocol. The method overcomes the problem where a portion of mobile electronic devices does not support Multicast UDP by overloading the mDNS (Multicast Domain Name Server) protocol.

In some cases, the controller application sends information to the mobile application running on each of the plurality of mobile electronic devices and each mobile application obtains a translation of the information to the language used by the mobile electronic device. In some cases, the method further comprises broadcasting the commands and data from the controller application. In some cases, the method further comprises executing the commands of the controller application by the mobile applications.

In some cases, the method further comprises mobile applications downloading content from content server prior to receiving the commands from the controller application. In some cases the mobile application does not download the content but rather receives the content from the controller application in real-time to be processes on-the-fly, for example a streaming video or audio. In some cases, the data and commands sent from the controller application include encoded meta-data such as the location of the mobile device that is associated to the command and data and the mobile application filters the received commands and data based on its location in the venue.

It is another object of the present invention to disclose a method of synchronizing a plurality of mobile electronic devices using synchronization commands sent at a predefined interval period from the controller application. The synchronization commands comprising command code, sequence number, the current time of the controller application, and the time for executing the command. Each mobile application receives said synchronization commands and selects a set of synchronization commands closest to the predefined interval period and uses that set to determine the time offset between the controller application time and the mobile application time. It is another object of the present invention to disclose a method of coordinating operation of the plurality of mobile electronic devices, comprising of each of the plurality of mobile electronic acquiring its location; the mobile application received a mapping of its location to a group number. The group number is encoded in the header of data and commands received by the mobile application which filters the data and command to apply and execute the data and commands intended for its group. The group may be defined by several mobile devices located in a specific location.

A second coordination method of the present invention is disclosed comprising the same data sent to the plurality of mobile electronic devices along with a map of the venue enabling each mobile electronic device to extract its own portion of the overall data to be played and/or displayed, according to the mobile device location acquired independently by each mobile device.

It is another object of the present invention to disclose a method for enabling a MIDI controller to control a plurality of mobile electronic devices using the controller application. The method comprising of the controller application receiving the MIDI commands from the MIDI controller and converting them to commands configured to be sent to the plurality of mobile electronic devices.

It is another object of the present invention to disclose a method for enabling the controller application to control a plurality of MIDI enabled devices comprising of augmenting the controller application commands with MIDI commands and having the controller application send both types of commands to the plurality of mobile devices as well as MIDI commands to the plurality of MIDI enabled devices.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments of the invention are herein described by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of embodiments of the invention. In this regard, the description taken with the drawings makes apparent to those skilled in the art how embodiments of the invention may be practiced.

In the drawings:

Fig 1 discloses a system for controlling multiple devices in concert during a mass spectator event, according to exemplary embodiments of the present invention;

Fig. 2A discloses a method of preparing multiple devices to operate in concert during a mass spectator event, according to exemplary embodiments of the present invention;

Fig 2B discloses a method for synchronizing the actions of the computerized mobile devices, according to an exemplary embodiment of the present invention;

Fig. 3 shows a method of employing multiple computerized mobile devices for streaming media in concert, according to exemplary embodiments of the present invention;

Fig. 4 discloses a method for coordinating the actions of a plurality of computerized mobile devices based on their specific location within the perimeter of the venue holding the event, according to exemplary embodiments of the present invention;

Fig. 5 discloses a secondary method for coordinating a plurality of mobile devices where they all receive the same data and each mobile device will play or display a portion of data according to its physical location in the venue;

Fig. 6 discloses a method of enabling a MIDI controller to control a plurality of mobile devices by sending MIDI commands to the controller application which, in turn, converts the MIDI commands to command for the mobile electronic devices;

Fig. 7 discloses a method of enabling the controller application to control a plurality of MIDI enabled devices and a plurality of mobile electronic devices simultaneously. DESCRIPTION OF THE INVENTION

The present invention discloses a system and a method for controlling multiple computerized mobile devices in a mass spectator event, such as a music show, sporting event, political event and the like. The method disclosed in the present invention enables a remote computerized device operating a controller application to send commands and data to multiple computerized mobile devices. The commands sent by the remote device may by such as device vibrate, turn on/off the camera flash, illuminate the display unit, and the like. The data sent by the remote device may by such as video stream, audio stream, images, text. The computerized mobile devices may operate a mobile application in order to receive and manage the communication with the remote device. The mobile application can execute the commands and utilize the data received by the computerized mobile devices.

The present invention also discloses a method of sending data and commands using UDP Multicast protocol by overloading an existing UDP multicast service known as mDNS (Multicast Domain Name Server) in such a way as to get the data across to computerized mobile devices that do not support UDP Multicast protocol. All Mobile electronic devices support the mDNS protocol, even those mobile electronic devices that do not support Multicast UDP otherwise. By defining our data and command set in such a way as to not interfere with the mDNS protocol, we are able to use the mDNS channel to transmit data and commands from the controller application to the plurality mobile electronic devices. The mDNS server expects a certain structure in the packets to be received. Packets that do not contain the correct structure are discarded by the mDNS server. The controller application sends command to the plurality of mobile devices that do not contain the expected header and are thus discarded by the mDNS server. The mobile application is listening for commands on the same IP address as the mDNS server (224.0.0.251:5353). The mobile application, however, is programmed to recognize the commands created by the controller application and discard other commands such as DNS requests from other devices.

The present invention also discloses a method of synchronizing computerized mobile devices in order to perform specific actions simultaneously such as playing a media file, vibrating the mobile device, turning on or off the camera light of a mobile device, playing a particular sound, displaying a text and the like. In some cases, the computerized mobile device synchronization may be based on the physical location of the computerized mobile devices in the venue holding the event. In some other cases, the computerized mobile device synchronization may be based on information encoded by the controller application to the commands or to the data sent to computerized mobile devices.

There are two scenarios for playing media (audio/video) using the mobile electronic devices. In the first scenario, the media file is downloaded to the device before it is played. The entire audio/video file is downloaded from the content server to the device any time before the event (does not have to be at the venue). Thus, the media file resides on the mobile electronic device and a command from the controller application is send to select the file to be played. Then, the set of synchronization commands are given so that the mobile devices can synchronize. Finally the "Start" command is sent to the devices to start playing the selected media file.

In the second scenario, no media file exists on the device: The media is not downloaded from the content server. Instead, the media data is streamed from the controller application via UDP multicasting to all the devices in the venue.

In both scenario, for synchronization purposes: Exactly the same set of synchronization commands is sent to the mobile applications so that the mobile application can compute the correct time to start playing the streamed data (that is buffered). In both scenarios, for coordination purposes: The streamed data is composed of frames. Each frame has a header which contains (in addition to the standard information like bit-rate, compression info), the Funtoad specific metadata like group number, LCD action, vibration duration, and the like.

Fig 1 discloses a system for controlling multiple devices in concert in a mass spectator event, according to exemplary embodiments of the present invention. The system is designed to maintain a plurality of communications with computerized mobile devices in order to send them content and commands. Fig 1 shows a content server 120 designed for populating content and configuration data to the computerized mobile devices. The content server 120 communicates with the mobile application 125, via communication channel 132 in order to send content. The communication channel 132 may be any standard wireless communication network. The content communicated by the content server 120 may comprise of configuration data, video files, sound files, text files, image files and the like. The system also comprises controller application 105 that communicates with the mobile applications 125 via communication channel 131 in order to send commands and data to the computerized mobile devices at the venue. The commands sent by the system may include commands such as: Play media content such as videos or sounds, play a video and audio streams, display text message, send text translation, display the screen or other display unit in a specific color, vibrate the mobile device, and the like. For example, a mobile application 125 communicating with content server 120 via communication channel 132 and with controller application 105 via communication channel 131, in order to receive content related to a specific event. The controller application 105 sends a command to play a video stream downloaded from content server 120. The mobile application 125 receives the command, and plays the video stream. In some exemplary embodiment of the present invention the controller application 105 may send to computerized mobile device some operational commands such as: Vibrate the device, turn the camera light on or off, turn the LED on or off, obtain GPS location, record from the device's microphone, allow a specific device to send data to the other peripherals at the venue, change the default language of the device, and the like. For example, the controller application 105 sends a command for vibrating the computerized mobile devices in a specific time at the event, to all computerized mobile devices in the venue holding the event. The mobile application operating on the computerized mobile devices receive the command and simultaneously vibrate the computerized mobile device. Fig 1 , also comprises content server database 115 in order to store information related to the mobile computerized mobile devices communicating with the content server 120. The content stored in the server database 115 may comprise data and information such as: computerized mobile devices' unique identifiers, geographical location of each particular computerized mobile device, and the like. In some optional embodiments of the present invention the server database 115 may be independent and located in a physical device other than the content server 120. The server database 115 located in different device may also be hosted by an independent operation system and communicate with the content server 120 via standard communication process and method exist in the industry

Fig. 2A discloses a method of preparing multiple devices to operate in concert in a mass spectator event, according to exemplary embodiments of the present invention. In step 205, a person operating a computerized mobile device launches a mobile application and receives configuration data prepared for a specific event. The configuration data received by the computerized mobile devices may comprise multiple parameters such as: List of video files used in the event, list of audio files used in the event, list of image files used in the event, list of translation text files used in the event, the network service set identifier also known as SSID of the venue holding the event, a list of group numbers, and the like. The mobile application operating on the computerized mobile devices is also designed to receive the configuration data from a remote location, from any network designed to provide file hosting and communication services. For example, a person intending to attend an event can receive the configuration settings in advance using any internet service.

In step 210, the mobile application operates on the computerized mobile devices interprets the configuration data for utilizing the information. Such interpretation performed by the mobile application may be parsing action, read a data file and the like and downloads the various video files, audio files, image files, text files, and other files specified in the configuration data from the content servers. The content server can reside locally at the venue or in another location with internet connectivity. In some cases, the mobile application may send information or data to the content server such as location of the mobile device in the venue represented by a seat number, a section number, and the like, language used by the mobile device, selections made within the mobile device application, device type, and the like.

Step 215 discloses the controller application, operated by a person controlling the mobile devices, sending commands utilizing a Multicast Domain Name Server IP address also known as mDNS. The mDNS IP address is originally designed to resolve host names to Internet Protocol addresses, also known as IP addresses, within small networks that do not include a local domain name server. Thus, the controller application and mobile devices are programmed to utilize the familiar mDNS channel for communications. The commands received by the mobile application may comprise actions such as play a video file or play a sound file which was downloaded as disclosed in step 210. In some cases, the command may also comprise actions such as display the device screen or display unit in a particular color, vibrate the device, and the like. In some cases, the command may be executed when received. In some other cases, the command may be associated with a timestamp to execute in order to execute the command. The timestamp may be any readable time value which the mobile application can utilize. In some other cases, the time value can also contain the phrase "now", for executing when received. In some cases, the commands received by the mobile application can be multiple commands in a sequence with a timestamp attached to each command. The mobile application receives a command sequence with timestamp can execute each command based on its timestamp.

Step 225 discloses sending commands to the computerized mobile devices, via a utilization of standard multicast method based on User Datagram Protocol (also known as UDP). The commands are sent by the controller application are identical to the commands and information sent via mDNS and disclosed in step 215. In step 220, mobile devices are configured to receive data on mDNS IP address. Similarity, in step 230, the computerized mobile devices receive the commands sent from the controller application via UDP Multicast or Broadcast on any valid Multicast or Broadcast address other than mDNS IP Address. For example, in case Internet Protocol version 4, also known as IPv4 is used, address 224.0.0.251:5353 is the IP address of the mDNS. In step 235, all devices receive the commands and process then to take actions like play a video stream, light the screen, and the like.

Fig 2B discloses a method for synchronizing the actions of the computerized mobile devices, according to an exemplary embodiment of the present invention. In step 245, the controller application sends sequential synchronization commands. Each command in the sequence contains the command code, the sequence number, the controller application current- time, and the time for executing the command. The period between the commands in the sequence is constant and predefined. The controller application and mobile application are designed and configured to utilize identical periods. The controller application sends sequential commands to the computerized mobile devices, in the predefined periods. In step 250, the mobile applications receive the commands associated with the controller application times, and then add to each command the local timestamp of the computerized mobile device receiving time. For example, a command received by the mobile application comprises the command code, sequence number, controller application system time, and the time for executing the command. In step 255, the mobile application identifies the commands and their associated timestamps which are closest to the period times. For example, when the controller application intervals were 300 milliseconds between the commands, the mobile application measures the period between the commands according to the computerized mobile device's timestamp and identifies the duration which is the closest to 300 milliseconds (or a multiple thereof). In step 260, the mobile application uses the controller application time and the timestamp of the computerized mobile device of one of the commands identified in step 255 and subtracts the timestamp of the computerized mobile device from the controller application time. Then, the mobile application marks the result of the subtraction operation as time offset. In step 265, the mobile application utilizes the time offset to calculate the actual execution time of the commands received from the application controller. For example, in case the time offset is 900 milliseconds, the mobile application will take the command Start Time from any one of the synchronization commands; add 900 milliseconds to it to get time to start the command relative to its own clock. In some exemplary cases, each synchronization command has the command start time relative to the controller time, not relative to the mobile device time. So, after the mobile application calculates the offset, it adds the calculated offset to the mobile device time to obtain the start time relative to the mobile device.

Fig. 3 shows a method of employing multiple computerized mobile devices for streaming media in concert, according to exemplary embodiments of the present invention. In step 305, a controller application, operated in the venue holding the event, sends a command to start a media streaming to the computerized devices in the venue holding the event. The media streaming can be a video file, an audio file and the like. In some cases, the commands sent to the computerized mobile devices can contain commands to download files such as video files, digital image files, audio files and the like. In step 310, the computerized mobile devices in the venue holding the event receive the commands and identifies the time value for execution of the command. In Step 315, computerized mobile devices execute the command according to the received time value and the time offset as disclosed in Fig 2, step 225. For example, a computerized mobile device receives a command to start a video streaming in a certain time. The computerized mobile device utilizes the time offset as disclosed in Fig 2, step 225 to acquire the accurate time for starting to play the streamed media data. In step 320, the computerized mobile devices in the venue hosting the event start playing the video simultaneously. In some cases, during a video streaming process additional media files can be downloaded to the mobile device.

Fig. 4 discloses a method for coordinating the actions of a plurality of computerized mobile devices based on their specific location within the perimeter of the venue holding the event, according to exemplary embodiments of the present invention. In step 405 the computerized mobile devices obtain their specific physical location in the venue holding the event. The location of the computerized mobile device may be the seat number entered by the person who operates the computerized mobile device, or the mobile application can compute the location based on beacons placed throughout the venue, or by some other method. In step 410, using the configuration data downloaded in step 205, the mobile application determines the GN (Group Number) for the specific media file or action. The GN values may not be unique. For example, all devices in rows 1, 3, 5, 7, 9 may have GN=1. And all devices in rows 2, 4, 6, 8, 10 may have GN = 2.

Step 415, discloses encoding the group numbers (GNs) within the header part of the frames composing the media stream. Some media standard, for example, AVI File Format, is a container format that enables the addition of application specific meta-data into the frame headers. Adding the GN into the frame header allows for mapping of commands and data to specific plurality of mobile electronic devices. In step 420, each mobile electronic device which receives the media stream can utilize the GNs encoded to the video streaming or in the commands to identify the specific data required to be used or displayed by the device.

In step 425 the controller application sends commands to computerized mobile devices displaying a video file. The commands are associated with multiple GN's represent specific physical location in the venue hosting the event. The computerized mobile devices identified by the GN's associated with the command may execute the commands. Commands such as: Play a video file, stop the streaming, light the screen in a specific color and the like.

Fig. 5 discloses a secondary method for coordinating a plurality of mobile devices where they all receive the same data and each mobile device will play or display a portion of data according to its physical location in the venue. The data can be an image, video, sound, and the like. For example, all mobile devices in a rectangular venue can receive an image of the American flag. Each mobile device, based on its location, will display the small portion of the entire image whereby the plurality of mobile devices in the venue collectively display the entire American flag to within the resolution allowed by the number and density of mobile devices. In step 505 the controller application sends a command to download an image file of the Italian flag. The computerized mobile devices received the command may identify the location of the file. The file location may be a content server located within the local network, an image located in a server on the internet, a server introduces an FTP service also known as File Transfer Protocol and the like. In step 510, the computerized mobile devices identify their relative location in the venue holding the event, as disclosed in Fig 4. The locations identified by the computerized mobile devices may be the seat number or specific physical location. In step 520, the computerized mobile devices light the screens at the color of the flag corresponds to their physical location.

Fig. 6 discloses a method of allowing a MIDI (Musical Instrument Digital Interface) controller to control a plurality of mobile devices. A MIDI controller refers to any hardware or software that generates and transmits MIDI data. MIDI is a technical standard that describes a protocol, digital interface, and connectors and allows a wide variety of electronic musical instruments, computers, and other related devices to connect and communicate with one another. In step 605, the controller application is programmed to translate specific MIDI commands to specific commands for the mobile application running on the mobile devices. For example, the MIDI command "193 21" to change the program sound on MIDI channel 2 to program #21, can be programmed to vibrate the mobile devices for 21 milliseconds. In step 610, in addition to the existing connections of the MIDI controller to the MIDI enabled devices, a new connection to the controller application is established. In step 615, the MIDI controller sends commands to the MIDI enabled devices, one of which is the controller application. In step 620, the controller application processes the MIDI command and converts it to one of many commands for the mobile devices running the mobile application. For example, when the MIDI controller sends the MIDI command "193 21", the MIDI command is received by the MIDI enabled devices, including the controller application. The controller application is programmed, in this example, to send a command to the mobile devices to vibrate for 21 milliseconds. In step 625, the mobile application received the command to vibrate for 21 milliseconds and commands the mobile device to vibrate for 21 milliseconds.

Fig. 7 discloses a method for enabling the controller application to control MIDI enabled devices in addition to controlling the mobile devices. In step 705, the controller application commands are augmented by the addition of MIDI commands. For example, when the control application operator presses the button to send the command to illuminate the mobile devices' display the color blue, the controller sends "C.Blue" to the plurality of mobile electronic devices. In addition, the controller application will also send the appropriate MIDI command to the MIDI enabled devices to play a specific chord. In step 710, in addition to the existing connection to the MIDI enabled devices, another connection is made between the MIDI enabled devices and the controller application. In step 715, whenever the operator of the controller application selects the "Illuminate Blue" command, the mobile applications will receive that command and illuminate their display the color blue. In step 720, the MIDI enabled devices receive the MIDI commands and play the specific chord commanded by the controller application.

While the disclosure has been described with reference to exemplary embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings without departing from the essential scope thereof. Therefore, it is intended that the disclosed subject matter not be limited to the particular embodiment disclosed as the best mode contemplated for carrying out this invention, but only by the claims that follow.