Login| Sign Up| Help| Contact|

Patent Searching and Data


Title:
SYSTEMS AND METHODS FOR CONTROLLING VEHICLES WITH VOICE COMMANDS
Document Type and Number:
WIPO Patent Application WO/2024/107403
Kind Code:
A1
Abstract:
Methods and systems for controlling vehicles through voice commands are disclosed. Voice commands are received locally at the vehicle and/or remotely at a remote station. An exemplary method of controlling a vehicle with a voice command can comprise: identifying an operation mode of the vehicle; receiving an audio input from an audio input device; receiving an environmental input from an environmental sensor input device; comparing the audio input to a first plurality of possible voice commands to determine a first command; validating whether the first command is allowable based on the identified operating mode and the environmental input; and controlling a machine control system in response to the first command when the first command is allowable. A control system can comprise a processor and an audio input device disposed on a vehicle, and/or a remote station and is configured to perform the methods described.

Inventors:
MCMICKELL MICHAEL BRETT (US)
KREIDER THOMAS (US)
Application Number:
PCT/US2023/037198
Publication Date:
May 23, 2024
Filing Date:
November 13, 2023
Export Citation:
Click for automatic bibliography generation   Help
Assignee:
CIRCLIO INC (US)
International Classes:
B60R16/00; G05D1/22; G05D13/00; G06F3/16; G10L15/22
Foreign References:
US20200278679A12020-09-03
US20200372908A12020-11-26
US20220101847A12022-03-31
US20200398842A12020-12-24
US20140136187A12014-05-15
US20210081683A12021-03-18
Attorney, Agent or Firm:
WOLF, Tyler et al. (US)
Download PDF:
Claims:
CLAIMS

1. A control system onboard a vehicle, comprising: a non-transitory computer-readable storage medium in electronic communication with an audio input device and a machine control system, having instructions stored thereon that, in response to execution by a processor, cause the processor to perform operations comprising: identifying, by the processor, an operation mode of the vehicle, resulting in an identified operating mode, receiving, by the processor, an audio input from the audio input device in communication with the processor, receiving, by the processor, an environmental input from an environmental sensor input device in communication with the processor, comparing, by the processor, the audio input to a first plurality of possible voice commands to determine a first command, validating, by the processor, whether the first command is allowable based on the identified operating mode and the environmental input, and controlling, by the processor, the machine control system in response to the first command when the first command is allowable.

2. The control system of claim 1, further comprising filtering, by the processor, the audio input based on at least one of the identified operating mode of the vehicle and the environmental input.

3. The control system of claim 2, further comprising, receiving, by the processor, a triggering event comprising at least one of a wake word, a start button signal, and a start signal.

4. The control system of claim 3, further receiving, by the processor, a confirmatory response through the audio input device, a visual display, a start button, or a network message.

5. The control system of claim 4, wherein the identified operating mode comprises a speed of the vehicle.

6. The control system of claim 5, wherein the environmental sensor input device comprises a GPS locator in electronic communication with the processor and configured to output a GPS location.

7. The control system of claim 6, wherein the environmental sensor input device comprises a LiDAR sensor in electronic communication with the processor and configured to output a terrain output.

8. The control system of claim 7, wherein the validating further comprises comparing, by the processor, the first command to the speed of the vehicle.

9. The control system of claim 8, wherein the validating further comprises comparing, by the processor, the first command to the GPS location of the vehicle.

10. The control system of claim 9, wherein the operations further comprise controlling, by the processor, at least one of the visual display, a speaker, and a haptic device to output an alert in response to at least one of the audio input or the first command.

11. An article of manufacture including a tangible, non-transitory computer-readable storage medium in electronic communication with an audio input device and a machine control system, having instructions stored thereon that, in response to execution by a processor, cause the processor to perform operations comprising: identifying, by the processor, an operation mode of a vehicle, resulting in an identified operating mode; receiving, by the processor, an audio input from the audio input device in communication with the processor; receiving, by the processor, an environmental input from an environmental sensor input device in communication with the processor; determining, by the processor, whether the audio input is a first command of a first plurality of possible voice commands; validating, by the processor, whether the first command is allowable based on the identified operating mode and the environmental input; and controlling, by the processor, the machine control system in response to the first command when the first command is allowable.

12. The article of manufacture of claim 11, wherein the operations further comprise filtering, by the processor, the audio input based on at least one of the identified operating mode of the vehicle and the environmental input.

13. The article of manufacture of claim 12, wherein the operations further comprise receiving, by the processor, a triggering event comprising at least one of a wake word, a start button signal, and a start signal.

14. The article of manufacture of claim 13, wherein the environmental sensor input device comprises a GPS locator in electronic communication with the processor and configured to output a GPS location.

15. The article of manufacture of claim 14, wherein the validating further comprises comparing, by the processor, the first command to a speed of the vehicle.

16. The article of manufacture of claim 15, wherein validating further comprises comparing, by the processor, the first command to the GPS location of the vehicle.

17. The article of manufacture of claim 16, wherein the operations further comprise controlling, by the processor, at least one of a visual display, a speaker, and a haptic device to output an alert in response to at least one of the audio input or the first command.

18. A method of operating a vehicle, comprising: identifying, by a processor, an operation mode of the vehicle, resulting in an identified operating mode; receiving, by the processor, an audio input from an audio input device in communication with the processor; receiving, by the processor, an environmental input from an environmental sensor input device in communication with the processor; determining, by the processor, whether the audio input is a first command of a first plurality’ of possible voice commands; validating, by the processor, whether the first command is allowable based on the identified operating mode and the environmental input; and controlling, by the processor, a machine control system in response to the first command when the first command is allowable.

19. The method of claim 18, further comprising filtering, by the processor, the audio input based on at least one of the identified operating mode of the vehicle and the environmental input.

20. The method of claim 19, further comprising, receiving, by the processor, a triggering event comprising at least one of a wake word, a start button signal, and a start signal.

21. The method of claim 20, further receiving, by the processor, a confirmatory response through the audio input device, a visual display, a start button, or a network message.

22. The method of claim 21, wherein the environmental sensor input device comprises a GPS locator in electronic communication with the processor and configured to output a GPS location.

23. The method of claim 22, wherein the validating further comprises comparing, by the processor, the first command to a speed of the vehicle.

24. The method of claim 23, wherein the validating further comprises comparing, by the processor, the first command to the GPS location of the vehicle.

25. The method of claim 24, wherein the vehicle comprises an interior cabin comprising at least one of the visual display, a speaker, and a haptic device in electronic communication with the processor, wherein the method further comprises controlling, by the processor, at least one of the visual display, the speaker, and the haptic device to output an alert in response to at least one of the audio input or the first command.

26. A vehicle control system, comprising: a cloud system comprising a cloud database and a cloud processor; and a non-transitory computer-readable storage medium in electronic communication with an audio input device, the cloud system, and a machine control system, having instructions stored thereon that, in response to execution by a processor, cause the processor to perform operations comprising: identifying, by the processor, an operation mode of a vehicle, resulting in an identified operating mode, receiving, by the processor, an audio input from the audio input device in communication with the processor, receiving, by the processor, an environmental input from an environmental sensor input device in communication with the processor, transmitting, by the processor, the identified operating mode of the vehicle, the audio input and the environmental input to the cloud processor via a network, receiving, by the processor, a first command from the cloud processor, and controlling, by the processor, the machine control system in response to the first command.

27. The control system of claim 26, further comprising filtering, by the processor, the audio input based on at least one of the identified operating mode of the vehicle and the environmental input.

28. The control system of claim 27, wherein the cloud system is configured to filter, by the cloud processor, the audio input based on the identified operating mode of the vehicle.

29. The control system of claim 28, wherein the cloud system is configured to determine, by the cloud processor, whether the audio input is the first command of a first plurality of possible voice commands stored in the cloud database.

30. The control system of claim 29, wherein the cloud processor is configured to validate whether the first command is allowable based on the identified operating mode and at least one environmental input. 31. The control system of claim 30, wherein the validating further comprises comparing, by the cloud processor, the first command to a speed of the vehicle.

Description:
TITLE: SYSTEMS AND METHODS FOR CONTROLLING VEHICLES

WITH VOICE COMMANDS

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Application Serial No. 63/425,905. filed November 16, 2022. and titled “SYSTEMS AND METHODS FOR CONTROLLING VEHICLES WITH VOICE COMMANDS ’ (hereinafter the ‘905 Application). The ‘905 Application is hereby incorporated by reference in its entirety for all purposes.

FIELD

[0002] The disclosure generally relates to controlling vehicles, and, more specifically to methods and systems for locally and/or remotely operating one or more vehicles using voice commands.

BACKGROUND

[0003] Most off-road vehicles used in the agriculture, mining, and construction industries are controlled by onboard human operators. Human operators are trained to perform potentially hazardous tasks safely and efficiently with these off-road vehicles. However, the operation of off-road vehicles is becoming increasingly complex as more automation and innovation is introduced. The complexity of skillful operation and continuous situational awareness drives a need for improved human-machine interfaces. There is a need to make these off-road vehicles more user-friendly for operators and develop safety features, which maintain the off-road vehicle remains in a safe mode of operation for both the off-road vehicle and the operator.

SUMMARY [0004] A control system onboard a vehicle is disclosed herein. In various embodiments the control system comprises a non-transitory computer-readable storage medium in electronic communication with an audio input device and a machine control system, having instructions stored thereon that, in response to execution by a processor, cause the processor to perform operations. In various embodiments the operations comprise: identifying an operation mode of the vehicle, resulting in an identified operating mode; receiving, by the processor, an audio input from the audio input device in communication with the processor; receiving, by the processor, an environmental input from an environmental sensor input device in communication with the processor; comparing, by the processor, the audio input to a first plurality of possible voice commands to determine a first command; validating, by the processor, whether the first command is allowable based on the identified operating mode and the environmental input; and controlling, by the processor, the machine control system in response to the first command when the first command is allowable. In various embodiments, the operations further comprise recognizing, by the processor, a wake word or an external start signal.

[0005] An article of manufacture is disclosed herein. In various embodiments, the article of manufacture comprises a tangible, non-transitory computer-readable storage medium in electronic communication with an audio input device and a machine control system, having instructions stored thereon that, in response to execution by a processor, cause the processor to perform operations comprising: identifying, by the processor, an operation mode of a vehicle, resulting in an identified operating mode; receiving, by the processor, an audio input from the audio input device in communication with the processor; receiving, by the processor, an environmental input from an environmental sensor input device in communication with the processor; determining, by the processor, whether the audio input is a first command of a first plurality of possible voice commands; validating, by the processor, whether the first command is allowable based on the identified operating mode and the environmental input; and controlling, by the processor, the machine control system in response to the first command when the first command is allowable. In various embodiments, the article of manufacture can be used along with any of the systems or methods described herein.

[0006] A method of operating a vehicle is described herein. In various embodiments, the method comprises: identifying, by the processor, an operation mode of a vehicle, resulting in an identified operating mode; receiving, by the processor, an audio input from the audio input device in communication with the processor; receiving, by the processor, an environmental input from an environmental sensor input device in communication with the processor; determining, by the processor, whether the audio input is a first command of a first plurality of possible voice commands; validating, by the processor, whether the first command is allowable based on the identified operating mode and the environmental input; and controlling, by the processor, the machine control system in response to the first command when the first command is allowable. In various embodiments, the method can be used with any of the systems disclosed herein.

[0007] A vehicle control system is disclosed herein. In various embodiments the vehicle control system comprises a cloud system comprising a cloud database and a cloud processor, and a non-transitory computer-readable storage medium in electronic communication with an audio input device, the cloud system and a machine control system, having instructions stored thereon that, in response to execution by a processor, cause the processor to perform operations. In various embodiments the operations comprise: identifying, by the processor, an operation mode of a vehicle, resulting in an identified operating mode; receiving, by the processor, an audio input from the audio input device in communication with the processor; receiving, by the processor, an environmental input from an environmental sensor input device in communication with the processor; transmitting, by the processor, the identified operating mode of the vehicle, the audio input and the environmental input to the cloud processor via a network; receiving, by the processor, a first command from the cloud processor; and controlling, by the processor, the machine control system in response to the first command. In various embodiments, the operations further comprise recognizing, by the processor, a wake word or an external start signal.

[0008] An article of manufacture is disclosed herein. In various embodiments, the article of manufacture comprises a tangible, non-transitory computer-readable storage medium in electronic communication with an audio input device, a cloud system and a machine control system, having instructions stored thereon that, in response to execution by a processor, cause the processor to perform operations comprising: identifying, by the processor, an operation mode of a vehicle, resulting in an identified operating mode; receiving, by the processor, an audio input from the audio input device in communication with the processor; receiving, by the processor, an environmental input from an environmental sensor input device in communication with the processor; transmitting, by the processor, the identified operating mode of the vehicle, the audio input and the environmental input to the cloud processor via a netw ork; receiving, by the processor, a first command from the cloud processor; and controlling, by the processor, the machine control system in response to the first command. In various embodiments, the article of manufacture can be used along with any of the systems or methods described herein.

[0009] A method of operating a vehicle is described herein. In various embodiments, the method comprises: identifying, by the processor, an operation mode of a vehicle, resulting in an identified operating mode; receiving, by the processor, an audio input from the audio input device in communication with the processor; receiving, by the processor, an environmental input from an environmental sensor input device in communication with the processor; transmitting, by the processor, the identified operating mode of the vehicle, the audio input and the environmental input to the cloud processor via a network; receiving, by the processor, a first command from the cloud processor; and controlling, by the processor, the machine control system in response to the first command. In various embodiments, the method can be used with any of the systems disclosed herein.

[0010] A vehicle control system is disclosed herein. In various embodiments the vehicle control system comprises a processor and a cloud system comprising a cloud database and a non-transitory computer-readable storage medium in electronic communication with an audio input device and the processor, having instructions stored thereon that, in response to execution by a cloud processor, cause the cloud processor to perform operations. In various embodiments the operations comprise: receiving, by the cloud processor, an identified operating mode of a vehicle, an audio input and an environmental input via a network; determining, by the cloud processor, whether the audio input is a first command of a first plurality of possible voice commands stored in the cloud database; validating, by the cloud processor, whether the first command is allowable based on the identified operating mode and the environmental input; and sending, by the cloud processor, the first command to the processor. In various embodiments, the operations further comprise recognizing, by the cloud processor, a wake word or an external start signal.

[0011] An article of manufacture is disclosed herein. In various embodiments, the article of manufacture comprises a tangible, non-transitory computer-readable storage medium in electronic communication with an audio input device and a processor, having instructions stored thereon that, in response to execution by a cloud processor, cause the cloud processor to perform operations comprising: receiving, by the cloud processor, an identified operating mode of a vehicle, an audio input and an environmental input via a network; determining, by the cloud processor, whether the audio input is a first command of a first plurality of possible voice commands stored in the cloud database; validating, by the cloud processor, whether the first command is allowable based on the identified operating mode and the environmental input; and sending, by the cloud processor, the first command to the processor. In various embodiments, the article of manufacture can be used along with any of the systems or methods described herein.

[0012] A method of operating a vehicle is described herein. In various embodiments, the method comprises: receiving, by the cloud processor, an identified operating mode of a vehicle, an audio input and an environmental input via a network; determining, by the cloud processor, whether the audio input is a first command of a first plurality of possible voice commands stored in the cloud database; validating, by the cloud processor, whether the first command is allowable based on the identified operating mode and the environmental input; and sending, by the cloud processor, the first command to the processor. In various embodiments, the method can be used with any of the systems disclosed herein.

[0013] A vehicle remote control system is disclosed herein. In various embodiments the vehicle remote control system comprises a first vehicle and a remote controller comprising a non-transitory computer-readable storage medium in electronic communication with an audio input device and a processor, having instructions stored thereon that, in response to execution by a remote processor, cause the remote processor to perform operations. In various embodiments the operations comprise: receiving, by the remote processor, an identified operating mode of the first vehicle, an audio input and an environmental input via a remote network; determining, by the remote processor, whether the audio input is a first command of a first plurality of possible voice commands stored in the remote controller; validating, by the remote processor, whether the first command is allowable based on the identified operating mode and the environmental input; and transmitting, by the remote processor, the first command to the processor to control the first vehicle. In various embodiments, the operations further comprise recognizing, by the remote processor, a wake word or an external start signal.

[0014] An article of manufacture is disclosed herein. In various embodiments, the article of manufacture comprises a tangible, non-transitory computer-readable storage medium in electronic communication with an audio input device and a processor, having instructions stored thereon that, in response to execution by a remote processor, cause the remote processor to perform operations comprising: receiving, by the remote processor, an identified operating mode of the first vehicle, an audio input and an environmental input via a remote network; determining, by the remote processor, whether the audio input is a first command of a first plurality of possible voice commands stored in the remote controller; validating, by the remote processor, whether the first command is allowable based on the identified operating mode and the environmental input; and transmitting, by the remote processor, the first command to the processor to control the first vehicle. In various embodiments, the article of manufacture can be used along with any of the systems or methods described herein.

[0015] A method of remotely operating a vehicle is described herein. In various embodiments, the method comprises: receiving, by the remote processor, an identified operating mode of the first vehicle, an audio input and an environmental input via a remote netw ork; determining, by the remote processor, whether the audio input is a first command of a first plurality of possible voice commands stored in the remote controller; validating, by the remote processor, whether the first command is allowable based on the identified operating mode and the environmental input; and transmitting, by the remote processor, the first command to the processor to control the first vehicle. In various embodiments, the method can be used with any of the systems disclosed herein.

[0016] A vehicle remote control system is disclosed herein. In various embodiments the vehicle remote control system comprises a remote controller comprising a remote processor; and a first vehicle comprising a non-transitory computer-readable storage medium in electronic communication with a machine control system, having instructions stored thereon that, in response to execution by a processor, cause the processor to perform operations. In various embodiments the operations comprise: identifying, by the processor, an operation mode of the first vehicle, resulting in an identified operating mode; receiving, by the processor, an environmental input from an environmental sensor input device in communication with the processor; transmitting, by the processor, the identified operating mode of the first vehicle and the environmental input to the remote processor; receiving, by the processor, a first command from the remote processor, and controlling, by the processor, the machine control system of the first vehicle in response to the first command. In various embodiments, the operations further comprise recognizing, by the processor, a wake word or an external start signal.

[0017] An article of manufacture is disclosed herein. In various embodiments, the article of manufacture comprises a tangible, non-transitory computer-readable storage medium in electronic communication with a machine control system, having instructions stored thereon that, in response to execution by a remote processor, cause the remote processor to perform operations comprising: identifying, by the processor, an operation mode of the first vehicle, resulting in an identified operating mode; receiving, by the processor, an environmental input from an environmental sensor input device in communication with the processor; transmitting, by the processor, the identified operating mode of the first vehicle and the environmental input to the remote processor; receiving, by the processor, a first command from the remote processor, and controlling, by the processor, the machine control system of the first vehicle in response to the first command. In various embodiments, the article of manufacture can be used along with any of the systems or methods described herein.

[0018] A method of remotely operating a vehicle is described herein. In various embodiments, the method comprises: identifying, by the processor, an operation mode of the first vehicle, resulting in an identified operating mode; receiving, by the processor, an environmental input from an environmental sensor input device in communication with the processor; transmitting, by the processor, the identified operating mode of the first vehicle and the environmental input to the remote processor; receiving, by the processor, a first command from the remote processor, and controlling, by the processor, the machine control system of the first vehicle in response to the first command. In various embodiments, the method can be used with any of the systems disclosed herein.

[0019] The forgoing features and elements may be combined in various combinations without exclusivity, unless expressly indicated herein otherwise. These features and elements as well as the operation of the disclosed embodiments will become more apparent in light of the following description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The subject matter of the present disclosure is particularly pointed out and distinctly claimed in the concluding portion of the specification. A more complete understanding of the present disclosures, however, may best be obtained by referring to the detailed description and claims when considered in connection with the drawing figures, wherein like numerals denote like elements.

[0021] FIG. 1 illustrates an off-road vehicle, in accordance with various embodiments; [0022] FIG. 2 illustrates a further off-road vehicle, in accordance with various embodiments;

[0023] FIG. 3 illustrates a still further off-road vehicle, in accordance with various embodiments;

[0024] FIG. 4 illustrates an interior cabin of an off-road vehicle, in accordance with various embodiments;

[0025] FIG. 5 illustrates a block diagram for a vehicle control system, in accordance with various embodiments;

[0026] FIG. 6 illustrates a block diagram for a vehicle control system, in accordance with various embodiments;

[0027] FIG. 7 illustrates a block diagram for a vehicle control system, in accordance with various embodiments;

[0028] FIG. 8 illustrates a method for controlling an off-road vehicle, in accordance with various embodiments;

[0029] FIG. 9 illustrates a method for controlling an off-road vehicle, in accordance with various embodiments;

[0030] FIG. 10 illustrates a method for controlling an off-road vehicle, in accordance with various embodiments

[0031] FIG. 11 illustrates a method for controlling an off-road vehicle, in accordance with various embodiments;

[0032] FIG. 12 illustrates a method for controlling an off-road vehicle, in accordance with various embodiments.

DETAILED DESCRIPTION

[0033] The detailed description of exemplary embodiments herein makes reference to the accompanying drawings, which show exemplary embodiments by way of illustration and their best mode. While these exemplary embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosures, it should be understood that other embodiments may be realized and that logical, chemical, and mechanical changes may be made without departing from the spirit and scope of the disclosures. Thus, the detailed description herein is presented for purposes of illustration only and not of limitation. For example, the steps recited in any of the method or process descriptions may be executed in any order and are not necessarily limited to the order presented. Furthermore, any reference to singular includes plural embodiments, and any reference to more than one component or step may include a singular embodiment or step. Also, any reference to attached, fixed, connected or the like may include permanent, removable, temporary, partial, full and/or any other possible attachment option. Additionally, any reference to without contact (or similar phrases) may also include reduced contact or minimal contact.

[0034] As used herein, "‘communication” means communication of electronic signals with physical coupling (e.g., “electrical communication” or “electrically coupled”) or without physical coupling and via an electromagnetic field (e g., “radio frequency communication” or “radio frequency coupled” or “radio frequency coupling”). As used herein, “transmit” may include sending electronic data from one system component to another via electronic communication between the components. Additionally, as used herein, “data” may include encompassing information such as commands, queries, files, data for storage, and the like in digital or any other form which can be transmitted via electronic communication.

[0035] As used herein, “audio input device” is any suitable hardware, software, and/or database components capable of sending and receiving audio data. For example, an audio input device may comprise a wired or wireless microphone, a wired or wireless lapel microphone, a wired or wireless headset/earpiece containing a microphone, and/or the like. The audio input device is in electronic communication with a processor, a cloud processor via a network and/or a remote processor. The audio input device can include frame mounted devices aimed away or towards the operator. The frame mounted devices aimed away from the vehicle operator can be used to gather environmental noise that may be used to filter the audio input. The audio input device can also be an audio output device such as a BLUETOOTH® headset.

[0036] Any databases discussed herein may include relational, hierarchical, graphical, blockchain, object-oriented structure, and/or any other database configurations. Any database may also include a flat file structure wherein data may be stored in a single file in the form of rows and columns, with no structure for indexing and no structural relationships between records. For example, a flat file structure may include a delimited text file, a CSV (comma- separated values) file, and/or any other suitable flat file structure. Common database products that may be used to implement the databases include DB2® by IBM 5 (Armonk, NY), various database products available from ORACLE® Corporation (Redwood Shores, CA), MICROSOFT ACCESS® or MICROSOFT SQL SERVER® by MICROSOFT® Corporation (Redmond, Washington), MYSQL® by MySQL AB (Uppsala. Sweden), MONGODB®, Redis, Apache Cassandra®, HBASE® by APACHE®, MapR-DB by the M APR 11 corporation, or any other suitable database product. Moreover, any database may be organized in any suitable manner, for example, as data tables or lookup tables. Each record may be a single file, a series of files, a linked series of data fields, or any other data structure.

[0037] As used herein, big data may refer to partially or fully structured, semistructured, or unstructured data sets including millions of rows and hundreds of thousands of columns.

[0038] Association of certain data may be accomplished through any desired data association technique such as those known or practiced in the art. For example, the association may be accomplished either manually or automatically. Automatic association techniques may include, for example, a database search, a database merge, GREP, AGREP, SQL, using a key field in the tables to speed searches, sequential searches through all the tables and files, sorting records in the file according to a known order to simplify lookup, and/or the like. The association step may be accomplished by a database merge function, for example, using a “key field” in pre-selected databases or data sectors. Various database tuning steps are contemplated to optimize database performance. For example, frequently used files such as indexes may be placed on separate file systems to reduce In/Out (“I/O”) bottlenecks.

[0039] One skilled in the art will also appreciate that, for security reasons, any databases, systems, devices, servers, or other components of the system may consist of any combination thereof at a single location or at multiple locations, wherein each database or system includes any of various suitable security features, such as firewalls, access codes, encryption, decryption, public and private keys, and/or the like.

[0040] Practitioners will also appreciate that there are a number of methods for displaying data within a brow ser-based document. Data may be represented as standard text or within a fixed list, scrollable list, drop-dowm list, editable text field, fixed text field, pop-up window and the like. Likewise, there are a number of methods available for modifying data in a w eb page such as, for example, free text entry using a keyboard, selection of menu items, check boxes, option boxes, and the like.

[0041] As used herein, the term “network” includes any cloud, cloud computing system, or electronic communications system or method which incorporates hardware and/or software components. Communication among the parties may be accomplished through any suitable communication channels, such as, for example, a telephone network, an extranet, an intranet, internet, personal internet device, online communications, satellite communications, off-line communications, wireless communications, transponder communications, local area network (LAN), wide area network (WAN), virtual private network (VPN), networked or linked devices, keyboard, mouse, and/or any suitable communication or data input modality. Moreover, although the system is frequently described herein as being implemented with TCP/IP communications protocols, the system may also be implemented using IPX, APPLETALK®, IPv6, NetBIOS, any tunneling protocol (e.g. IPsec, SSH, etc.), or any number of existing or future protocols. If the network is in the nature of a public network, such as the internet, it may be advantageous to presume the network to be insecure and open to eavesdroppers. Specific information related to the protocols, standards, and application software utilized in connection with the internet is generally know n to those skilled in the art and, as such, need not be detailed herein.

[0042] "Cloud” or "Cloud computing” includes a model for enabling convenient, on- demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. Cloud computing may include location-independent computing, whereby shared servers provide resources, software, and data to computers and other devices on demand.

[0043] Computer programs (also referred to as computer control logic) are stored in main memory and/or secondary' memory. Computer programs may also be treated as data and transmitted or received via communications interface as a means of dissemination prior to execution. Such computer programs, when executed, enable the computer system to perform the features as discussed herein. In particular, the computer programs, when executed, enable the processor to perform the features of various embodiments. Accordingly, such computer programs represent controllers of the computer system.

[0044] Computer program instructions may be loaded onto a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions that execute on the computer or other programmable data processing apparatus create means for implementing the functions specified in the flowchart block or blocks as shown in the Figures. These computer program instructions may also be stored in a computer-readable memory 7 that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory' produce an article of manufacture including instruction means which implement the function specified in the flowchart block or blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0045] In various embodiments, software may be stored in a computer program product and loaded into a computer system using a removable storage drive, hard disk drive, or communications interface. The control logic (software), when executed by the processor, causes the processor to perform the functions of various embodiments as described herein. In various embodiments, hardware components may take the form of application specific integrated circuits (ASICs) or commercially available packaged integrated circuits such as micro-controller units (MCUs), central processing units (CPUs), or field programmable gate arrays (FPGAs). Implementation of the hardware so as to perform the functions described herein w ill be apparent to persons skilled in the relevant art(s).

[0046] As will be appreciated by one of ordinary skill in the art, the system may be embodied as a customization of an existing system, an add-on product, a processing apparatus executing upgraded software, a stand-alone system, a distributed system, a method, a data processing system, a device for data processing, and/or a computer program product. Accordingly, any portion of the system or a module may take the form of a processing apparatus executing code, an internet based embodiment, an entirely hardware embodiment, or an embodiment combining aspects of the internet, software, and hardware. Furthermore, the system may take the form of a computer program product on a computer-readable storage medium having computer-readable program code means embodied in the storage medium. Any suitable computer-readable storage medium may be utilized, including hard disks, CD- ROM, BLU-RAY DISC®, optical storage devices, magnetic storage devices, flash disk, flash memory' and/or the like.

[0047] In various embodiments, components, modules, and/or engines of the systems may be implemented as applications or apps. Apps are typically deployed in the context of a mobile operating system, including for example, a WINDOWS® mobile operating system, an ANDROID 8 operating system, an APPLE® iOS operating system, a BLACKBERRY® company’s operating system, and the like. The app may be configured to leverage the resources of the larger operating system and associated hardware via a set of predetermined rules which govern the operations of various operating systems and hardware resources. For example, where an app desires to communicate with a device or network other than the mobile device or mobile operating system, the app may leverage the communication protocol of the operating system and associated device hardware under the predetermined rules of the mobile operating system. Moreover, where the app desires an input from a user, the app may be configured to request a response from the operating system which monitors various hardware components and then communicates a detected input from the hardw are to the app.

[004S] The system and methods are described herein with reference block diagrams and flowchart illustrations of methods, apparatus, and computer program products according to various embodiments. It will be understood that each functional block of the block diagrams and the flowchart illustrations, and combinations of functional blocks in the block diagrams and flowchart illustrations, respectively, can be implemented by computer program instructions.

[0049] Accordingly, functional blocks of the block diagrams and flowchart illustrations support combinations of means for performing the specified functions, combinations of steps for performing the specified functions, and program instruction means for performing the specified functions. It will also be understood that each functional block of the block diagrams and flowchart illustrations, and combinations of functional blocks in the block diagrams and flowchart illustrations, can be implemented by either special purpose hardware-based computer systems which perform the specified functions or steps, or suitable combinations of special purpose hardware and computer instructions. Further, illustrations of the process flows, and the descriptions thereof may make reference to user WINDOWS® / LINUX® / UNIX® applications, webpages, websites, web forms, prompts, etc. Practitioners will appreciate that the illustrated steps described herein may comprise, in any number of configurations, including the use of WINDOWS® / LINUX® / UNIX® applications, webpages, web forms, popup WINDOWS® / LINUX® / UNIX® applications, prompts, and the like. It should be further appreciated that the multiple steps as illustrated and described may be combined into single webpages and/or WINDOWS® / LINUX® / UNIX® applications but have been expanded for the sake of simplicity. In other cases, steps illustrated and described as single process steps may be separated into multiple webpages and/or WINDOWS® / LINUX® I UNIX® applications but have been combined for simplicity.

[0050] The computers discussed herein may provide a suitable website or other internet-based graphical user interface (GUI) which is accessible by users. In one embodiment, MICROSOFT® company’s Internet Information Services (IIS), Transaction Server (MTS) service, and an SQL SERVER® database, are used in conjunction with MICROSOFT® operating systems, WINDOWS NT® web server software, SQL SERVER® database, and MICROSOFT® Commerce Server. Additionally, components such as ACCESS® software, SQL SERVER® database, ORACLE® software, SYBASE® software, INFORMIX® software, MYSQL® software, INTERBASE® software, etc., may be used to provide an Active Data Object (ADO) compliant database management system. In one embodiment, the APACHE® web server is used in conjunction with a LINUX® operating system, a MYSQL® database, and PHP, Ruby, and/or PYTHON® programming languages.

[0051] The system and method may be described herein in terms of functional block components, screen shots, optional selections, and various processing steps. It should be appreciated that such functional blocks may be realized by any number of hardware and/or software components configured to perform the specified functions. For example, the system may employ various integrated circuit components, e.g.. memory elements, processing elements, logic elements, look-up tables, and the like, which may carry out a variety of functions under the control of one or more microprocessors or other control devices. Similarly, the software elements of the system may be implemented with any programming or scripting language such as C, C++. C#, JAVA®, JAVASCRIPT®, VBScript, Macromedia COLD FUSION, COBOL, MICROSOFT® company’s Active Server Pages, assembly, PERL® , PHP, awk, PYTHON®, Visual Basic, SQL Stored Procedures, PL/SQL, and any UNIX® shell script with the various algorithms being implemented with any combination of data structures (such as JAVASCRIPT® Object Notation (JSON) and extensible markup language (XML)), objects, processes, routines or other programming elements. Further, it should be noted that the system may employ any number of conventional techniques for data transmission, signaling, data processing, network control, and the like. Still further, the system could be used to detect or prevent security' issues with a client-side scripting language, such as JAVASCRIPT®, VBScript, or the like. [0052] For the sake of brevity, conventional data networking, application development, and other functional aspects of the systems (and components of the individual operating components of the systems) may not be described in detail herein. Furthermore, the connecting lines shown in the various figures contained herein are intended to represent exemplary' functional relationships and/or physical couplings between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may be present in a practical system.

[0053] Computer-based system program instructions and/or processor instructions may be loaded onto a tangible, non-transitory computer readable medium having instructions stored thereon that, in response to execution by a processor, cause the processor to perform various operations. The term "non-transitory" is to be understood to remove only propagating transitory signals per se from the claim scope and does not relinquish rights to all standard machine-readable media that are not only propagating transitory signals per se. Stated another way, the meaning of the term “non-transitory machine-readable medium" and “non-transitory machine-readable storage medium” should be construed to exclude only those types of transitory machine-readable media which were found in In re Nuijten to fall outside the scope of patentable subject matter under 35 U.S.C. § 101.

[0054] Operators of off-road vehicles can commit errors when command options are not fully known, or particular operating data is not available to the operator. Additionally, operators can commit errors by inputting commands that are inappropriate to the current operating mode of the off-road vehicle, whether intentionally or by accident. For example, operator errors can be caused by an operator entering incorrect data including, but not limited to, entering data outside the acceptable ranges for the current operating mode of the off-road vehicle. Voice control systems for off-road vehicles can provide additional support and safety measures by identifying available commands and verifying the commands are safe before executing.

[0055] With reference to FIG. 1, an off-road vehicle 100 having an interior cabin 102, a drive system comprising an engine 104 and tires 106 is illustrated, in accordance with various embodiments. The off-road vehicle 100 comprises a tool system 1 8 such as a bucket or attachment used for mining and agricultural applications. Off-road vehicle 100 also comprises an image sensor 110. The image sensor 110 can be, for example, a camera, a RADAR sensor, a LiDAR sensor, or the like. In various embodiments, the image sensor 110 can be an IR sensor and/or a structured light 3D scanner which comprises an IR dot projector and an IR sensor for producing 3D images. The image sensor 110 is used to map the terrain around the off-road vehicle 1 . Although illustrated with one image sensor 110, the off-road vehicle 100 can be equipped with multiple image sensors 110 to provide a terrain output with a full 360 degree terrain map around the off-road vehicle 100.

[0056] Knowing the location of the off-road vehicle 100 can be helpful in determining whether a command is safe and allowable for the off-road vehicle 100. In various embodiments, the off-road vehicle 100 can comprise a global positioning system (GPS) receiver 107. The GPS receiver 107 can also be a global navigation satellite system (GNSS) receiver. In various embodiments, the GPS receiver 107 can be a stand-alone device disposed on the roof of the off road vehicle. GPS and GNSS are a network of satellites that orbit Earth and can pinpoint the geographic location of a device using wireless signals. An off-road vehicle location can be determined from a GPS location of the GPS receiver 107. In various embodiments, the GPS receiver 107 is configured to transmit the GPS location to a processor. In various embodiments, the GPS receiver 107 is configured to continuously transmit the GPS location to the processor.

[0057] With reference to FIG. 2, an off-road vehicle 200 having an interior cabin 202, a drive system comprising an engine 204 and tires 206 is illustrated, in accordance with various embodiments. The off-road vehicle 200 comprises a tool system 208 such as a dump body used for mining and agricultural applications to haul and dump materials. Off-road vehicle 200 also comprises an image sensor 210. The image sensor 210 can be the same as the image sensor 110 described herein. In various embodiments, the off-road vehicle 200 can comprise a GPS receiver 207. The GPS receiver 207 can also be a GNSS receiver. In various embodiments, the GPS receiver 207 can be a stand-alone device disposed on the roof of the off road vehicle. The GPS receiver 207 can be the same as described herein for the GPS receiver 107.

[0058] With reference to FIG. 3, an off-road vehicle 300 having an interior cabin 302, a drive system comprising an engine 304 and tires 306 is illustrated, in accordance with various embodiments. The off-road vehicle 300 comprises a tool system 308 such as a hitch for a plow used in agricultural applications. Off-road vehicle 300 also comprises an image sensor 310. The image sensor 310 can be the same as the image sensor 110 described herein. In various embodiments, the off-road vehicle 300 can comprise a GPS receiver 307. The GPS receiver 307 can also be a GNSS receiver. In various embodiments, the GPS receiver 307 can be a stand-alone device disposed on the roof of the off road vehicle. The GPS receiver 307 can be the same as described herein for the GPS receiver 107.

[0059] With combined reference to FIGS. 1-4, an interior cabin 400 of an off-road vehicle is illustrated, in accordance with various embodiments. Interior cabin 400 can be used for any of the interior cabins 102, 202, and 302. The interior cabin 400 can be used to seat the operator of the off-road vehicle in the event the off-road vehicle is being operated locally or the interior cabin 400 can be vacant if the off-road vehicle is being operated remotely. The interior cabin 400 comprises a processor 402 in communication with an audio input device 404 (such as an audio input device as described herein). The processor 402 can also be in electronic communication with a machine control system 403, a tool system (such as tool systems 108, 208. and 308), an image sensor (such as image sensors 110, 210, and 310), a haptic device 410 imbedded in a steering wheel 408 and/or a seat 409, a start button 411, a visual display 412, a driving display 406 and a speaker 414. The start button 411 is configured to transmit a start button signal to the processor 402 in response to being engaged.

[0060] In various embodiments, the off-road vehicle can be operated remotely. When the off-road vehicle is operated remotely, a remote audio input device is used to transmit remote audio input to a remote processor. Additionally, the remote processor can be in communication with a remote visual display, a remote speaker and a remote haptic device all disposed in a remote station.

[0061] The machine control system 403 may include one or more logic devices such as one or more of a central processing unit (CPU), a graphical processing unit (GPU), an accelerated processing unit (APU), a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. In various embodiments, the processor 402 may further include any non-transitory memory known in the art and store instructions usable by the logic device to perform operations. Any appropriate computer-readable type/con figuration may be utilized as a part of the logic device, and any appropriate data storage architecture may be utilized by the logic device, or both.

[0062] The processor 402 is configured to control the movement of the off-road vehicle through a drive output signal transmitted to the machine control system 403. The machine control system 403 is in communication with a drive system (such as the drive systems described herein) and is configured to send commands to the drive system. In various embodiments, the drive output can control the machine control system 403 to command the engine to operate at a set output and for a set time until a desired distance is achieved. In various embodiments, the drive output can control the machine control system 403 to command the steering wheel 408 to turn for a set time until a desired orientation is achieved. In that regard, closed loop control is implemented, where a command is performed until a contingent event occurs, though open loop control is also possible, where a command is executed at a single time or is continually executed until a command to cease execution is received.

[0063] The processor 402 may include one or more logic devices such as one or more of a central processing unit (CPU), a graphical processing unit (GPU), an accelerated processing unit (APU), a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. In various embodiments, the processor 402 may further include any non-transitory memory known in the art and store instructions usable by the logic device to perform operations. Any appropriate computer-readable type/configuration may be utilized as a part of the logic device, and any appropriate data storage architecture may be utilized by the logic device, or both.

[0064] In various embodiments, the processor 402 is configured to transmit a signal, such as an alert, to the speaker (as an audio output), to the visual display (as a visual output), and/or to the haptic device (as a vibrating output). In various embodiments, the processor 402 is configured to receive signals from the machine controller 403 which is in communication with an environmental input device, for example, the image sensor (such as image sensors 110, 210, and 310), the GPS receiver 407 and other sensors that may be used on the off-road vehicle to determine the safety of the off-road vehicle. Some other sensors that could be used as environment input devices are wheel sensors embedded in the tire which can transmit a wheel spin rate to determine whether each wheel has lost traction or may soon lose traction with the driving surface, and fuel sensors connected to the fuel tank of the off-road vehicle to transmit fuel levels to help determine whether the off-road vehicle has enough fuel to complete a command.

[0065] In various embodiments, the processor 402 is also connected to a cloud system via a network and is configured to transmit signals to and receive signals from the cloud system. In various embodiments, the processor 402 is in electronic communication with a remote system (such as another off-road vehicle, an on-site operator, or an off-site operator) via a network and the processor 402 is configured to transmit signals to and receive signals from the remote system.

[0066] With reference to FIG. 5, a block diagram for a control system 500 is illustrated in accordance with various embodiments. The control system 500 can be a control system onboard a vehicle, such as off-road vehicles 100, 200, and 300. In various embodiments, the control system 500 comprises a processor 502 (such as processor 402). The processor 502 may include one or more logic devices such as one or more of a central processing unit (CPU), a graphical processing unit (GPU), an accelerated processing unit (APU), a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. In various embodiments, the processor 502 may further include any non-transitory memory known in the art and store instructions usable by the logic device to perform operations. Any appropriate computer-readable type/ configuration may be utilized as a part of the logic device, and any appropriate data storage architecture may be utilized by the logic device, or both.

[0067] In various embodiments, the control system 500 further comprises an environmental input device 504 (such as any of the environmental input devices described herein). In various embodiments, there are more than one environmental input devices 504. The environmental input device 504 is in communication with processor 502 and is configured to output an environmental input (such as GPS location, fuel levels, terrain output, and wheel spin rate). For example, wheel spin rate is measured as the number of wheel revolutions per minute.

[0068] In various embodiments, the control system 500 further comprises an audio input device 506 (such as the audio input device 404). In various embodiments, there are more than one audio input device 506. The audio input device 506 is in communication with processor 502 and is configured to electronically output an audio input. In various embodiments, the audio input device 506 can continuously transmit audio input to the processor 502.

[0069] In various embodiments, the control system 500 further comprises a machine control system 508 (such as machine control system 403). The machine control system 508 is in communication with processor 502 and is configured to control the engine and steering wheel of the off-road vehicle based on a drive output from the processor 502. The drive output can comprise an engine input and a steenng wheel input to control the off-road vehicle. In various embodiments, the drive output is a command, for example make speed 5 miles per hour, or nudge vehicle right 6 inches, and the drive output is then converted by the machine control system into the engine input and the steering wheel input.

[0070] In various embodiments, the control system 500 further comprises an alert device 510 (such as the haptic device 410, the visual display 412, and the speakers 414). In various embodiments, there are more than one alert devices 510, including a mix of any quantity and combination of alert devices 510.

[0071] Visual display 412 can be a screen display or a light (for example an LED) that can illuminate to indicate the type of alert being relayed to the operator. The alert device 510 is in communication with processor 502 and is configured to output an alert notification. In various embodiments, the alert notification can be a vibrating output for a haptic device to vibrate, for example, a haptic device causing vibration of a steering wheel (such as steering wheel 410) or vibration of a seat (such as seat 409) to alert the operator. In various embodiments, the alert notification can be a visual output for a visual display to visually alert the operator. In various embodiments, the alert notification can be an audio output for a speaker to audially alert the operator. The audio output can comprise a tone, a chime, a beep, and/or the like. The audio output can also be a text-to-voice response, for example output through an attached internal or external speaker. In various embodiments, the alert notification can be used to alert the operator of a command that would be dangerous to execute given the environmental and/or operating conditions, and/or the alert notification can be used to confirm with the operator that the command determined by the processor is the desired command before outputting said command.

[0072] With reference to FIG. 8, a method 800 for operating a vehicle is shown. The method may comprise identifying, by a processor (shown as processors 402 and 502) an operation mode of a vehicle, resulting in an identified operating mode (step 802). The processor is configured to receive operation data from the vehicle and a machine control system (shown as machine control system 508), for example, an engine state (whether engine is off or on) a vehicle speed, a tool state (whether the off-road vehicle's tool system is being used) and other operation data of the like. The processor, in response to receiving the operation data, can determine the identified operating mode.

[0073] In various embodiments, the method 800 may comprise receiving, by the processor, an audio input from an audio input device (show n as audio input devices 404 and 506) in communication with the processor (step 804). In various embodiments, the method 800 may comprise receiving, by the processor, an environmental input from an environmental sensor input device (shown as environmental input device 504) in communication with the processor (step 806).

[0074] In various embodiments, the method 800 may comprise determining, by the processor, whether the audio input is a first command of a first plurality of possible voice commands (step 808). The processor comprises the first plurality of possible voice commands in the non-transilory memory of the processor. In various embodiments, the first plurality’ of possible voice commands can be a list of all executable commands and functions for the vehicle. If the audio input is determined to match with a command from the first plurality' of possible voice commands, then the processor assigns the audio input as the first command. If the audio input is determined to not match with a command from the first plurality of possible voice commands, then the processor does not assign the first command and the processor can communicate the rejection using a default response signal which can be sent to any alert device.

[0075] In various embodiments, the audio input received by the audio input device and transmitted to the processor can be continuous or substantially continuous, though it is understood that not all audio received by the audio input device is intended to be a command. This would be useful in preventing miscellaneous audio from being further analyzed by the processor. In various embodiments, the triggering event can be the use of a wake word. In that regard, the audio input may be received continuously or substantially continuously from the audio input device and analyzed, for example in real time, by the processor for purposes of detecting the wake word. Upon identifying the wake work, the processor may regard this as the triggering event. In various embodiments, however, the triggering event can be another suitable electronic input, such as the engaging of a start button (such as start button 411) by the operator or sending of a start signal over the network from a user device to the processor. In various embodiments, the operations of method 800 can further comprise receiving, by the processor, a triggering event comprising at least one of a wake word, a start button signal, and a start signal.

[0076] In various embodiments, the method 800 may comprise validating, by the processor, whether the first command is allowable based on the identified operating mode and the environmental input (step 810). In various embodiments, the step of validating further comprises comparing, by the processor, the first command to the speed of the vehicle. For example, if the first command is turn off the engine, and the speed of the vehicle is 50 miles per hour, it would not be safe to turn off the engine at that speed. Therefore, the first command would not be validated as allowable. If the first command is, for example, move forward 50 feet, and the speed of the vehicle is 0 mph, it would be safe to command the machine control system to move the vehicle forward 50 feet. Therefore, the first command would be validated as allowable.

[0077] In various embodiments, the step of validating further comprises comparing, by the processor, the first command to the state of the vehicle. For example, if the first command is to start autosteering, and the vehicle is currently in park. Therefore, the first command would not be validated as allowable. If the first command is start autosteering, and the vehicle is in motion, it would be safe to control the machine control system to start autosteering. Therefore, the first command would be validated as allowable.

[0078] In various embodiments, the step of validating further comprises comparing, by the processor, the first command to the terrain output. For example, if the first command is move forward 50 feet, and the terrain output shows a slope that exceeds a set negative grade, for example -10% or more, it would not be safe to move forward 50 feet in heavy off-road vehicles. Therefore, the first command would not be validated as allowable. If the first command is move forward 50 feet, and the terrain output shows no physical limitations or a slope exceeding the set negative grade for the next 50 feet, it would be safe to control the machine control system to move the vehicle forward 50 feet. Therefore, the first command would be validated as allowable. Additionally, the off-road vehicle can comprise an Inertial Measurement Unit (IMU), in communication with the processor, configured to transmit an attitude of the off-road vehicle to the processor. In various embodiments, the attitude of the off-road vehicle can also be compared to the first command. Moreover, in various embodiments, the off-road vehicle can comprise an angular velocity sensor, such as a gyroscopic sensor, for the detection of angular velocity.

[0079] In various embodiments, the step of validating can comprise comparing one or more of the environmental sensor inputs (GPS location, terrain output, and speed of the vehicle) to validate the first command. In various embodiments, if the first command is not allowable, then the processor outputs an alert notification to one or more alert devices. In various embodiments, the alert notification can be an alert to the operator that the first command is not allowable.

[0080] In various embodiments, the method 800 may comprise controlling, by the processor, the machine control system in response to the first command being allowable (step 812). In various embodiments, method 800 may comprise controlling, by the processor, a tool system (shown as tool systems 108, 208, and 308) in response to the first command being allowable.

[0081] In various embodiments, the processor outputs a confirmation notification requesting a confirmatory response from the operator to confirm that the first command is correct. The confirmation notification can be displayed by the visual display or played via audio signal through the speakers. In various embodiments, the method 800 may comprise receiving, by the processor, a confirmatory response through the audio input device, the start button, or the visual display. In various embodiments, the processor controls the vehicle system or the tool system with the first command in response to the processor receiving an affirmative confirmatory response (i.e., yes).

[0082] In various embodiments, the method 800 can further comprise filtering, by the processor, the audio input based on the identified operating mode of the vehicle and/or an environmental audio input from a second audio input device (step 807). The processor can filter the audio input to obtain a more ideal sample of audio input for processing, to remove background noises from the audio input which can be found using the identified operating mode of the vehicle and/or the second audio input, and can convert the audio input to a list of recognized words and confidences for the processor to analyze with the first plurality of possible voice commands. The second audio input device can be positioned and configured to only transmit a second audio input received from the drive system and the environment around the vehicle. In various embodiments, the processor (such as processors 402 and/or 502) compares the audio input to the second audio input to filter the audio input by removing overlapping audio data between the audio input and the second audio input. The resulting output is a filtered audio input.

[0083] In various embodiments, the processors 402 and 502 can comprise instructions stored thereon, that in response to execution by the processors 402 and 502, cause the processors 402 and 502 to perform operations comprising any of the steps and embodiments of method 800. The method 800 can be performed by the processors 402 and 502.

[0084] With reference to FIG. 6, a block diagram for a control system 600 is illustrated in accordance with various embodiments. The control system 600 can be a vehicle control system onboard a vehicle and connected to a network, such as off-road vehicles 100, 200, and 300. In various embodiments, the control system 600 comprises the processor 502. In various embodiments, the processor 502 is in electronic communication with a network 602. In various embodiments, a cloud system 604 comprising a cloud database 606 and a cloud processor 608 are also in electronic communication with the network 602. The cloud database 606 may store instructions usable by the cloud processor 608 to perform operations. Any appropriate computer-readable type/configuration may 7 be utilized as the cloud database 606, any appropriate data storage architecture may be utilized by the cloud database 606, or both.

[0085] The cloud processor 608 may include one or more logic devices such as one or more of a central processing unit (CPU), a graphical processing unit (GPU), an accelerated processing unit (APU), a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. In various embodiments, the cloud processor 608 may further include any non-transitory memory known in the art and store instructions usable by a logic device to perform operations. Any appropriate computer-readable type/configuration may be utilized as a part of the logic device, and any appropriate data storage architecture may be utilized by the logic device, or both. In various embodiments, the cloud processor 608 can be an on-premise processor or a dedicated server processor.

[0086] In various embodiments, the control system 600 further comprises the environmental input device 504. In various embodiments, there are more than one environmental input devices 504. The environmental input device 504 is in communication with processor 502 and is configured to output an environmental input. In various embodiments, the control system 600 further comprises the audio input device 506. In various embodiments, there are more than one audio input device 506. In various embodiments, the audio input device 506 can continuously transmit audio input to the processor 502. In various embodiments, the control system 600 further comprises a machine control system 508. In various embodiments, the control system 600 further comprises an alert device 510 (such as the haptic device 410, the visual display 412, and the speakers 414). In various embodiments, there are more than one alert devices 510. The alert device 510 is in communication with processor 502 and is configured to output an alert notification as described herein.

[0087] With reference to FIG. 9, a method 900 for operating a vehicle is shown. The method may comprise identifying, by a processor (shown as processors 402 and 502) an operation mode of a vehicle, resulting in an identified operating mode (step 902). The processor is configured to receive operation data from the vehicle and a machine control system (shown as machine control system 508), for example, an engine state (whether engine is off or on) a vehicle speed, a tool state (whether the off-road vehicle’s tool system is being used or not) and other operation data of the like. The processor, in response to receiving the operation data, can determine the identified operating mode.

[0088] In various embodiments, the method 900 may comprise receiving, by the processor, an audio input from an audio input device (shown as audio input devices 404 and 506) in communication with the processor (step 904). In various embodiments, the method 900 may comprise receiving, by the processor, an environmental input from an environmental sensor input device (shown as environmental input device 504) in communication with the processor (step 906). In various embodiments, the method 900 may comprise transmitting, by the processor, the identified operating mode of the vehicle, the audio input and the environmental input to the cloud processor (shown as cloud processor 608) via a network (shown as network 602) (step 908). The cloud processor is in communication with the cloud database. The cloud database comprises a first plurality of possible voice commands in the non-transitory memory of the cloud database. In various embodiments, the first plurality of possible voice commands can be a list of all executable commands and functions for the vehicle. If the audio input is determined to match with a command from the first plurality’ of possible voice commands, then the processor assigns the audio input as the first command. If the audio input is determined to not match with a command from the first plurality of possible voice commands, then the processor does not assign the first command.

[0089] In various embodiments, the audio input received by the audio input device and transmitted to the processor can be continuous or substantially continuous, though it is understood that not all audio received by the audio input device is intended to be a command. This would be useful in preventing miscellaneous audio from being further analyzed by the processor or by the cloud processor. In that regard, the audio input may be received continuously or substantially continuously from the audio input device and analyzed, for example in real time, by the processor for purposes of detecting the wake word. Upon identifying the wake work, the processor may regard this as the triggering event. In various embodiments, however, the triggering event can be another suitable electronic input, such as the engaging of a start button (such as start button 411) by the operator or sending of a start signal over the network from a user device to the processor. In various embodiments, the operations of method 900 can further comprise receiving, by the processor, a triggering event comprising at least one of a wake word, a start button signal, and a start signal.

[0090] In various embodiments, the method 900 may comprise receiving, by the processor, the first command from the cloud processor (step 910). In various embodiments, the method 900 may comprise controlling, by the processor, the machine control system in response to the first command (step 912). In various embodiments, method 900 may comprise controlling, by the processor, a tool system (shown as tool systems 108, 208, and 308) in response to the first command.

[0091] In various embodiments, the processor outputs a confirmation notification requesting a confirmatory response from the operator to confirm that the first command is correct. The confirmation notification can be displayed by the visual display, played through the speakers, vibrated with haptics in a seat or steering wheel, a network message or via a wireless audio device. In various embodiments, the method 900 may comprise receiving, by the processor, a confirmatory response through the audio input device, the visual display, the start button, or a network message. In various embodiments, the processor controls the vehicle system or the tool system with the first command in response to the processor receiving an affirmative confirmatory response (i.e., yes).

[0092] In various embodiments, the method 900 can further comprise filtering, by the cloud processor, the audio input based on the identified operating mode of the vehicle and/or an environmental audio input from a second audio input device. The cloud processor can filter the audio input to obtain a more ideal sample of audio input for processing, to remove background noises from the audio input which can be found using the identified operating mode of the vehicle and/or the second audio input, and can convert the audio input to a list of recognized words and confidences for the cloud processor to analyze with the first plurality of possible voice commands. The second audio input device can be positioned and configured to only transmit a second audio input received from the drive system and the environment around the vehicle. In various embodiments, the cloud processor compares the audio input to the second audio input to filter the audio input by removing overlapping audio data between the audio input and the second audio input. The resulting output is a filtered audio input.

[0093] In various embodiments, the processors 402 and 502 can comprise instructions stored thereon, that in response to execution by the processors 402 and 502, cause the processors 402 and 502 to perform operations comprising any of the steps and embodiments of method 900. The method 900 can be performed by the processors 402 and 502.

[0094] With reference to FIG. 10, a method 1000 for operating a vehicle is shown. The method may comprise receiving, by a cloud processor (shown as cloud processor 608), an identified operating mode of a vehicle (shown as off-road vehicles 100, 200, and 300), an audio input and an environmental input via a network (shown as network 602) (step 1002). A processor (shown as processor 502) is configured to receive operation data from the vehicle and a machine control system (shown as machine control system 508), for example, an engine state (whether engine is off or on) a vehicle speed, a tool state (whether the off-road vehicle’s tool system is being used or not) and other operation data of the like. The processor, in response to receiving the operation data, can determine the identified operating mode. The processor is configured to transmit the identified operating mode of the vehicle, the audio input and the environmental input via the network.

[0095] In various embodiments, the method 1000 may comprise determining, by the cloud processor, whether the audio input is a first command of a first plurality of possible voice commands stored in the cloud database (step 1004). The cloud database comprises the first plurality of possible voice commands in the non-transitory memory of the cloud database. In various embodiments, the first plurality of possible voice commands can be a list of all executable commands and functions for the vehicle. If the audio input is determined to match wi th a command from the first plurality of possible voice commands, then the cloud processor assigns the audio input as the first command. If the audio input is determined to not match with a command from the first plurality of possible voice commands, then the cloud processor does not assign the first command and the cloud processor can communicate the rejection using a default response signal which can be sent to any alert device.

[0096] In various embodiments, the method 1000 may comprise validating, by the cloud processor, whether the first command is allowable based on the identified operating mode and the environmental input (step 1006).

[0097] In various embodiments, the step of validating further comprises comparing, by the cloud processor, the first command to the speed of the vehicle. For example, if the first command is turn off the engine, and the speed of the vehicle is 50 miles per hour, it would not be safe to turn off the engine at that speed. Therefore, the first command would not be validated as allowable. If the first command is move forward 50 feet, and the speed of the vehicle is 0 mph, it would be safe to control the machine control system to move the vehicle forward 50 feet. Therefore, the first command would be validated as allowable.

[0098] In various embodiments, the step of validating further comprises comparing, by the cloud processor, the first command to the GPS location of the vehicle. For example, if the first command is move forward ten miles, and the GPS location of the vehicle is two miles from a body of water, it would not be safe to move forward ten miles. Therefore, the first command would not be validated as allowable. If the first command is move forward one mile, and the GPS location of the vehicle shows no physical barriers within one mile, it would be safe to control the machine control system to move the vehicle forward one mile. Therefore, the first command would be validated as allowable.

[0099] In various embodiments, the step of validating further comprises comparing, by the cloud processor, the first command to the terrain output. For example, if the first command is move forward 50 feet, and the terrain output shows a slope that exceeds a set negative grade, for example -10% or more, it would not be safe to move forward 50 feet in heavy off-road vehicles. Therefore, the first command would not be validated as allowable. If the first command is move forward 50 feet, and the terrain output shows no physical limitations or a slope exceeding the set negative grade for the next 50 feet, it would be safe to control the machine control system to move the vehicle forward 50 feet. Therefore, the first command would be validated as allowable. Additionally, the off-road vehicle can comprise an Inertial Measurement Unit (IMU), in communication with the processor, configured to transmit an attitude of the off-road vehicle to the processor. In various embodiments, the attitude of the off-road vehicle can also be compared to the first command. Moreover, in various embodiments, the off-road vehicle can comprise an angular velocity sensor, such as a gyroscopic sensor, for the detection of angular velocity.

[00100] In various embodiments, the step of validating can comprise comparing each of the environmental sensor inputs (GPS location, terrain output, and speed of the vehicle) to validate the first command. In various embodiments, if the first command is not allowable then the processor outputs an alert notification to one or more alert devices. In various embodiments, the alert notification can be an alert to the operator that the first command is not allowable.

[00101] In various embodiments, the method 1000 may comprise transmitting, by the cloud processor, the first command to a processor (shown as processor 402 and 502) (step 1008). In various embodiments, the method 1000 may comprise controlling, by the processor, the machine control system in response to the first command (step 1010). In various embodiments, method 1000 may comprise controlling, by the processor, a tool system (shown as tool systems 108, 208, and 308) in response to the first command.

[00102] In various embodiments, the processor outputs a confirmation notification requesting a confirmatory response from the operator to confirm that the first command is correct. The confirmation notification can be displayed by the visual display or played through the speakers. In various embodiments, the method 1000 may comprise receiving, by the processor, a confirmatory response through the audio input device or the visual display. In various embodiments, the processor controls the vehicle system or the tool system with the first command in response to the processor receiving an affirmative confirmatory response (i.e.. yes).

[00103] In various embodiments, the method 1000 can further comprise filtering, by the cloud processor, the audio input based on the identified operating mode of the vehicle and/or an environmental audio input from a second audio input device. The cloud processor can filter the audio input to obtain a more ideal sample of audio input for processing, to remove background noises from the audio input which can be found using the identified operating mode of the vehicle and/or the second audio input, and can convert the audio input to a list of recognized words and confidences for the cloud processor to analyze with the first plurality of possible voice commands. The second audio input device can be positioned and configured to only transmit a second audio input received from the drive system and the environment around the vehicle. In various embodiments, the cloud processor compares the audio input to the second audio input to filter the audio input by removing overlapping audio data between the audio input and the second audio input. The resulting output is a filtered audio input. [00104] In various embodiments, the cloud processor 608 can comprise instructions stored thereon, that in response to execution by the cloud processor 608, cause cloud processor 608 to perform operations comprising any of the steps and embodiments of method 1000. The method 1000 can be performed by the cloud processor 608.

[00105] With reference to FIG. 7, a block diagram for a control system 700 is illustrated in accordance with various embodiments. The control system 700 can be a vehicle remote control system used with remote operators of off-road vehicles such as off-road vehicles 100, 200, and 300. Control system 700 can be used to control one off-road vehicle at a time or entire fleets of off-road vehicles. In various embodiments, the control system 700 comprises the processor 502. In various embodiments, the processor 502 is in electronic communication with a network 702. In various embodiments, a remote system 704 comprising a remote database 706 and a remote processor 708 are also in electronic communication with the network 702. The remote database 706 may store instructions usable by the remote processor 708 to perform operations. Any appropriate computer-readable type/configuration may be utilized as the remote database 706, any appropriate data storage architecture may be utilized by the remote database 706, or both.

[00106] The remote processor 708 may include one or more logic devices such as one or more of a central processing unit (CPU), a graphical processing unit (GPU), an accelerated processing unit (APU), a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. In various embodiments, the remote processor 708 may further include any non-transitory memory known in the art and store instructions usable by a logic device to perform operations. Any appropriate computer-readable type/configuration may be utilized as a part of the logic device, and any appropriate data storage architecture may be utilized by the logic device, or both.

[00107] In various embodiments, the control system 700 further comprises the environmental input device 504. In various embodiments, there are more than one environmental input devices 504. The environmental input device 504 is in communication with processor 502 and is configured to output an environmental input over the network 702 to the remote system 704. In various embodiments, the control system 700 further comprises a machine control system 508.

[00108] In various embodiments, the remote system 704 further comprises an audio input device 710 configured to receive an audio input from the remote operator. In various embodiments, the remote system 704 further comprises an alert device such as a visual display 712 and/or a speaker 714 configured to output an alert notification receive from the remote processor 708. The visual display 712 and the speaker 714 are in communication with the remote processor 708.

[00109] With reference to FIG. 11, a method 1100 for remotely operating a vehicle is shown. The method may comprise identifying, by a processor (shown as processors 402 and 502) an operation mode of a vehicle, resulting in an identified operating mode (step 1102). The processor is configured to receive operation data from the vehicle and a machine control system (shown as machine control system 508), for example, an engine state (whether engine is off or on) a vehicle speed, a tool state (whether the off-road vehicle’s tool system is being used or not) and other operation data of the like. The processor, in response to receiving the operation data, can determine the identified operating mode.

[00110] In various embodiments, the method 1100 may comprise receiving, by the processor, an environmental input from an environmental sensor input device (shown as environmental input device 504) in communication with the processor (step 1104). In various embodiments, the method 1100 may comprise transmitting, by the processor, the identified operating mode of the vehicle and the environmental input to a remote processor (shown as remote processor 708) via a network (shown as network 702) (step 1106). The remote processor is in communication with a remote database (shown as remote database 706).

[00111] In various embodiments, the method 1100 may comprise receiving, by the processor, a first command from the remote processor (step 1108). In various embodiments, the method 1100 may comprise controlling, by the processor, the machine control system in response to the first command (step 1110). In various embodiments, method 1100 may comprise controlling, by the processor, a tool system (shown as tool systems 108, 208, and 308) in response to the first command.

[00112] In various embodiments, the remote processor outputs a confirmation notification requesting a confirmatory response from the operator to confirm that the first command is correct. The confirmation notification can be displayed by a visual display (shown as visual display 712) or played through a speaker (shown as speaker 714) of the remote system 704. In various embodiments, the method 1100 may comprise receiving, by the remote processor, a confirmatory response through an audio input device (shown as audio input device 710) or the visual display. In various embodiments, the processor controls the vehicle system or the tool system with the first command in response to the remote processor receiving an affirmative confirmatory response (i.e., yes).

[00113] In various embodiments, the processors 402 and 502 can comprise instructions stored thereon, that in response to execution by the processors 402 and 502, cause the processors 402 and 502 to perform operations comprising any of the steps and embodiments of method 1100. The method 1100 can be performed by the processors 402 and 502.

[00114] With reference to FIG. 12, a method 1200 for remotely operating a vehicle is shown. The method may comprise receiving, by a remote processor (shown as remote processor 708), an identified operating mode of a vehicle (shown as vehicles 100, 200, and 300), an audio input and an environmental input via a netw ork (shown as netw ork 702) (step 1202). A processor (shown as processor 502) is configured to receive operation data from the vehicle and a machine control system (shown as machine control system 508), for example, an engine state (whether engine is off or on) a vehicle speed, a tool state (whether the off-road vehicle’s tool system is being used or not) and other operation data of the like. The processor, in response to receiving the operation data, can determine the identified operating mode. The processor is configured to transmit the identified operating mode of the vehicle, the audio input and the environmental input via the network.

[00115] In various embodiments, the remote system comprises an audio input device (shown as audio input device 710) configured to receive an audio input from the remote operator. In various embodiments, the method 1200 may comprise determining, by the remote processor, whether the audio input is a first command of a first plurality of possible voice commands stored in the remote database (shown as remote database 706) (step 1204). The remote database comprises the first plurality of possible voice commands in the non-transitory memory of the cloud database. In various embodiments, the first plurality of possible voice commands can be a list of all executable commands and functions for the vehicle. If the audio input is determined to match with a command from the first plurality of possible voice commands, then the remote processor assigns the audio input as the first command If the audio input is determined to not match with a command from the first plurality of possible voice commands, then the remote processor does not assign the first command and the remote processor can communicate the rejection using a default response signal which can be sent to any alert device.

[00116] In various embodiments, the method 1200 may comprise validating, by the remote processor, whether the first command is allowable based on the identified operating mode and the environmental input (step 1206).

[00117] In various embodiments, the step of validating further comprises comparing, by the remote processor, the first command to the speed of the vehicle. For example, if the first command is turn off the engine, and the speed of the vehicle is 50 miles per hour, it would not be safe to turn off the engine at that speed. Therefore, the first command would not be validated as allowable. If the first command is move forward 50 feet, and the speed of the vehicle is 0 mph, it would be safe to control the machine control system to move the vehicle forward 50 feet. Therefore, the first command would be validated as allowable.

[00118] In various embodiments, the step of validating further comprises comparing, by the remote processor, the first command to the GPS location of the vehicle. For example, if the first command is move forward ten miles, and the GPS location of the vehicle is two miles from a body of water, it would not be safe to move forward ten miles. Therefore, the first command would not be validated as allowable. If the first command is move forward one mile, and the GPS location of the vehicle shows no physical barriers within one mile, it would be safe to control the machine control system to move the vehicle forward one mile. Therefore, the first command would be validated as allowable.

[00119] In various embodiments, the step of validating further comprises comparing, by the remote processor, the first command to the terrain output. For example, if the first command is move forward 50 feet, and the terrain output shows a slope that exceeds a set negative grade, for example -10% or more, it would not be safe to move forward 50 feet in heavy off-road vehicles. Therefore, the first command would not be validated as allowable. If the first command is move forward 50 feet, and the terrain output shows no physical limitations or a slope exceeding the set negative grade for the next 50 feet, it would be safe to control the machine control system to move the vehicle forward 50 feet. Therefore, the first command would be validated as allowable. Additionally, the off-road vehicle can comprise an Inertial Measurement Unit (IMU), in communication with the processor, configured to transmit an attitude of the off-road vehicle to the processor. In various embodiments, the attitude of the off-road vehicle can also be compared to the first command. Moreover, in various embodiments, the off-road vehicle can comprise an angular velocity sensor, such as a gyroscopic sensor, for the detection of angular velocity.

[00120] In various embodiments, the step of validating can comprise comparing each of the environmental sensor inputs (GPS location, terrain output, and speed of the vehicle) to validate the first command. In various embodiments, if the first command is not allowable then the processor outputs an alert notification to one or more alert devices. In various embodiments, the alert notification can be an alert to the operator that the first command is not allowable.

[00121] In various embodiments, the method 1200 may comprise transmitting, by the remote processor, the first command to a processor (shown as processor 402 and 502) (step 1208). In various embodiments, the method 1200 may comprise controlling, by the processor, the machine control system in response to the first command. In various embodiments, method 1200 may comprise controlling, by the processor, a tool system (shown as tool systems 108, 208, and 308) in response to the first command.

[00122] In various embodiments, the remote processor outputs a confirmation notification requesting a confirmatory response from the operator to confirm that the first command is correct. The confirmation notification can be displayed by a visual display (shown as visual display 712) or played through a speaker (shown as speaker 714) of the remote system 704. In various embodiments, the method 1100 may comprise receiving, by the remote processor, a confirmatory response through the audio input device or the visual display. In various embodiments, the processor controls the vehicle system or the tool system with the first command in response to the remote processor receiving an affirmative confirmatory response (i.e., yes). [00123] In various embodiments, the remote processor 708 can comprise instructions stored thereon, that in response to execution by the remote processor 708, cause remote processor 708 to perform operations comprising any of the steps and embodiments of method 1200. The method 1200 can be performed by the remote processor 708

[00124] Benefits, other advantages, and solutions to problems have been described herein with regard to specific embodiments. Furthermore, the connecting lines shown in the various figures contained herein are intended to represent exemplary functional relationships and/or physical couplings between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may be present in a practical system. However, the benefits, advantages, solutions to problems, and any elements that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as critical, required, or essential features or elements of the disclosures.

[00125] The scope of the disclosures is accordingly to be limited by nothing other than the appended claims, in which reference to an element in the singular is not intended to mean “one and only one” unless explicitly so stated, but rather “one or more.” Moreover, where a phrase similar to “at least one of A, B, or C” is used in the claims, it is intended that the phrase be interpreted to mean that A alone may be present in an embodiment, B alone may be present in an embodiment, C alone may be present in an embodiment, or that any combination of the elements A, B and C may be present in a single embodiment; for example, A and B, A and C, B and C, or A and B and C. Different cross-hatching is used throughout the figures to denote different parts but not necessarily to denote the same or different materials.

[00126] Systems, methods and apparatus are provided herein. In the detailed description herein, references to “one embodiment”, “an embodiment”, “an example embodiment”, etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described. After reading the description, it will be apparent to one skilled in the relevant art(s) how to implement the disclosure in alternative embodiment

[00127] Furthermore, no element, component, or method step in the present disclosure is intended to be dedicated to the public regardless of whether the element, component, or method step is explicitly recited in the claims. No claim element is intended to invoke 35

U.S.C. 112(f) unless the element is expressly recited using the phrase "'means for.” As used herein, the terms “comprises.” “comprising,” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.