CONNECTIVITY

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. application number 62/851,921 filed on May 23, 2019, the entire contents of which are hereby incorporated by reference in its entirety.

TECHNICAL FIELD

Example embodiments generally relate to wireless communications and, more particularly, relate to providing augmented reality services to pilots using real time connectivity that can only be provided by a high bandwidth, bi-directional wireless communication link.

BACKGROUND

A flight management system (FMS) is a computer system that is onboard an aircraft, and is specially designed to automate certain tasks in order to make the workload of the flight crew more manageable. In particular, if given a flight plan, the FMS can use sensors to determine aircraft position (along with other flight parameters) and guide the aircraft along the flight plan. In a typical situation, the FMS includes a flight management computer (FMC) and a control display unit (CDU), which is provided in the aircraft for user interface capability. The FMC manages various other components of the FMS including navigational components, flight and instrument displays, flight control systems, engine and fuel systems, data link, etc. The FMC therefore provides the primary means of controlling functions associated with navigation, flight planning, route guidance, trajectory prediction, etc.

Controlling these functions typically requires interaction with various databases associated with navigation, basic operations and engine/aircraft performance data.

The database associated with navigation is called a navigation database, orNDB. The NDB is used for building and processing flight plans. The NDB (like the other databases) is stored in the FMC on a read-only memory device that is updated via a data loader. The data stored in the NDB (i.e., NDB data or information) includes waypoints, airways, runway information, holding patterns, and numerous other important aids to navigation and instructions.

As can be appreciated from the descriptions above, the FMS operates using real time information that is generated in situ, on the aircraft, and incorporates that locally gathered real time information with the static, pre-stored information to drive informational displays and operate or provide guidance for various control surfaces of the aircraft. This combination of real time information that can be locally generated with mainly static information regarding things external to the aircraft has worked reasonably well for many years. While sufficient for minimum operational and safety requirements, this paradigm could certainly use improvement.

The main problem limiting improvement has not been the imagination of those in the aviation industry. Instead, the main problem has been bandwidth. In this regard, the ability to communicate information to the aircraft in real time and, even more limiting, the ability to receive information from the aircraft in real time, has been stifling relative to the universe of what services can realistically be brought into use in the cockpit. Accordingly, some example embodiments provide tools by which to improve this situation dramatically.

BRIEF SUMMARY OF SOME EXAMPLES

Some example embodiments may provide a communication structure capable of delivering a strong bi-directional wireless communication link. The bi-directional communication link may enable additional real time information generated external to the aircraft to be incorporated with locally generated information to deliver a vast array of augmented reality services that can improve both safety and operational performance for aircraft.

In one example embodiment, a system for improving cockpit operations in an aircraft is provided. The system includes an airborne augmented reality module and a ground-based information management module. The augmented reality module may be disposed in the cockpit of the aircraft and may include processing circuitry configured to generate an augmented reality display in the cockpit based at least in part on selected external environmental data received. The information management module may be configured to include external environmental data received from a plurality of sources and generate the selected external environmental data for communication to the augmented reality module via a wireless communication network while the aircraft is in flight. The selected external environmental data may be selected based on aircraft location and a determination of temporal relevance and locational relevance of the selected external environmental data. BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

Having thus described the invention in general terms, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and wherein:

FIG. 1 illustrates a block diagram of a system in accordance with an example embodiment;

FIG. 2 illustrates a block diagram of an information management module in accordance with an example embodiment;

FIG. 3 illustrates a block diagram of an ARM in accordance with an example embodiment;

FIG. 4 illustrates image data overlaid in accordance with an example embodiment; and

FIG. 5 illustrates image data having numerous indicators overlaid thereon according to an example embodiment.

DETAILED DESCRIPTION

Some example embodiments now will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all example embodiments are shown. Indeed, the examples described and pictured herein should not be construed as being limiting as to the scope, applicability or configuration of the present disclosure.

Rather, these example embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like reference numerals refer to like elements throughout. Furthermore, as used herein, the term“or” is to be interpreted as a logical operator that results in true whenever one or more of its operands are true. As used herein, operable coupling should be understood to relate to direct or indirect connection that, in either case, enables functional interconnection of components that are operably coupled to each other.

As used in herein, the term "module" is intended to include a computer-related entity, such as but not limited to hardware, firmware, or a combination of hardware and software (i.e., hardware being configured in a particular way by software being executed thereon). For example, a module may be, but is not limited to being, a process running on a processor, a processor (or processors), an object, an executable, a thread of execution, and/or a computer. By way of example, both an application running on a computing device and/or the computing device can be a module. One or more modules can reside within a process and/or thread of execution and a module may be localized on one computer and/or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures stored thereon. The modules may communicate by way of local and/or remote processes such as in accordance with a signal having one or more data packets, such as data from one module interacting with another module in a local system, distributed system, and/or across a network such as the Internet with other systems by way of the signal. Each respective module may perform one or more functions that will be described in greater detail herein. However, it should be appreciated that although this example is described in terms of separate modules corresponding to various functions performed, some examples may not necessarily utilize modular architectures for employment of the respective different functions. Thus, for example, code may be shared between different modules, or the processing circuitry itself may be configured to perform all of the functions described as being associated with the modules described herein. Furthermore, in the context of this disclosure, the term "module" should not be understood as a nonce word to identify any generic means for performing functionalities of the respective modules. Instead, the term "module" should be understood to be a modular component that is specifically configured in, or can be operably coupled to, the processing circuitry to modify the behavior and/or capability of the processing circuitry based on the hardware and/or software that is added to or otherwise operably coupled to the processing circuitry to configure the processing circuitry accordingly.

Some example embodiments described herein provide a system, architectures and/or methods for improved real time services provision to aircraft by leveraging bi-directional communication links. In this regard, some example embodiments may provide a system that provides reliable, continuous and real-time connectivity to aircraft so that real time information can be incorporated into augmented reality applications running in the cockpit.

In some cases, models of airports and/or airspace can be generated onto vision displays in the cockpit, and the models can be overlaid with data of various kinds. By providing a high level of reliable connectivity to the aircraft, the provision of augmented reality applications in the cockpit can add safety in ways not previously possible (e.g., generating visible

representations of wake vortexes and their movement), and improve the nature of other safety and operational improvement offerings (e.g., by providing overlays on vision displays, providing real time synthetic vision and/or providing interaction through voice recognition technology) for improved guidance through connectivity.

FIG. 1 illustrates a block diagram of various components of a system, which may include one or more wireless communication networks that may be employed to

communicate with an aircraft 100 according to an example embodiment. In this regard, as shown in FIG. 1, a terrestrial network 110, an ATG network 120 and a satellite network 130 are each represented. However, it should be appreciated that example embodiments could be employed with only one such network, with two of the networks, or even with other networks capable of communication with the aircraft 100.

As shown in FIG. 1, each of the wireless communication networks may include wireless access points (APs) that include antennas configured for wireless communication. Thus, for example, the terrestrial network 110 may include a first terrestrial AP 112 and a second terrestrial AP 114, each of which may be base stations, among a plurality of geographically distributed base stations that combine to define the coverage area for the terrestrial network 110. The first and second terrestrial APs 112 and 114 may each be examples of terrestrial base stations that are placed adjacent to each other to provide coverage in overlapping cells that each extend outwardly from the respective base stations in substantially all directions. Thus, the terrestrial base stations may provide a constant layer of coverage near the ground and up to a maximum altitude.

The first and second terrestrial APs 112 and 114 may each be in communication with the terrestrial network 110 via a gateway (GTW) device 116. The terrestrial network 110 may further be in communication with a wide area network such as the Internet 115, Virtual Private Networks (VPNs) or other communication networks. In some embodiments, the terrestrial network 110 may include or otherwise be coupled to a packet-switched core or other telecommunications network. Thus, for example, the terrestrial network 110 may be a cellular telephone network (e.g., a 4G, 5G, LTE or other such network).

The ATG network 120 may similarly include a first ATG AP 122 and a second ATG AP 124, each of which may be base stations, among a plurality of geographically distributed base stations that combine to define the coverage area for the ATG network 120. The first and second ATG APs 122 and 124 may each be in communication with the ATG network 120 via a GTW device 126. The ATG network 120 may also be in communication with a wide area network such as the Internet 115, VPNs or other communication networks. In some embodiments, the ATG network 120 may also include or otherwise be coupled to a packet-switched core or other telecommunications network. Thus, for example, the ATG network 120 may be a network that is configured to provide wireless communication to airborne assets and may employ 4G, 5G, LTE and/or other proprietary technologies. The ATG network 120 may include base stations that define a coverage area substantially above a minimum altitude, which may or may not overlap with the maximum altitude defined by the terrestrial network 110. Moreover, in some cases, the ATG network 120 may be configured to employ beamforming technology that involves either steering narrow beams between the aircraft 100 and the base stations of the ATG network (e.g., the first and second ATG APs 122 and 124) or forming selected ones of a plurality of fixed beams that are each oriented in adjacent and overlapping areas to define full zones of coverage that also overlap. In an example embodiment, the ATG network 120 may be configured to employ unlicensed band frequencies to massively increase the bandwidth capability of the ATG network 120 beyond that of licensed band communications. Moreover, the ATG network 120 may be

bidirectional in nature, such that high bandwidths and low latencies can be achieved in both directions. For example, the ATG network 120 may be capable of delivering download speeds (to the aircraft 100) of greater than 4 Mbps and an upload speed (from the aircraft 100) of greater than 1 Mbps along with latency of less than 100 ms and a jitter of less than 10,000 ms.

The satellite network 130 may include one or more ground stations and one or more satellite access points 132. The satellite network 130 may employ Ka band, Ku band, or any other suitable satellite frequencies/technologies (including Low Earth Orbit (LEO) to provide wireless communication services to the aircraft 100 either while in-flight, or on the ground. Although the satellite network 130 may have good download speeds, upload speeds can be poorer, and latency will be a significant problem for orbits above LEO. Accordingly, the ATG network 120 may be preferred to satellite communications via the satellite network 130 whenever the ATG network 120 is accessible (e.g., due to altitude limitations). However, the satellite network 130 may be a reliable or useful alternative to the ATG network 120 in some cases. Additionally, in some examples it may be desirable to have multiple networks available for establishing redundant sources of connectivity. Redundancy of communication networks may increase the overall reliability and capability of the system.

As shown in FIG. 1, an information management module 150 may be disposed at a location accessible to one or more of the networks. Thus, for example, the information management module 150 may be operably coupled to the Internet 115. However, the information management module 150 may be disposed at a particular one of the networks (e.g., the ATG network 120) in some cases. The information management module 150 may be configured to provide external environmental data (e.g., weather information, wind speed/direction, wind shear data, precipitation information, wake vortex information, animal activity (e.g., animals on or near a runway, or in flight near airports), aircraft traffic information, various special interest-related information, etc.) to an augmented reality module (ARM) 160 disposed on the aircraft 100. Moreover, it should be appreciated that the information management module 150 may be configured to communicate (including simultaneously) with many aircraft and with many individual instances of ARMs on each respective one of the aircraft. Thus, the information management module 150 may be configured to supply external environmental data to many aircraft associated with a particular airline, or with multiple different airlines, as described herein. Then, the ARM of each respective aircraft can use the external environmental data as described herein for driving outputs at the ARM 160 while the aircraft are in-flight.

An example structure for the information management module 150 of an example embodiment is shown in the block diagram of FIG. 2. In this regard, as shown in FIG. 2, the information management module 150 may include processing circuitry 210 configured to perform data processing, control function execution and/or other processing and management services according to an example embodiment of the present invention. In some

embodiments, the processing circuitry 210 may be embodied as a chip or chip set. In other words, the processing circuitry 210 may comprise one or more physical packages (e.g., chips) including materials, components and/or wires on a structural assembly (e.g., a baseboard).

The structural assembly may provide physical strength, conservation of size, and/or limitation of electrical interaction for component circuitry included thereon. The processing circuitry 210 may therefore, in some cases, be configured to implement an embodiment of the present invention on a single chip or as a single“system on a chip.” As such, in some cases, a chip or chipset may constitute means for performing one or more operations for providing the functionalities described herein.

In an example embodiment, the processing circuitry 210 may include one or more instances of a processor 212 and memory 214 that may be in communication with or otherwise control a device interface 220. As such, the processing circuitry 210 may be embodied as a circuit chip (e.g., an integrated circuit chip) configured (e.g., with hardware, software or a combination of hardware and software) to perform operations described herein. In some embodiments, the processing circuitry 210 may communicate with various internal and/or external components, entities, modules and/or the like, e.g., via the device interface 220. The processing circuitry 210 may also communicate with one or more instances of the ARM 160 via one or more of the networks of FIG. 1 (and more specifically, via an antenna assembly andradio that is configured to wirelessly interface with an aircraft via the corresponding one of the networks).

The device interface 220 may include one or more interface mechanisms for enabling communication with other internal and/or external devices (e.g., modules, entities, sensors and/or other components of the networks and/or aircraft). In some cases, the device interface 220 may be any means such as a device or circuitry embodied in either hardware, or a combination of hardware and software that is configured to receive and/or transmit data from/to modules, entities, sensors and/or other components of the networks and/or aircraft that are in communication with the processing circuitry 210. In this regard, for example, the device interface 220 may be configured to operably couple the processing circuitry 210 to a distribution module 250, and/or to various external entities (including flight control entities, aircraft themselves, weather services, etc.) that may provide various types of information including traffic information 260, weather information 270, and numerous other types of information associated with specific interests (i.e., special interest information 280) with respect to which operation of the aircraft 100 may be managed, optimized or otherwise considered. These other types of special interest information 280 may include carbon management activities, rule violations, noise management issues, predictive maintenance, animal activity, logistics and management information and/or the like.

The processor 212 may be embodied in a number of different ways. For example, the processor 212 may be embodied as various processing means such as one or more of a microprocessor or other processing element, a coprocessor, a controller or various other computing or processing devices including integrated circuits such as, for example, an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), or the like. In an example embodiment, the processor 212 may be configured to execute instructions stored in the memory 214 or otherwise accessible to the processor 212. As such, whether configured by hardware or by a combination of hardware and software, the processor 212 may represent an entity (e.g., physically embodied in circuitry - in the form of processing circuitry 210) capable of performing operations according to embodiments of the present invention while configured accordingly. Thus, for example, when the processor 212 is embodied as an ASIC, FPGA or the like, the processor 212 may be specifically configured hardware for conducting the operations described herein. Alternatively, as another example, when the processor 212 is embodied as an executor of software instructions, the instructions may specifically configure the processor 212 to perform the operations described herein.

In an example embodiment, the processor 212 (or the processing circuitry 210) may be embodied as, include or otherwise control the operation of the distribution module 250.

As such, in some embodiments, the processor 212 (or the processing circuitry 210) may be said to cause each of the operations described in connection with the distribution module 250. The processor 212 may also control function execution and instruction provision related to operations of the distribution module 250 based on execution of instructions or algorithms configuring the processor 212 (or processing circuitry 210) accordingly. In particular, the instructions may include instructions for determining which information and/or services to provide to the ARM 160 of a particular aircraft, and then the provision of such information and/or services.

In an exemplary embodiment, the memory 214 may include one or more non- transitory memory devices such as, for example, volatile and/or non-volatile memory that may be either fixed or removable. The memory 214 may be configured to store information, data, applications, instructions or the like for enabling the processing circuitry 210 to carry out various functions in accordance with exemplary embodiments of the present invention. For example, the memory 214 could be configured to buffer input data for processing by the processor 212. Additionally or alternatively, the memory 214 could be configured to store instructions for execution by the processor 212. As yet another alternative, the memory 214 may include one or more databases that may store a variety of data sets associated with the traffic information 260, weather information 270 and/or special interest information 280 that is provided for storage prior to ultimate distribution by the distribution module 250. Among the contents of the memory 214, applications and/or instructions may be stored for execution by the processor 212 in order to carry out the functionality associated with each respective application/instruction.

The distribution module 250 may be configured to receive the traffic information 260, weather information 270 and/or special interest information 280, and store such information in a way that enables specific portions of such information to be requested, disseminated, updated, and/or the like. In an example embodiment, the distribution module 250 may also be configured to store all information in association with a respective location (in 2D and/or 3D space) and time. Thus, for example, each piece of external environmental data received by the distribution module 250 may be stored in association with respective geographic areas, airports, specific locations in airspace or on the ground, and a corresponding time at which the information was received so that, for example, the distribution of information to specific ARMs 160 and corresponding aircraft can be made on the basis of temporal and locational relevance (for current time or for future time, based on trajectory information).

In some examples, some or all of the information stored in the information management module 150 may be stored in association with confidence ratings so that any information having temporal and locational relevance may further be considered in light of a confidence score or rating associated with the information as well. In some cases, the confidence score or rating may be generated or change based on the age of the information, or based on knowledge of the speed at which such information decays or changes in relevance over time. For example, information about lightning strikes at a particular location may be relevant to the particular location only for a time at which the corresponding thunderstorm is in the area. Speed of movement of a weather front or storm cell (if known), may be used to generate a confidence rating for the information about lightning strikes that causes the confidence rating to be high for a certain period of time (e.g., a half hour), but then decay to the point at which the information becomes irrelevant or not useful (e.g., four hours later). Thus, weather information and certain types of traffic information can be augmented based on additional knowledge (e.g., of system or object movement and speed), or based on average or estimated speeds for movement when specific information is lacking. A starting confidence score or rating associated with various types of information may therefore be lower when less additional knowledge is available (and estimates must be used). In any case, the ARM 160 may use the confidence score to determine when and how to represent certain types of information (as discussed in greater detail below) such that, for example, the confidence score could determine color, intensity or other aspects of how a particular feature is represented, but the confidence score could also determine whether a particular feature is represented at all. In this regard, various thresholds could be defined for the confidence score with respect to how the feature is represented or whether the feature is represented at all.

In an example embodiment, the information module 150 may be configured to attempt to gather data at certain confidence intervals that are defined or based on the type of information associated therewith. For example, traffic information may have a very short confidence interval such that new information must be received and supplied to aircraft in real time or at least within minutes to several minutes in order to have any relevance due to the speed at which aircraft move (in the air and on the ground) relative to their relevance to any particular location. Weather information may have a longer confidence interval (e.g., quarter or half hour intervals) given the general size and slow speed of movement of weather systems. The confidence intervals associated with various types of the special interest information 280 may vary widely with the type of information. In some cases, data of various types that is collected from or in the aircraft 100 may also be gathered, replaced or reported at intervals that also depend upon confidence ratings associated with their accuracy. In such examples, each different type of data may have a corresponding different confidence rating or confidence interval.

In some cases, confidence ratings may be dynamic and be allowed to decay over time, as mentioned above. However, in other cases, a confidence rating associated with any particular piece of information may remain static and the corresponding piece of information may only be replaced or updated when better information is received. Thus, for example, certain types of information may be updated in the information management module 150 only when the confidence score or rating of new information received is higher than the confidence score or rating of the information being updated or replaced.

In some cases, the distribution module 250 may receive information indicative of aircraft location (or requests from aircraft for external environmental data) for a plurality of aircraft. The distribution module 250 may be configured to determine which portions of the traffic information 260, weather information 270 and/or special interest information 280 have sufficient (e.g., relative to a threshold) temporal and locational relevance to the aircraft based on the respective locations of the aircraft. In some cases, a temporal or locational relevance threshold may be defined to guide determinations as to which information to send to which respective aircraft. Similarly, a confidence rating threshold may dictate or influence which information is sent to or from respective aircraft as well. As such, any or all of the temporal relevance, locational relevance, and the confidence score of the information stored at the information management module 150 may be used by the distribution module 250 in order to determine which portions of the traffic information 260, weather information 270 and/or special interest information 280 to send to respective aircraft (and their respective ARMs 160), or from those aircraft to data systems on the ground.

The distribution module 250 may be configured to interface with the ARM 160 to distribute (e.g., wirelessly via one of the networks of FIG. 1) data or content from the information management module 150 to the ARM 160 of one or more instances of the aircraft 100. In some cases, the distribution module 250 may communicate specific information intentionally to the aircraft for which the information is most relevant (as described above). However, in some models, all information having sufficient temporal relevance to a very large region (e.g., a country or continent) may be broadcast to all aircraft in the region. The broadcast model may lower the processing and decision making load on the distribution module 250 and substantially simplify decision making on that end.

However, as will be seen later, broadcasting information would simply shift the processing burden to the air-side, which may be disadvantageous in some cases. Thus, instead of broadcast communications to aircraft in general, the communications between the information management module 150 and the ARM 160 may be direct and targeted in some cases. This targeted communication may be both enabled and enhanced by the fact that the ATG network 120 of some example embodiments may be configured to utilize beamforming technology to form narrow beams between the aircraft 100 and the base stations of the ATG network (e.g., the first and second ATG APs 122 and 124). The narrow beams that are formed directly between the aircraft 100 and the base stations may also enhance security since the beams are formed with knowledge of the location of the aircraft 100 relative to the corresponding base station that is currently serving the aircraft 100. Thus, the aircraft 100 may receive only information deemed to be pertinent (e.g., having sufficient locational relevance, temporal relevance and confidence scores) to the aircraft 100, and the ARM 160 of the aircraft 100 may have a lower computational load as a consequence. Given that weight and size restrictions are lower on the ground, that power is readily available, and that one unit can service multiple aircraft, this paradigm may be favored in some embodiments.

FIG. 3 illustrates a block diagram of various components of the ARM 160 of an example embodiment. In an example embodiment, the ARM 160 may include processing circuitry 310 configured to perform data processing, control function execution and/or other processing and management services according to an example embodiment of the present invention. In some embodiments, the processing circuitry 310 may be embodied as a chip or chip set. In other words, the processing circuitry 310 may comprise one or more physical packages (e.g., chips) including materials, components and/or wires on a structural assembly (e.g., a baseboard). The structural assembly may provide physical strength, conservation of size, and/or limitation of electrical interaction for component circuitry included thereon. The processing circuitry 310 may therefore, in some cases, be configured to implement an embodiment of the present invention on a single chip or as a single“system on a chip.” As such, in some cases, a chip or chipset may constitute means for performing one or more operations for providing the functionalities described herein.

In an example embodiment, the processing circuitry 310 may include one or more instances of a processor 312 and memory 314 that may be in communication with or otherwise control a device interface 320. As such, the processing circuitry 310 may be embodied as a circuit chip (e.g., an integrated circuit chip) configured (e.g., with hardware, software or a combination of hardware and software) to perform operations described herein. In some embodiments, the processing circuitry 310 may communicate with various components, entities and/or sensors of the aircraft 100, e.g., via the device interface 220.

Thus, for example, the processing circuitry 310 may communicate with a sensor network 330 of the aircraft 100 to receive information from sensors on flight control surfaces, one or more cameras, equipment and environmental sensors, and/or the like. The processing circuitry 310 may also communicate with the information management module 150 via one of the networks (and more specifically, via an antenna assembly andradio that is configured to wirelessly interface with the corresponding one of the networks).

The device interface 320 may include one or more interface mechanisms for enabling communication with other internal and/or external devices (e.g., modules, entities, sensors and/or other components of the networks and/or aircraft). In some cases, the device interface 320 may be any means such as a device or circuitry embodied in either hardware, or a combination of hardware and software that is configured to receive and/or transmit data from/to modules, entities, sensors and/or other components of the networks and/or aircraft that are in communication with the processing circuitry 310. In this regard, for example, the device interface 320 may be configured to operably couple the processing circuitry 310 to the flight management system (FMS) 340 of the aircraft 100, a user interface 350, an overlay module 360, a rules module 370, a voice recognition (VR) module 380, and/or an

optimization module 390 along with the information management module 150.

The processor 312 may be embodied in a number of different ways. For example, the processor 312 may be embodied as various processing means such as one or more of a microprocessor or other processing element, a coprocessor, a controller or various other computing or processing devices including integrated circuits such as, for example, an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), or the like. In an example embodiment, the processor 312 may be configured to execute instructions stored in the memory 314 or otherwise accessible to the processor 312. As such, whether configured by hardware or by a combination of hardware and software, the processor 312 may represent an entity (e.g., physically embodied in circuitry - in the form of processing circuitry 310) capable of performing operations according to embodiments of the present invention while configured accordingly. Thus, for example, when the processor 312 is embodied as an ASIC, FPGA or the like, the processor 312 may be specifically configured hardware for conducting the operations described herein. Alternatively, as another example, when the processor 312 is embodied as an executor of software instructions, the instructions may specifically configure the processor 312 to perform the operations described herein.

In an example embodiment, the processor 312 (or the processing circuitry 310) may be embodied as, include or otherwise control the operation of the overlay module 360, the rules module 370, the voice recognition module 380, and/or the optimization module 390. As such, in some embodiments, the processor 312 (or the processing circuitry 310) may be said to cause each of the operations described in connection with the overlay module 360, the rules module 370, the voice recognition module 380, and/or the optimization module 390. The processor 312 may also control function execution and instruction provision related to operations of the overlay module 360, the rules module 370, the voice recognition module 380, and/or the optimization module 390 based on execution of instructions or algorithms configuring the processor 312 (or processing circuitry 310) accordingly.

In an exemplary embodiment, the memory 314 may include one or more non- transitory memory devices such as, for example, volatile and/or non-volatile memory that may be either fixed or removable. The memory 314 may be configured to store information, data, applications, instructions or the like for enabling the processing circuitry 310 to carry out various functions in accordance with exemplary embodiments of the present invention. For example, the memory 314 could be configured to buffer input data for processing by the processor 312. Additionally or alternatively, the memory 314 could be configured to store instructions for execution by the processor 312. In some cases, the memory 314 may also store destination data 395 that may be provided pre-flight or in-flight. In some cases, the destination data 395 may include information provided by the information management module 150 while the aircraft 100 is in-flight. However, the destination data 395 may also be pre-loaded information modules that are loaded based on the flight plan or destination of the aircraft 100 on a given flight. In any case, the memory 314 may store applications or instructions that, when executed by the processor 312 enable launching and execution of software programs that perform various functions including, for example, several of the functions described herein.

The ARM 160 may, in some cases, be operably coupled to the FMS 340 to allow integration of the operations of the ARM 160 with the FMS 340. However, in other cases, the ARM 160 may operate separate from the FMS 340. As such, the ARM 160 could be a separate advisory or informational services platform for the pilot or crew. However, in other cases, the ARM 160 could be an input to, or even receive outputs from the FMS 340 to facilitate incorporation of the information associated with the ARM 160 into automatically generated flight controls and/or autopilot functions of the aircraft 100.

The ARM 160 may further include or be operably coupled to the user interface 350, which may be in communication with the processing circuitry 310 to receive an indication of a user input at the user interface 350 and/or to provide an audible, visual, mechanical or other output to the user (i.e., pilot or crew member). As such, the user interface 350 may include, for example, one or more instances of a keyboard, display, levers, switches, indicator lights, touchscreens, microphone, buttons or keys (e.g., function buttons), and/or other input/output mechanisms. In some cases, the user interface 350 may include a vision system (e.g., heads up display, goggles or another head mounted display that can be worn by the pilot or a crew member) to generate an immersive augmented reality environment that is visible to the pilot or crew member through the vision system. In some embodiments, the vision system may generate a synthetic vision system with a display of the vision system displaying a camera view taken from (or similar to) the perspective from the cockpit. The synthetic vision system created may essentially display that camera view with other information overlaid on top by the overlay module 360. Moreover, it should be appreciated that the overlay module 360 may be configured to overlay graphical representations of various different pieces of information on top of a camera view, a map view, or various other views that can be generated on the user interface 350 (or via the vision system).

In an example embodiment, the overlay module 360 may be configured to modify image data provided to a display to overlay indicia, graphical representations, or various other enhancements over the image data. In some cases, the image data may be stored image data (e.g., of map views or captured images of places or things) that can be oriented relative to the heading of the aircraft 100 to prevent confusion for the pilot or crew. Meanwhile, additionally or alternatively, the image data may be real-time (or near real-time) image data captured by a camera forming a portion of the sensor network 330 or otherwise provided on the aircraft 100.

When the image data is a map view, a position of the aircraft 100 may be displayed on the image data by the overlay module 360 along with various other pieces of information or enhancements. For example, an intended track may be displayed along with various graphical representations of individual items of external environmental data. Each of the graphical representations may be distinct in terms of labeling, colors, intensity, icons used, or other differentiating details. Some examples of graphical representations may include weather information or data, wake vortex data, traffic data (e.g., the locations and trajectories of other aircraft or objects), animal activity (e.g., animals on or near a runway, or in flight near airports), flight path optimization data (in cooperation with the optimization module 390 as discussed below), or the like.

When the image data is a camera view, live image data may be presented on a display and the live image data may be modified by the overlay module 360 to include graphical representations as discussed above. In an example embodiment, the overlay module 360 may be configured to drive a display based on a the live image data (e.g., a real time camera view) of an area visible from the cockpit, and generate driving directions on the display.

Thus, for example, as the aircraft is taxiing, signals for turning right or left (or proceeding ahead) on the taxiway may be provided in the cockpit on a display. Synthetic vision versions may also overlay the image data with images or representations of other aircraft or objects (e.g., from the external environmental data) onto the synthetic vision display generated based on the camera view. Camera or map views could also be overlaid with information indicative of restricted or authorized airspace. Warnings (audible and visual) may be provided based on proximity to restricted airspace, and when boundaries are crossed. Instructions for exiting the restricted airspace may also be automatically provided. Additionally, any of the graphical representations that can be provided on map views can also be provided on camera views as an overlay.

Wake vortex data may be generated based on information indicative of the type of aircraft generating the wake vortex and the speed, heading, altitude, etc. of the aircraft. This information may be reported by the aircraft to the information management module 150, and then communicated to other aircraft in the area (and therefore having temporal and locational relevance) to display an overlay showing the wake vortex graphically on a map or camera view. The ARM 160 may therefore be configured to model a wake vortex based on the information above, but also model movement of the wake vortex based on wind direction and strength or other environmental factors that may be included in the external environmental data.

In an example embodiment, the rules module 370 may be used to define rules for how to generate representations for certain phenomena or situations based on models and/or aviation (e.g., FAA rules). For example, the rules module 370 may include information on airspace management rules defined by the FAA so that warnings can be issued regarding restricted airspace. The rules module 370 may also include models for modeling wake vortexes and their movement. In some cases, the rules module 370 may further include information on noise restriction and models for predicting noise generation based on aircraft flying through certain airspace and environmental conditions. Various rules regarding noise generation (again, determined by the FAA) may then be enforced or at least aircraft can be notified of such rules. Accordingly, the rules module 370 may be configured to store models and/or rule sets that can be used to generate information for display as an overlay by the overlay module 360. In some cases, the rules module 370 may also allow operator input (or have predefined rules) for selection of preferred models for certain locations or situations. Thus, for example, an ability to source data from among various options or models may be provided so that weather data, location information, or other information can be obtained from the best sources at any given time or under certain circumstances. In an example embodiment, the ARM 160 may include the voice recognition module 380, which may interface with any or all of the other modules of the ARM 160. The voice recognition module 380 may be configured to receive (e.g., via a microphone of the user interface 350) instructions from a pilot or crewmember of the aircraft 100. The voice recognition module 380 may therefore allow audible commands or inputs to be received and input into the ARM 160 so that the pilot, for example, can have hands free (relative to the ARM 160) communication with the ARM 160 to free his/her hands up for other flight related tasks.

The optimization module 390 may be configured to receive information from the external environmental data and determine optimal flight path or routing details to implement or suggest in order to optimize certain selectable parameters. For example, the optimization module 390 may be configured to determine optimal routes to minimize fuel consumption, to minimize carbon emissions, to manage contrail generation (e.g., providing instructions relative to flying above/below the tropopause and therefore in or out of the stratosphere). In some cases, the optimization module 390 may further be configured to make logistics management optimization determinations. For example, the optimization module 390 may be provided with information on fuel (including fuel cell, and electric charge capacity) capabilities at various airports or landing stations for urban air mobility environments. The ARM 160 may therefore provide instructions to the pilot or crew, via overlaid information on a display, regarding where refueling can be optimally performed and instructions on how to get there. In some cases

In some cases, the destination data 395 (stored at the ARM 160 or provided thereto by the information management module 150) may include models and/or images of airport runways and approaches, models and/or images of taxi ways, models and/or images of gates or hangers, and/or the like. Thus, for example, 3D images of airports and the approached thereto can be provided by the ARM 160. However, the 3D images may be further augmented to include overlaid information of various types. Moreover, the overlaid information may include external environmental data that is supplied to the aircraft 100 in real time and while the aircraft 100 is in-flight. It should also be appreciated that the aircraft 100 may communicate (again in real time) data from the sensor network 330 back to the ground. The real time sensor network information may include camera views from the aircraft 100, and information that may also be useful for generating weather data, traffic data, animal activity data, etc. For example, if camera images from one aircraft reporting live feed to the information management module 150 shows any relevant data to another aircraft (i.e., the aircraft 100) based on locational and temporal relevance, the camera images may be used to extract external environmental data that can be communicated to the aircraft 100 and be overlaid onto display outputs to create an immersive and real time updated augmented reality (AR) environment that is facilitated by the bidirectional nature of the connectivity with all of the aircraft that communicate with the information management module 150. Moreover, this bidirectional link also creates the possibility of a live stream for sensor data off the aircraft and the creation of a real time internet of things (IOT) environment for monitoring the aircraft environment, health of various systems, maintenance situations, and/or the like.

FIG. 4 illustrates an image 400 (which could either be live camera image data or synthetic image data) showing a runway 410 and taxi ways 420. The overlay module 360 may add various overlays to the image 400 including, for example, weather data 430 (e.g., a wind direction and strength indicator) and driving directions 440 (e.g., a turn direction). However, it should be appreciated that any of the other items discussed above could also be added to the image 400, some examples of which are shown in FIG. 5. In this regard, FIG. 5 illustrates image data (e.g., image 500) showing an airport with multiple runways. Guidance information provided via the ARM 160 may include highlighting of a runway assignment 510 (or runway recommendation). The runway assignment 510 could further include an indication of a direction of approach 520, weather information 530 (in the form of wind strength and direction), obstacle or obstruction information 540 (e.g., an animal sighting), traffic indicators 550, and a final destination indicator 560. Each of these indicators may be differentiated by color, shape, labeling, and may move with updated external environmental data at confidence intervals, or when available due to communication from the respective services or entities (e.g., other aircraft) with the information management module 150.

Accordingly, example embodiments may enable high quality information provision into the cockpit, in real time, on the basis of connectivity. Moreover, the information provided can be used to power augmented reality vision systems that make the information easier to consume by adding it to an intuitive interface that is updated in real time.

Thus, in accordance with an example embodiment, a system for improving cockpit operations in an aircraft is provided. The system may include an airborne augmented reality module and a ground-based information management module. The augmented reality module may be disposed in the cockpit of the aircraft and may include processing circuitry configured to generate an augmented reality display in the cockpit based at least in part on selected external environmental data received. The information management module may be configured to include external environmental data received from a plurality of sources and generate the selected external environmental data for communication to the augmented reality module via a wireless communication network while the aircraft is in flight. The selected external environmental data may be selected based on aircraft location and a determination of temporal relevance and locational relevance of the selected external environmental data.

In some embodiments, the system may include additional, optional features, and/or the features described above may be modified or augmented. Some examples of

modifications, optional features and augmentations are described below. It should be appreciated that the modifications, optional features and augmentations may each be added alone, or they may be added cumulatively in any desirable combination. In an example embodiment, the information management module may store the external environmental data in association with respective geographic areas, airports, specific locations in airspace, and specific locations on the ground for determining locational relevance, and may store the external environmental data in association with a corresponding time of collection for determining temporal relevance. In an example embodiment, the selected external environmental data may be further selected based on a confidence score. In some cases, the information management module may be configured to communicate a different set of selected external environmental data to each of a plurality of aircraft based on temporal relevance and locational relevance determinations for each respective one of the aircraft. In an example embodiment, the information management module may be configured to simultaneously communicate the different set of selected external environmental data to multiple ones of the aircraft. In some cases, the selected external environmental data may include weather data relevant to the aircraft based on a current trajectory of the aircraft, and the augmented reality module may be configured to overlay a display in the cockpit with information indicative of the weather data. In an example embodiment, the selected external environmental data may include traffic data relevant to the aircraft based on a current or future location of the aircraft, and the augmented reality module may be configured to overlay a display in the cockpit with information indicative of the traffic data. In some cases, the selected external environmental data may include wake vortex data relevant to the aircraft based on a current trajectory of the aircraft, and the augmented reality module may be configured to overlay a display in the cockpit with a visual indication of a location of a wake vortex based on the wake vortex data. In an example embodiment, the visual indication of the location of the wake vortex may be moved based on a model for approximating wake vortex movement and decay. In some cases, the selected external environmental data may include animal activity data relevant to the aircraft based on animal activity reported proximate to an airport the aircraft is departing or approaching, and the augmented reality module may be configured to overlay a display in the cockpit with visual information indicative of the animal activity data. In an example embodiment, the selected external environmental data may include airspace availability data relevant to the aircraft based on a current trajectory of the aircraft, and the augmented reality module may be configured to overlay a display in the cockpit with information indicative of available airspace in which the aircraft is authorized to fly and an indication of any restricted airspace in which the aircraft is not authorized to fly. In some cases, the selected external environmental data may include carbon or emissions management data relevant to the aircraft, and the augmented reality module may be configured to overlay a display in the cockpit with information indicative of the carbon or emissions management data. In an example embodiment, the selected external environmental data may include logistics management data relevant to the aircraft, and the augmented reality module may be configured to overlay a display in the cockpit with information indicative of the logistics management data. In some cases, the selected external environmental data may include noise management data relevant to the aircraft based on a current trajectory of the aircraft, and the augmented reality module may be configured to overlay a display in the cockpit with information indicative of the noise management data. In an example embodiment, the external environmental data may be gathered at predefined intervals, and the external environmental data gathered may replace previously stored data only when a confidence score of new data exceeds a corresponding confidence score of old data. In some cases, the augmented reality module may be operably coupled to a flight management system of the aircraft, and the augmented reality module may generate a display to indicate suggested guidance for aircraft operation based on information received from the flight management system. In an example embodiment, the augmented reality module may include a voice recognition module, and instructions for display of content via the augmented reality module may be provided by a pilot or crewmember via the voice recognition module. In some cases, the aircraft location may be either a current aircraft location or a future aircraft location. In an example embodiment, the augmented reality module may be configured to generate a display based on a real time camera view of an area visible from the cockpit, and generate an overlay on the display to indicate driving directions for the aircraft.

Many modifications and other embodiments of the inventions set forth herein will come to mind to one skilled in the art to which these inventions pertain having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the inventions are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, although the foregoing descriptions and the associated drawings describe exemplary embodiments in the context of certain exemplary combinations of elements and/or functions, it should be appreciated that different combinations of elements and/or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, different combinations of elements and/or functions than those explicitly described above are also contemplated as may be set forth in some of the appended claims. In cases where advantages, benefits or solutions to problems are described herein, it should be appreciated that such advantages, benefits and/or solutions may be applicable to some example embodiments, but not necessarily all example embodiments. Thus, any advantages, benefits or solutions described herein should not be thought of as being critical, required or essential to all embodiments or to that which is claimed herein. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.