AN AIRCRAFT VISION SYSTEM

Technical Field

The present invention is related to vision systems especially used in airliners.

Prior Art

Augmented Reality systems which are enriched by superimposing various objects on a real image/video via computer graphics systems and change the visual/audio reality perception of a person are already known. When used together with digital maps and 3 dimension model database, augmented reality systems can superimpose the data held in the database, by using the position and angle information of the camera, over the image/video captured from the camera. For example by using the coordinates stored in the database of various buildings, the names of the buildings are superimposed on the images/video at the positions where these buildings are located in accordance with the viewpoint of the camera. In the case that a plurality of users wants to watch images/video from the same fixed camera in such systems, each one of the users are obliged to watch the same image/video and it is not possible for each user to look at a different direction.

In this case as a solution even if an articulated or gimbal mounted camera is used, the field of view of the camera is adjusted accordance to the request of only one user. In order for each user to be able to look at different directions a plurality of rotatable camera systems need to be used; however this situation gets more expensive when there are more than 2 or 3 users. Moreover the camera motions aren't sufficiently fast; and as it comprises a mechanical component, there is a higher risk of breakdown and said camera necessitates maintenance. Another technique used in order to reach this aim nowadays is to use panoramic imaging systems. In such a case a part of the desired panoramic image/video captured can be displayed on the device of a user and according to the field of view at which the panoramic image/video is taken, the superimposing (augmented reality) of images at the field of view which said user wants to display, is carried out. By this means, the panoramic image/video captured from a certain monitoring point can be shared with a plurality of users. The users are not only able to indicate their field of view with various input units (joystick, keyboard etc) but also the field of view of the user can be tracked with a head tracking device, and the images/video can be monitored via imaging systems that are mounted on the headpiece (Helmet/Head Mounted Display). Panoramic imaging systems can only capture 2 dimensioned images/video and 3 dimension visualisation procedures become more difficult. Panoptic cameras formed of sensors that are adjusted to look in various directions, which generally protrude out from a certain point and which are positioned at different angles are present on said systems. These devices are able to capture images/video of a spherical field of view with the certain refreshing rates as 3 dimension or 2 dimension images/video. For example when looked from the centre a video within the scope of a hemisphere around said centre can be obtained as an output. The distance to each point on an image/video can be determined with a panoptic camera and by this means a depth map can be genarated.

When airliners and the number of passengers transported with such airliners are taken into account, it is clear that it is not possible for all passengers to easily look outside due to various reasons. The wings of the aircraft, the size of the windows and the positions of the seats prevent the view of some passengers. The passengers with window seats can only see a limited area, cannot obtain detailed information regarding what they are seeing and even though it is not prohibited to take pictures in the aircraft, they cannot always capture the image/video they desire freely. There are some mapping and information systems in commercial aircrafts however it is difficult for passengers to identify such information with real images. It is obvious that articulated or fixed camera systems cannot solve these problems. Moreover, even though said systems are integrated to a vehicle it will not be possible to capture images in insufficient lighting conditions or bad weather conditions.

In the United States patent document numbered US6559846 a system wherein the data related with each frame of the image captured from a panoramic camera or a video recorder is covered over a determined peripheral model and wherein the user determines a part of the panoramic video to the direction at which he/she wants to look at is described.

In the United States patent document numbered US2011 141254 an augmented reality method wherein an augmented reality image is superimposed over a real image dependent on the three dimensioned position of the imaging device is discussed.

In the International patent document numbered WO2008147561 the imaging of panoramic images and adding notes to said images are described. This system comprises a panoramic projector which captures at least a section of the panoramic image and an imaging unit which displays this. The imaging unit superimposes a three dimentional image over a panoramic image and the panoramic display rotates and matches this image so that it is suitable with the panoramic image.

The United States patent document numbered US2002036649 describes an image adapting device which creates an augmented reality by forming a panoramic image with the images captured from a plurality of cameras, with multi user support, determines the field of view of a user via a tracking device and combines a virtual picture with the panoramic view of the area looked at. Brief Description of the Invention

The aim of this invention is to provide an aircraft vision system wherein all of the passengers and crew in an aircraft can see outside and watch any direction independently from each other with augmented reality and with digital map support in real time.

Another aim of this invention is to provide an aircraft vision system with which passengers can watch, even when there is no visibility outside the aircraft (like in bad weather conditions or at night).

Another aim of this invention is to provide an aircraft vision system which enables the passengers to view the images/video of a region without waiting for the aircraft to reach said region and which alerts the passengers when said region is reached.

Another aim of this invention is to provide an aircraft vision system for all of the passengers and the crew to see the other aircrafts that are close by, independent from each other with augmented reality and in real time.

Another aim of this invention is to provide an aircraft vision system wherein a wide angle view can be recorded during flights.

Detailed description of the invention

An aircraft vision system produced in order to reach the aims of the present invention has been illustrated in the attached figures, wherein said figures show the following:

Figure 1 - Is the schematic view of an aircraft vision system subject to the invention. The parts in the figures have been each numbered and the references of said numbers have been given below:

1. Aircraft vision system

2. Main control unit

3. Panoptic camera

4. Positioning system

5. Digital Map database

6. User device

7. Printer

8. Aircraft

9. Image database

10. Panoptic image/video information transmitter

1 1. Ground station

An aircraft vision system (1) comprises:

- At least one panoptic camera positioned outside the aircraft (8) to be able to capture a semi-spherical field of view segment around the installation point,

- At least one positioning system (4) used to determine the position and angles of the aircraft (8),

At least one map database (5) used to accommodate a digital map and contents regarding the positions determined in said map,

- At least one user device (6) which displays the sections of the panoptic camera (3) image/video, that have been requested via tri-axial interfaces to be displayed.,

- At least one main control unit (2) (Figure 1) connected with a panoptic camera (3), positioning system (4), map database (5), imaging database (9), and user device; enabling the communication between said parts (3), (4), (5), (6), (9), which is adapted to carry out the following functions;

- Determining the position and angles of the panoptic camera (3) using the data obtained from the positioning system (4),

Recording the images/video and depth information received from the panoptic camera (3) during the whole journey together with the position and angles of the panoptic camera at that moment, Submitting the requested section of the image/video that has been recorded beforehand to the database (9) or which is captured live to at least one user device (6) by superimposing augmented reality layer ,

Determining at least one region or one point on a map via the user device (6) by the user and carrying out an action related with said region or point that again the user has determined when the aircraft gets close to the region.

In a preferred embodiment of the invention, the panoptic camera (3) is a visible light panoptic camera adapted to capture a whole hemisphere field of view, which is positioned at a point at the bottom of the aircraft (8) to have the widest view possible except for aircraft (8) parts . By this means, the camera can have 360 degrees of a horizontal field of view and 180 degrees of a vertical field of view . In order to have smooth video streaming, at least 30 frames per second of imaging capability is preferred.

In another preferred embodiment the panoptic camera (3) operated at the near infrared range. A panoptic camera which has night vision feature combined with visible light and infrared light is being used in another exemplary embodiment.

The positioning system (4), is pereferably a Global Positioning System (GPS) working together with an Inertial Navigation System (INS). The digital map database (5) comprises data related to air routes, geographical elements like mountains, lakes, rivers, seas, valleys, historical and touristic places, buildings, towns/villages/cities/countries and borders besides landforms and object digital models. All of these elements and pre-defined points/regions can be associated with pictures, videos, audio recordings and information and the associated data are also stored in the map database (5). The stored digital map data may include satellite pictures besides vectoral maps. All of the map elements can be stored in different map layers and can be fetched

independently from the database (5).

The user device (6), has been adapted such that it can request the image/video section that is desired to be displayed and all or any of the augmented reality overlays from the main control unit (2). Namely, a requested section of a wide angled image/video captured with a panoptic camera (3) is processed by the main control unit (2) and submitted to the user device (6).

In a preferred embodiment of the invention, the user device (6) is located within the aircraft (8) while in another embodiment, the device is a portable device adapted suitably and brought by the passenger.

In a preferred embodiment of the invention the section that is requested to be viewed is determined with a joystick, in the user device (6)

In another preferred embodiment of the invention, the part of the image/video that is requested to be viewed is determined in the user device (6) with a head tracker.

In a preferred embodiment of the invention, the section of the image/video that is requested to be viewed in the user device (6) is determined with a motion and position sensor unit which is installed in the display device. This unit is preferably an inertial measurement unit (IMU), an optical or magnetic motion and position sensor unit. In a preferred embodiment of the invention the user device (6) displays the image/video in 2 or 3 dimension and preferably via a monitor mounted behind a seat or via a portable device such as tablet pc. When a portable device is used, the motion and position sensor unit, preferably is an inertial measuring unit (IMU) embedded in the portable device.

In a preferred embodiment of the invention the user device (6) displays 2 or 3 dimensional image/video via a Head mounted display (HDM).

In a preferred embodiment of the invention the user device (6) displays the image/video via a 2 or 3 dimensional dome display device.

The main control unit (2) is responsible for the communication between all of the components (panoptik camera (3), positioning system (4), map database (5), image database (9), image/video transmitter (10) and user device (6). It provides the distribution of the images/video captured from the panoptic camera (3) to the user devices (6) according to requests. In order to superimpose augmented reality overlays correctly over images/video , the position and angles of the panoptic camera (3) need to be known. For this reason, the position and angles of the panoptic camera (3) is calculated in the main control unit (2) using the data obtained from the positioning system (4).

Different images/video can be submitted to various user devices (6) by superimposing augmented reality layer over the chosen sections of the image/video received from the panoptic camera (3). By means of the augmented reality, the objects that cannot be viewed due to weather conditions can be viewed via the digital map. However despite all this, if a user wants to view the real image/video instead of digital map in bad weather conditions, it is possible to use the panoptic camera images/video that have been recorded previously for the same flight. For this reason the main control unit (2) has been adapted to carry out the function of recording the images/video and depth data taken from the panoptic camera (3) throughout the journey to the database (9) together with the position and angles of the panoptic camera (3) at that time. As a result, assuming that the routes of aircrafts are usually the same or very similar to each other, it is possible to display the panoptic camera records for that position belonging to a prior flight in the user device (6).

Besides providing vision with augmented reality in unsuitable weather conditions and at night, even in suitable visibility conditions, recordings belonging to another time frame or season can be enriched by superimposing augmented contents according to the current geographic location information. Even so, by this means, in suitable weather conditions, for example when making a journey during the dayj the night image/video or when travelling in summer the winter image/video can be watched just like a live view. For this reason, the user device (6) has been adapted such that it can request a live or recorded image/video from the main control unit (2). At the same time, the user device (6) has been adapted such that it can request the augmentations that are desired to be superimposed and transparencies of said augmentations.

The superimpositions that can be requested includes air routes; geographical elements such as mountains, lakes, rivers, seas, valleys; historical and touristic places; buildings; towns/villages/cities/countries and borders. Another augmentation that the user device (6) can request is the image/video and information of other aircrafts that are close by. The determination of the locations of aircrafts that are close by can be easily applied to aircrafts that have been equipped with ADS-B transmitters. The flight information broadcasted by aircrafts is received by ground stations, combined and sent to airplanes. Received data is used to calculate the position and angles of closeby airplanes. The calculation are used to display some information about those airplanes on live images/video as an augmented reality object, at the position where the aircraft is really located. in a preferred embodiment of the invention, an image/video transmitter has been adapted such that it shall send the image/video, location and angles of the panoptic camera (3) from the aircraft (8) to an independent ground station (1 1) wirelessly.

In a preferred embodiment of the invention, a printer (7) connected to the main control unit (2) has been adapted such that it shall print out an image displayed in a unit (6).

In a preferred embodiment of the invention the main control unit (2), has been adapted such that the user is able to determine at least a point or a region on a map via the user device (6) and when the aircraft (8) enters into any of the determined regions, or when the aircraft comes as close to the point as determined by the user, the device shall carry out a function to realize an action related to said region or point determined by the user. Preferably this action can be an audio and/or visual alarm, automatically capturing a photograph or video of that region, turning on the display device or printing an output from the priner (7). By this means, the user does not need to wait to reach a zone or point of intereset. He/she can take the photo/video of that region automatically or can be warned when said region is reached. In a preferred embodiment, the action can only be carried out when there is an open view (suitable weather/lighting conditions) in accordance with the user preferences.

The main control unit (2), has been adapted such that all of the crew and/or passengers can place marks on the image/video via user devices (6) and record said markings by associating them to the related position. By this means, for example the objects which cannot be identified by the passengers when seen from the sky, can be defined by other users by displaying the markson their screens. This function can be used for example by crew to define objects as a tour guide to the passengers or in order to create an interactive environment amongst passengers. Within the scope of this basic concept, it is possible to develop many various applications of the aircraft vision system (1) subject to the invention; wherein the invention here cannot be limited to the examples described herein and is principally as explained in the claims.