Field of the invention

The present invention relates to a stabilization control system of a sensor device mounted on a stationary or flying platform, and more particularly, it relates to a sensor device control system controlled with a head-tracking system and a remote command unit.

Known solutions

This invention relates in general to a control of stationary or flying platforms, i.e. single or multi- rotor unmanned aircraft systems (UAS) equipped with any sensor system, but primarily for camera system for still photography or video shooting from the air.

Single- and multi-rotor aircraft systems are currently widely used in a number of applications, whereat they come in various configurations depending on their purpose and use.

From US 2011/0017865 Al, a copter comprising of at least four rotors is known where rotors are arranged in such a way that a free field of vision is defined along a longitudinal axis of the aircraft at least between two terminal rotors.

From US 7,658,555 Bl, a stabilizing and vibration-isolating mount is known, which facilitates the operation of a camera from mobile platforms such as aircraft, wherein movement around all three axes of rotation can be stabilized by gyrostabilizers.

In US 2006083501 (Al), an aerial photography apparatus is invented and designed for taking aerial photos with a three hundred and sixty degree rotation possibility.

From CN 101619971 (A), an aero-photography gyro-stabilized platform is known with three degrees of freedom.

From US 2003213868 (Al), a camera system for tracking a target from an aircraft is known, whereat camera head is moveable, and user interface and control circuit track and control the movement of an aircraft and camera.

In US 5995758 (A), a stabilization apparatus for aiming an instrument such as a camera at a target object is described, whereat rotational sensors mounted on the camera platform detect rotation of the platform from an initial position about roll, tilt and pan axes.

From JP2006264568 (A), a helicopter capable of constructing the light fuselage is known, simplifying the drive control system of a camera and suppressing the influence of vibration on the image using simple construction. From CN201002722 (Y), a two-degree-of-freedom image stabilization platform for aircrafts is known, comprising of detection device, action device and image collection device. Invention relates to stabilizing of the lens and compensating aircraft movements.

From CN201287830 (Y), a stabilising bracket for aerial photography camera is known, whereat systems compensates shaking and movement of the aircraft.

From US7658555 (Bl), an aerial video mount is known that is stabilizing and vibration-isolating and facilitates the manual operation of a camera from a mobile platform such as aircraft.

From GN101619971 (A), an aero-photography gyro-stabilized platform with three freedom degrees is known, used for isolating the angular movement and high-frequency vibration.

From US5995758 (A), stabilizing apparatus for aiming an instrument such as camera at a target object is known, whereat rotational rate sensors mounted on the camera platform detect rotation of the platform from an initial position about roll, tilt and pan axes. Detected rotation is counter-acted with the use of torque motors for rotating the platform back to its initial position.

From WO2005114366 (Al), self-referencing head tracker is known, whereat tracker is supplemented with a transmission unit and accumulator into an earpiece.

From EP2428813 (Al), a head tracking system is known for determining a head position of a user and to a sound system generating a virtual surround sound.

From US2009237355 (Al) a tracking device is known for determining position of at least one user relative to a video display, where tracking device has a wearable structure configured to be mounted on a human such as a headset, eyeglasses or arm bands.

From US2011293129 (Al) a head tracking system is known, which determines rotation angle of a head of a user with respect to a reference direction, which is dependent on a movement of a user. From US5138555(A) a system for predicting head movements by an aircrew member is known. From US2008120408 (Al) a system and method for tracking of a head is known, based on acoustical signals.

From US2006146046 (Al) a method of tracking an expected location of a head in a computerized head tracking environment is known.

Critique of known technical solutions

The unmanned aircraft system according to the state of the art has in common that it requires at least two operators, one for the aircraft and its movements in space, and one for camera control and tracking of objects on video Or photography. Consequences of this are higher operating costs of the system and un-harmonized video or photography shooting.

In addition, the aircraft according to the state of the art uses either only one or both object tracking controls, that is _> movement of the aircraft and/or orientation of the camera mount. However, in case of using both tracking controls, the combination of both is neither methodical nor optimized for best video or photography shooting.

In addition, state of the art head-tracking camera control does riot enable combination of proportional and progressive camera head movements depending on the declination from the zero reference point. This kind of control and operation limits flexibility of aerial camera system and makes the whole system less responsive. It also makes it difficult for the operator to follow objects without quick or extreme head movements. This is clearly evident in case of movements in the pitch direction. Normally human head can move 55° from normal horizontal view to extreme upward view. Beyond 55°, the eyes take over and cover the extra 35° of view. However, head-tracking system only detects the 55° movement making such system less flexible in case of overhead film or photography shooting. More advanced systems use eye-tracking systems to accommodate the extra 35° angle, however the need for such a system can be avoided with the use of progressive, i.e. exponential control of the camera by the head-tracking system.

Aim and goal of the invention

Aim of this invention is to provide a stationary and aerial sensor control system that can overcome mentioned disadvantages of the state of the art. Effortless, economical, precise and reliable use of various sensors and especially shooting of video or still photography is achieved by the features described in this patent.

Description

Invention solves the problem of controlling sensor device 4 of the flying platform 1 or stationary platform 8, whether it is a camera, laser pointer, measuring device or similar, with a combination of the following systems:

a) A flying platform 1 (aerial) or a stationary platform 8 (structural);

b) A system for control of the platform (motors 35 with rotors 33, or wings of a fixed-wing

aerial system, in case of flying platform 1, and pulleys, cables, tripod, etc. in case of stationary platform 8); c) Sensor device 4 stabilizing system, i.e. gyro-head 5;

d) Remote command unit 6 for control of the flying platform 1 or stationary platform 8;

e) Head-tracking system 7 for control of the sensor device 4.

This invention relates to the sensor device control system, which can be attached to a flying platform 1 or stationary platform 8 and can be used for any service, including video or still photography, surveillance, geo-mappirig, laser marking, or similar. General characteristics are common for both platforms and are thus described below. Embodiments related to a use of the system in two example platforms are presented thereafter.

Sensor device stabilizing system, i.e. gyro-head

Sensor device 4 is positioned on the gyro-head 5 stabilizing system. Gyro-head 5 enables sensor device 4 movements around all three axes: pitch 10, roll 11 and yaw 12. All three rotations are enabled with bearings 34 positioned inside the gyro-head 5. Gyro-head 5 is designed so that all three axes, i.e. pitch 10, roll 11 and yaw 12 axis, cross in the centre of gravity 40 of the serisor device 4 and its mount. Such design minimizes loads on the actuators 15, which control the sensor device 4 movements, and at the same time increases responsiveness of the system due to a change in orientation.

Stabilizing system, i.e. gyro-head 5 incorporates actuators 15 and gears 16 which are designed so that maximum speed of rotation is around 70° per second. Gyro-head 5 actuators 15 use feedback control loop, which enables very accurate and smooth stabilization of the sensor device 4, because system actively adjusts velocity and torque of the motor 15 according to requirements.

Gyro-head 5 also incorporates transmission module 14 for signal transmission from the stabilized gyro-head 5 to the remote command unit 6.

Further still, gyro-head 5 can have additional control units 13 installed, such as secondary inertial measurement unit (IMU) or control unit for stabilization of the gyro-head 5 itself.

Sensor device control system

Sensor device 4 is actively controlled by the operator 17 with use of a remote command unit 6 for control of the flying 1 or stationary 8 platform, and head-tracking system 7 for control of the sensor device 4.

Operator 17 controls the movement of the flying 1 or stationary 8 platform by using the remote command unit 6, whereat the unit comprises of one or more joysticks. One example of such implementation is the following: left-hand joystick 18 controls the vertical velocity - up/down (along z-axis) - and angular velocity around yaw axis (z-axis), while right-hand joystick 19 controls horizontal velocities: forward/backward (along x-axis) and left/right (along y-axis).

At the same time, the operator 17 controls a sensor device 4 attached to the gyro-head 5 by using a head-tracking system 7. Head-tracking system 7 consists of a video display 20 for image/signal display from the sensor device 4 and display of platform's operational data, an inertial measurement unit (IMU) 21 and a control computer 22 for processing and transmission of data regarding orientation and movement of operator's 17 head from the IMU 21 to the on-board computer 23 positioned on the flying 1 or stationary 8 platform.

Communication between sensor device, remote command unit and head-tracking system

Communication between the head-tracking system 7, which can be designed as goggles or a helmet visor, and a remote command unit 6 can be either via communication cable 24 or as a wireless transmission 25. Data transmission between a flying platform 1 or stationary platform 8 and a remote command unit 6 can be of at least two types:

• first as a data link 26 regarding desired velocity and/or orientation of the flying platform 1 or stationary platform 8, desired orientation of the gyro-head 5 and operation status information;

• and second as a video link 27 between sensor device 4 and operator's 17 head-tracking system 7 via remote command unit 6.

Sensor device 4 position and orientation measurement can be fed back via data transmission (feedback loop) between sensor device 4 and head-tracking system 7 of the operator 17. Information can be transmitted to the video display 20 of the head-tracking system 7 or any other data display system.

Orientation and movement of the platform and sensor device

Control of the flying platform 1 or stationary platform 8 and sensor device 4 follows the following process control logic:

1. Orientation and movement of the sensor device 4 and consequently control of it, is determined by continuous measurements by the on-board sensors (inertial measurement unit - IMU - 28, real time kinematic GPS 29, pressure sensor 30, ultrasound or laser distance- measuring unit 31, or any other direct or indirect position-/velocity-/orientation-measuring sensor 32), which are all installed in and measure orientation and movement of the flying platform 1 or stationary 8 platform, and optional secondary inertial measurement unit - IMU - 13, which is installed at the mounting point of the sensor device 4 and measures orientation and movement of the sensor device 4 itself).

2. On-board computer 23 collects all measured data, and calculates position, velocity and orientation of the flying 1 or stationary 8 platform.

3. On-board computer 23 then compares measured position and velocity of the flying platform 1 or stationary platform 8 with desired position/velocity set by the operator 17 via the remote command unit 6 and head-tracking system 7.

4. Based on the difference between measured and desired position and velocity of the flying platform 1 or stationary platform 8, the on-board computer 23 estimates the preferred control of the flying platform's 1 or stationary platform's 8 drive/movement systems (whether these are motors, rails, cranes, propellers, pulleys, etc).

5. Based on the difference between the measured and the desired orientation of the flying platform 1 or stationary platform 8, and on a desired orientation of the sensor device 4, the computer then determines the desired positions of actuators 15 of the gyro-head 5.

Improvement of the state estimation algorithm

Position, velocity and orientation of the flying platform 1 are determined primarily based on accelerometer and gyro sensor measurements. Velocity is calculated by integrating the accelerometer measurements, while position is determined by integrating the velocity calculations. This introduces the basic problem of integration error propagation. No measurement can be 100% accurate and every measurement has a certain error. These errors accumulate in the first integral of velocity calculations and in the second integral of position calculations. In order to reduce error propagation the position and velocity estimations are augmented with measurements from additional sensors. Integrating different sensor measurements into the position and velocity estimation is carried out in the Kalman filter. Position and velocity integration error propagation can be significantly limited with the use of additional measurements e.g. GPS. However, in certain situations GPS measurements can be intermittently or completely unavailable, for example indoors. For these situations optical flow measurements can be implemented.

Embodiments

The invention is further described using embodiments and figures: Fig. 1 is a schematic of a flying platform 1 with four rotors 33 o UAS arms 3 attached on a UAS fuselage 2 and a gyro-head 5 according to an embodiment of the invention;

Fig. 2 is a schematic of a stabilizing system, i.e. gyro-head 5, of a sensor device 4, in this case a camera for still photography or video services (view from all three sides and a 3D model) according to a further embodiment of the invention;

Fig. 3 is a schematic of all flying platform 1 and sensor device 4 command systems according to an even further embodiment of the invention;

Fig. 4 is a schematic of a flying platform 1 and sensor device 4 control system but from an operation point of view, i.e. link between operator's 17 head movements and movements of the flying platform 1 together with a gyro-head 5 according to a another further embodiment of the invention; Fig. 5 is a schematic of a stationary platform 8 with a sensor device 4, which in this case is camera for still photography or video services, according to another embodiment of the invention.

In Fig. 1, a flying platform 1 with four rotors 33 is depicted. Rotors 33 are disposed on four UAS arms 3 in an X-like form and are attached to a UAS fuselage 2. Arrangement of UAS arms 3 in this case can be of any shape or form, like Χ-,Υ-,Ζ-,Α-like. Rotors 33 are driven by motors 35.

A sensor device 4, in this case a camera, is mounted within the stabilizing system, i.e. gyro^head 5 that compensates for all unwanted orientation changes of the sensor device 4.

Flying platform 1 is equipped with an on-board computer 23 which controls the flying platform 1 and the gyro-head 5 of the sensor device 4. On-board computer 23 incorporates inertial measurement unit 28 for calculation of the position, velocity and orientation of the flying platform 1. Flying platform 1 further incorporates real time kinematic GPS 29, pressure sensor 30, ultrasound or laser distance-measuring sensor 31 and any additional position-/velocity-/orientation-measuring sensor 32 for additional calculations of the velocity, orientation and/or position of the flying platform 1. In Fig. 3, flying platform 1 also incorporates a data link 26 for data transmission between the flying platform 1 and a remote command unit 6 by a radio signal, and a video link 27 for transmission of video images to a video display 20 of a head-tracking system 7 via remote command unit 6.

Such a flying platform 1 can in addition to a brief description hereinabove be equipped with one or more systems mentioned in the description above, i.e. one or more sensor device 4 stabilising systems, i.e. gyro-heads 5, a sensor device 4 control system via a head-tracking 7 and a remote command unit 6, and communication system between the flying platform 1, head-tracking system 7 and remote command unit 6. ln Fig. 2, a further embodiment of the invention explained above, such a flying platform 1 can be further equipped with an additional control unit 13 (gyro sensor) installed on the sensor device 4. This measurement can be backed up and verified by an additional unit, such as accelerometer and/or sensor for measuring Earth magnetic field, which can provide time-stable calculation of orientation of the flying platform 1.

In an even further embodiment of the invention explained above orientation of the gyro head 5 can be determined in two ways: either directly with a help of an additional control unit 13 (IMU) installed on the gyro^head 5, or indirectly via IMU 28 installed on the flying platform 1 and with a feedback loop from the actuators 15 of the gyro-head, i.e. feedback info regarding their position. In a further embodiment of the invention, the gyro-head 5 is controlled only by the signal from the head-tracking system 7. In other words, even if the flying platform 1 is moving or rotating due to desired inputs from the operator 17 and/or due to external effects such as wind, the gyro-head 5 will keep the sensor device 4 in a desired direction. Gyro-head 5 cancels any rotational movement of the flying platform 1.

In an even further embodiment of the invention as described above, the gyro-head 5 can incorporate predictive control of the flying platform 1 movements. In other words, actuators 15 can be controlled based on current actual (measured) data of the flying platform's 1 movements, and also with an additional prediction of the future platform movements, whereat prediction is determined based on the previous and current control inputs by the operator 17. Such control can further improve quality of stabilization of the flying platform 1.

In a further embodiment of the invention, the velocity of the flying platform 1 can be proportional to the deflection angle of the joysticks 18 and/or 19. Such functionality can be achieved with the use of fly-by-wire system.

In Fig. 4, a further, particularly preferred embodiment of the invention, the gyro-head 5 of the sensor device 4 can be designed in one, two or three axes. At the same time the head-tracking system 7 can be implemented in one, two or three axes. Combination of these options allows for example, that control of the flying platform 1 around yaw-axis 41, that is rotation of the whole flying platform 1 around yaw-axis 41, also takes over the control of the gyro-head 5 around the yaw-axis 12. Such configuration omits the need for stabilization of the sensor device 4 in yaw-axis 12 in applications that do not require extra stabilization control.

Technical solution as described above in this embodiment, can involve also any other sensor device 4 mounted on a single or multi-rotor unmanned aircraft system. These can be laser pointers, thermal IR cameras, radar, scanners or similar. ln a further, particularly preferred embodiment, orientation of the operator's 17 head is measured by an IMU 21 installed in the head-tracking unit 7, and is functionally correlated in a linear (for example: 1° rotation of head equals 1° rotation of camera's gyro-head 5) or progressive, eg. exponential, way according to a prescribed function. Later allows operation of the camera beyond 55° angle upward in pitch-axis 37. 55° is a physical limit of the human head to turn upward. Such solution omits the need of any eye-tracking systems.

In a further preferred embodiment, measurement of the orientation of operator's 17 head is performed by the IMU 21 installed on the head-tracking system 7 and control computer 22 of the head-tracking system 7 for data processing. For more accurate and/or time-stable calculation of the operator's 17 head orientation, axial accelerometer or sensor for measuring Earth magnetic field can be used. Other sensors that can determine orientation of operator's 17 head, directly or indirectly, can also be used.

Head-tracking system 7 for estimating and calculating operator's 17 head orientation can be mounted on goggles, visor or any other system mounted on operator's head, or can be installed in the vicinity of the operator 17.

In one further embodiment, above-described platform-control system, including all subordinate embodiments, can in a similar way be implemented in a fixed-wing flying system, whereat platform's lift is provided by wings instead of rotbrs.

In Fig. 5,a stationary platform 8 with a gyro-head 5, and a sensor device 4 mounted on it, is illustrated. Same principle of operation as for a flying platform 1 described before, also applies here: control of the sensor device 4 is achieved by using a head-tracking system 7 and a gyro-head 5. All subordinate embodiments applied to a flying platform 1 can also be applied here where stationary platform 8 is described. Stationary platform 8 can be mounted on either fixed (tripod) or any movable (rail, crane, pulleys, etc.) system 9.

Other embodiments of the invention ill be evident to experts from the field of unmanned aircraft systems and aerial photography. Such as that UAS can be replaced with any ground, water or air vehicle/system and subsequently equipped with gyro-head stabilizing platform on which a sensor device is mounted.

Descriptions and the examples mentioned hereinbefore are to be considered merely as exemplary and not limiting in any way the scope of this invention.