BACKGROUND ART Navy and other military targeting applications depend upon sensor systems and realtime communications networks to direct weapons systems capable of attacking targets as soon as adequate targeting information is available. The quicker that the commander receives intelligence and other relevant information, and the quicker that the information is updated, the greater his ability to efficiently manage his battle space, and better use his limited resources. Central to this is the management of surveillance and reconnaissance assets, and the fusion of multi-sensor data. The timely acquisition of this data requires the use of multiple sensors with variable formatted data, and the use of image formation algorithms (including realtime image decompression) which provide useful realtime display and dissemination, also referred to as"image processing".

Navy and other military air reconnaissance is typically performed by carrying a camera aboard an aircraft, shooting a sequence of step-stare image frames as the aircraft progresses across the ground (or body of water), and, in this manner, recording on film, tape, or like medium a series of images. After completing its mission, the aircraft flies to a base where the storage medium with the data is delivered and processed through an image processor. Such image processors have the ability to zoom, pan, or otherwise vary image resolution in response to an operator's commands. The processor does this by reading an image from memory, and sampling the image according to the desired resolution. Typically, when an operator wishes to change the zoom, e. g., to look closely at an intriguing part of the image, the processor must re-read the data from memory, and re-sample it.

Because the data can be processed only after physical delivery to a distant station, and because repeated changes of the resolution at which the data displays requires repeatedly uploading image data prior to re-sampling, this whole approach is inherently slow, and ill-adapted to realtime image processing. Furthermore, cameras presently available have high data rates, on the order of one gigabyte per second. The existing slow, non-real time, approach to processing camera data fails to exploit the full potential of these cameras.

Thus, there is a need in the art for a system, apparatus and method for image processing to permit image receipt, and realtime image display, commensurate with the data rate of the above described imaging cameras. Another desirably feature is to permit realtime variable resolution of such images, e. g., permitting an operator to selectably choose the degree of zoom or pan with which such images are displayed.

DISCLOSURE OF INVENTION The present invention is a system, apparatus and method for image processing, in which a digital image is received into memory, the memory is uploaded, a decision made on the resolution with which to display the image, and the uploaded image sampled to produce a display having the desired resolution. Because the data stays uploaded after sampling, it is available for further sampling should a change of resolution be desired, e. g., by an operator, and eliminates the slow conventional process of repeatedly uploading the image each time one wishes to change the resolution. The system, apparatus and method includes direct system connection between data reception, and data processing and display, which eliminates the need to transport the data to a distant processing station. This feature makes the image processor of the present invention sufficiently fast compared to existing systems to permit realtime operation, and in turn, permitting realtime decision making predicated on the data which the system presents.

The system, apparatus and method of the present invention are further understood from the following detailed description of particular embodiments of the present invention. It is understood, however, that the invention is capable of extended application beyond the precise details of the embodiments. Changes and modifications can be made to the embodiments that do not affect the spirit of the invention, nor

exceed its scope, as expressed in the appended claims. These embodiments and methods of the present invention will be readily understood by reading the following detailed description in conjunction with the accompanying figures of the drawings.

BRIEF DESCRIPTION OF DRAWINGS In the drawings, which illustrate what is currently regarded as the best mode for carrying out the invention and in which like reference numerals refer to like parts in different views or embodiments: FIG. 1 is a schematic of a system according to the present invention.

FIG. 2 is a flow chart of a method for image processing raw image data in accordance with the present invention.

BEST MODES FOR CARRYING OUT THE INVENTION Broadly speaking the invention is a system, apparatus and method for realtime image display and manipulation.

Referring to FIG. 1, an image processing system 100 which receives data from aircraft 10, which itself preferably has one or more digital cameras taking step-stare images of the terrain below. Airplane 10 transmits the image data to a telemetry link, which may be, for example and not by way of limitation, a data link 12 fed by antenna 12', as illustrated. In practice, data link 12 is preferably a common data link (CDL) disposed to receive inputs from a plurality of sources, e. multi-spectral sources, infrared sources, electro-optic sources, microwave sources, etc. Initial memory storage 14 takes raw data from data link 12 and stores it. Initial memory storage 14 preferably re-spaces the data in time in a manner know to those skilled in this art to compensate for variances in transmission, e. g., variable atmospheric conditions.

Data processor 16 reads one image frame of data from initial memory storage 14 and first strips the raw data of formatting, and then temporarily dumps it via data bus 24 into compressed memory 20. Data processor 16 may also direct the raw data in parallel to a recorder 18 for archiving, and optional playback at a later point in time.

Thereafter, data processor 16 sends the image in compressed memory 20 via data lines 24 and 25, to decompressor 19, which expands the image data, and returns it to

processor 16 for storage in decompressed memory 22. Compresed memory 20 and decompressed memory 22 may, for example and not by way of limitation, each be 256 Mbyte buffers or 8 Gbyte buffers, which typically would permit storage of thousands of compressed frames in compressed memory 20 and hundreds of decompressed images in decompressed memory 22. One of skill in the art will appreciate that the memory buffer size selected may be chosen to satisfy the particular application and that the particular sizes stated above are merely exemplary.

Display processor 28 links to decompressed memory 22 via data link 26, and sequentially reads frames of image data for viewing in waterfall display 30 and variable display 32. Variable display 32 is preferably used for viewing expanded details of portions of the data in waterfall display 30. Data link 26 is configured to transmit an entire image to data processor 28 at one time, rather than only sampled portions of an image as is the case with conventional data links. Display processor 28 has parallel data tap 34 for transmitting data to and from a remote source (not shown), for example and not by way of limitation, a ground station via an Ethernet link. Additionally, data in display processor may be recorded or archived in recorder 36.

Waterfall display 30 and variable display 32 sample image data in display processor 28 in accordance with an amount of detail desired, preferably responsive to an operator's command. Preferably, waterfall display 30 shows image data at a wide angle in waterfall, or scrolling, format. Preferably, variable display 32 is configured to show zoomed-in details of selected portions of the data on shown on waterfall display 30.

To generate the images presented on waterfall display 30 and variable display 32, respectively, the data in display processor 28 must be sampled to different degrees of fineness or resolution. Because the decompressed image data is stored in display processor 28 for manipulation, it is unnecessary to repeatedly upload image data from decompressed memory 22, as does the prior art. For this reason, and because image processing system 100 takes input data via data link 12, as well as having recorded data delivered to image processing system 100, the image processing system 100 of the present invention permits realtime image acquisition and analysis heretofore missing from the battlefield. In the same fashion, image processing system 100 permits higher processor data rates that are better matched to newer high speed digital cameras.

Furthermore, the ability to zoom in realtime, and link display information to a distant location instantly via data tap 34, means that not only does the system permit an operator to view processed image data in realtime, but also that the operator can inspect details of the data in realtime via variable display 32, and decide, in realtime, to forward important data to a distant location, e. g., a battle space commander, also in realtime.

In practice, data processor 16 may apply any of a large number of conventional data analysis techniques, many of which require operation on plural images at the same time, for example images taken of the same terrain at the same time, but imaged in different frequency of light. This might typically be done by aircraft 10 having plural cameras operating and transmitting to data link 12 simultaneously, each camera imaging in different frequency bands. These images are sent to data link 12 preferably spaced in time, with images representing the same time preferably interleaved to permit separation. Of course, one of skill in the art will recognize that any other method of transmitting simultaneous data will also suffice, for example and not by way of limitation, modulating image data onto a carrier, and providing data link 12 separate channels for separately detecting and demodulating the carrier.

Data processor 16 handles the data as above described, but may leave distance data tags on the data to indicate the nature of the image, e. g., frequency band, in addition to time and/or place the image was generated. Display processor 28 may upload the plurality of images necessary for display which is desired. For example, and not by way of limitation, the desired display may include images of the same terrain taken at the same time but in different frequency of light, for operator directed overlay, side by side contrast, etc.

Processors 16 and 28 may be general purpose processors programmed to operate according to software instructions. Alternatively, processors may be special purpose processors, for example and not by way of limitation, digital signal processors.

Furthermore, processors, 16 and 28 may be combined with memory 20 and/or 22 and/or decompressor 19 in the form of an application specific integrated circuit (ASIC).

The various processors and memories described herein may also be implemented in the form of circuit boards subsystems or other higher order systems. In practice, any embodiment of the invention will involve tradeoffs between hardware implementation

and software implementation, the details for which will depend on the needs of any given system. Determining such tradeoffs is within the competence of one of ordinary skill in the art.

Referring to FIG. 2, a flow chart illustrating a method 200 of image processing raw step-stare image data from a data link, is shown. Method 200 includes storing 202 the raw step-stare image data, generating 204 compressed image data by stripping formatting information from the stored raw step-stare image data and storing 206 the compressed image data. The raw step-stare image data may or may not be compressed.

Additionally, the act of storing 202 the raw step-stare image data may include some processing, for example to demultiplex multiple channels of raw step-stare image data in the time domain. Additionally, compressed image data may be displayed concurrently with storing 206 the compressed image data. Method 200 further includes generating 208 decompressed image data from the stored compressed image data and storing 210 the decompressed image data. Method 200 may further include displaying 212 sequential frames of the decompressed image data, sampling 214 the decompressed image data in accordance with a selected resolution and displaying the sampled decompressed image data in accordance with the selected resolution. One of skill in the art will recognize that the order of processing steps presented above may be changed in certain implementations without departing from the scope of the invention.

Although this invention has been described with reference to particular embodiments, the invention is not limited to these described embodiments. Rather, it should be understood that the embodiments described herein are merely exemplary and that a person skilled in the art may make many variations and modifications without departing from the spirit and scope of the invention. All such variations and modifications are intended to be included within the scope of the invention as defined in the appended claims.