The invention relates to progressive transmission of images, and in particular altnoug net exclusively to tr.e progressive transmission of medical images across a computer network from a server to a workstation.

There is often a need, within tne nospital environment, for pnysicians and surgeons to have rapid access to a variety of medical images sucn as, for example, x-ray images or CAT scan images. Typically, the images may be stored m digital form on a central server so that tney may be called up as required by tne aser on a local workstation or display client.

The problem with such an arrangement is that tne transmission of images, particularly detailed medical images, across a network puts a great strain on the available bandwidth. In a realistic nospital environment, the time taken for transmission of one detailed image is likely to b^ high because of the competition for bandwidth arising from otner users of tne system who may be attempting to view different images at the same time on other workstations. A furtner difficulty is that the user may wish to see only one of a series of perhaps rather similar images, and t s of course extremely inefficient to have to transmit each of those images, one by one, in full detail before the user is able to cnoose tne image ne or sne actually requires.

One method of dealing with tr.is problem s to present the user with a set of small "tnumnnail" images, eacn of which s a low-resolution version cf one of tne images cf tne set tne user s interested ir.. The user selects the required image using tne displayed tnuπona l images, the required image tnen oemg sent ιr_ igner-

BAD ORIGINAL

There are two ways m whicn the image information may be updated from a low-resolution thumbnail image to a hign-resolution display of the same image. The first way is to retransmit the full content of the high- resolution image. This represents a wastage of oandwidtr., since some of the image information is already present at tne viewing console. The second way m which the display may be updated is to transmit only the difference between the current image resolution and the desired nign-resolution image from the server to the workstation. In the case of a wide area distributed viewing system, the retransmission of full resolution images represents a potential problem: for an individual image the retransmission of the full resolution image may not be significant; for the case in which there are many active viewing consoles, and a steady traffic of viewing and selection operations, the retransmission of the entire image may represent a significant bandwidth requirement over the transmission of the detail only. It is an object of the present invention at least to alleviate the problems of the prior art.

It is a further object to provide an efficient progressive transmission system having an improved (smaller) bandwidth requirement. It is a further object to provide a system in which progressive transmission may be applied to any selected portion of an image that tne user wishes to see at higher-resolution.

According to a first aspect of the present invention there is provided a metnod cf progressively transmitting images across a computer networ from a server to a worKStaticn, tne metnod comprising. ιa; storing an image on the server as a digital file, the file comprising a low-rescluticn portion

BAD ORIGINAL

representative of a low-resolution version of the image, and a detail portion representative cf detail in the image not contained within the low-resolution version; ⁽ b) transmitting first information from the low- resciuticn portion to the workstation, and displaying the low-resolution version of the image at the workstation;

,c) selecting from the displayed image a user- defined area to be viewed at higher resolution;

(d) transmitting from the server second information from that part of the detail portion of the file which relates to the user-defined area;

(e) combining at the workstation the second information and that part of the first information which relates to the user-defined area to create a higher-resolution version cf the user-defined area; and

(f) displaying at the workstation the higher- resolution version of the user-defined area.

The method of the present invention is particularly although not exclusively of use in the hospital environment, for the transmission of medical images such as echocardiographic images . There is often a requirements in a hospital environment to display a selection cf mosaic of images, which may typically be image slices of a 3D data set, or frames of an image sequence. The present invention allows the individual images making up the mosaic to comprise small iow- resolution "thumbnail" images. A user who wishes to see more detail for one particular image, simply selects the image (for example by clicking on it using the mouse) . Further detail is then sent from the server, this further detail being combined with the information already held

BAD ORIGINAL

at the workstation to produce a slightly higher resolution image. Yet further information may be requested from the server if the user needs to see an even higher resolution version of the image. In each case, the only information that needs to be transmitted across the network is the detail information, that is the difference between the current low-resolution version of the image already held at the workstation, and the required higher-resolution version. In some embodiments, the information may be sent automatically from the server. In other embodiments, a specific request may be sent by the workstation to the server each time further information is required. This request may conveniently be user-generated. The preferred encoding method is to use the orthogonal wavelet transform. This is a transform which is discrete in both the time domain and in the scale- space (wavelet) domain. The small extent of the functions in the spatial domain allows for rapid convolution. The orthogonal nature of the functions used in the preferred embodiment guarantees that the minimum amount of information needs to be sent across the network.

The user may select a desired area of the image which he or she wishes to see in more detail. A desired region could for example be picked out on the screen with a mouse, preferably by dragging a box over the area of interest. If only an area of the image is selected by the user, the server sends information only on that selected area across the network.

The invention extends to a computer network and/or system for operating the method of the present invention.

The computer network may comprise a local area network or a wide area network, connected by wires or by

j ffiffi -NAL-

a wire-less link. The term "network" includes any means of remote communication between computers, and accordingly encompasses (without limitation) communication via the telephone system and/or satellites. Such means cf remote connection may be useful where for example the server is in a different hospital, and perhaps even in a different country, from the workstation.

The invention may be carried into practice in a number cf ways and one specific embodiment will now be described, by way of example, with reference to the accompanying drawings, in which:

Figure 1 is a block diagram showing, schematically, the operation of the discrete wavelet transform; Figure 2 (a) shows the discrete wavelet transform decomposition process;

Figure 2 (b) shows the decimation process at successive scales;

Figure 3 shows the Daubechies 4, 12 and 20 scaling functions, evaluated at different respective scales;

Figure 4 illustrates the decimation process in graphical form; and

Figure 5 shows how the multi-resolution image may be stored. The preferred embodiment of the present invention is a method and/or system for the progressive transmission of medical images from a remote file server to a local workstation. Each image is stored on the server in a multi-resolution format comprising a low-resolution representation which may be called up as a small "thumbnail" image en the workstation, and a plurality cf "detail" representations. If the user wishes tc see the image at lowest resolution, the thumbnail image alone is sent. If, having viewed the thumbnail image, the user

BAD ORIGINAL

wishes to receive a higher-resolution representation, the first "detail" representation is passed along the network to tne workstation. At the workstation, the "detail" representation is added to tne low-resolution image to provide tne required higner-resoluticn image If yet furtner resolution is required, tne user can call upon tne next "detail" representation, stored on the server, which m a like manner is added to tne information which has previously oeen passed to the workstation to provide an image at the next highest resolution. The process may oe continued until the user has the image resolution he or she requires, or has an image of the maximum resolution wnich is available from tne server.

The first stage in the process is, of course, to take an original image and to encode it in digital form in a digital data file which forms the multi-resolution representation that is actually stored on tne server. In order to split the original image up into a low- resolution image plus a plurality of "detail" representations or images, the preferred embodiment uses the orthogonal wavelet transform.

The orthogonal wavelet transform is a discrete transform, similar to the Foulier transform but instead of using sin functions wavelet mother functions are used instead. Each wavelet mother function has a limited extent m wavelet space, wnicn contrasts witn the sm functions used for Foulier transforms wnicn are of course infinite m extent.

The wavelet transform has the following general form:

BAD ORIGINAL

The discrete wavelet transform, which s used here, is the same, except that the values of a and b are restricted to discrete values only. In the preferred embodiment, orthogonal wavelets are cnosen, for example the Daubecnies wavelets shown in Figure 3.

For any given signal, one can compute a series of weights for discrete values of a and b whicn, when multiplied by the basis function and scaled leads back to the representation of f . This can be done, in principle, without losses (other than those arising from rounding errors) . The fact that the wavelets are ortnogonal guarantees the lowest possible error f certain terms are thrown away, for example if the user wishes to see an image of only intermediate resolution.

The advantage of the present scheme is that the full set of coefficients or weights W(a,b) can be arranged into a multi-resolution representation of the original signal . As an illustration, consider the one-dimensional example shown in Figure 4. If one takes f (n) to be a discrete signal to be approximated, one can obtain a first approximation by calculating f ' where f (n) is calculated by determining the mean cf adjacent samples within f (n) . For example, f (0) is half the sum of f (Oj and f (1) . Likewise, f (1) is half the sum of (2) and f .3) . F' (n) may therefore be considered as an approximation g(n) to f (n) . One may also define a "detail" representation h(n) as the difference oetween f (n) and g(n) .

The process may be repeated, decimating tne sample each time, thereby creating a plurality of individual "detail" representations, each indicative of the difference between the current approximation and the last

BAD ORIGINAL

previous approximation.

To store the image m multi-resolution format, the final approximated signal is kept, along with all of the detail signals. The process may be represented schematically at eacn stage πy tne block diagram shown in Figure 1. At eacn stage, an approximation signal g(nj and a detail signal h(n) is determined from the input signal. The approximation signal is then fed back nto the output and a further approximation and a further detail signal oDtamed. At an appropriate stopping point, the final very low-resolution g(n) output is retained, along with ail of the detail outputs h(n) .

The structure shown in Figure 1 can be used for multi-resolution orthogonal biorthogonal, or quadrature mirror filters (QMF) where eacn of the above cases s specified by the chosen filter coef icients.

The filters are simply denoted as low pass, g(n) and high pass, h(n) , to present a general form, but m fact these filters are related to scaling function, φ(x) , and wavelet function ψ (x) [2] . The decomposition process is illustrated in Figure 2.

Each block has both a low pass filter and a nigh pass filter. The output of each filter is decimated by a factor of two. The resolution of the output cf the low pass filter is also changed cecause cf losing nigh frequency detail.

The output of the filter chain, A- ^' ^S, signal represents a discrete approximation of signal at the resolution 2 ^r . The other output is called tne detail signal, D _^ ^m S, at the resolution 2 ~ If tne original signal r.as N samples, then the total numoer cf samples in tne detail and approximation signals is also N. Thus tne discrete wavelet representation has tne same total numoer

BAD ORIGINAL

of samples as the original signal. The smoothness of A2 ^: S depends on the shape of the scaling function. In general we would wish to have a smooth scaling function, but this function should satisfy some conditions such as orthogonality, support width etc. Dne cf the important things is the spatial support .width) of the scaling function which divides the wavelet transform into compact into compact and non-compact wavelets.

In compactly supported wavelets the scaling function and the wavelet function have mere regularity at the expense cf a large number of non-zero coefficients of h(n, , which results in a large support for φ ( x ) . This support width increases linearly with their regularity in some wavelets such as the so-called Daubechies wavelets (Daubechies, 1988) . As shown in Figure 3, increasing the number of coefficients of h(n) , leads to an increase in the smoothness of the scaling function. Note that the three traces of Figure 3 have been evaluated at resolution in inverse proportion to their support, so

support. Because of the compact support and orthogonality the Daubechies wavelets provide a suitable basis set. The allat basis set is constructed from splines in continuous space, then discretized, and hence is not compact .

A multi-resolution decomposition is a method of representing signals at different scales of magnification. The concept cf multi-resolution decomposition was initially developed by Mallat [2] to address the problem of characterising image scenes in a scale independent manner. The approach cf Mallat is essentially a pyramidal decomposition, along the lines cf Burt and Adelson, but uses values of h(n; and g n) such that the decomposition is onto an orthogonal basis. The

BAD ORIGINAL

basic idea is that of successive approximation, together with adding detail signal from one approximation to the next . Assume that we have a ladder of spaces such that

- .'. - V. ~ V . V .

With the property that if f i x) v.

:hen

f , x - 2 - ^' k) EV. ; k E Z ; and f { 2 x) ' V...__

If W. is the orthogonal complement of V. in V . _: so we have

V, V.0 w.

Thus, W. contains the detail signal necessary to go from V. to V.. _: . If we have an approximation of a signal at a resolution corresponding to V _: , then a better approximation to the signal is obtained by adding the detail signal corresponding W.. This detail signal is the projection of the signal in W _: . By interpolating the output

A- -. Ξ

of the decomposition depleted in Figure (2) , one obtains an approximation

BAD ORIGINAL

\ n)

cf the signal S'n) at a resolution 2 ^": . The interpolation is defined by the following equation:

wnere

Λ.-

denotes insertion of a zero between each sample of the discrete signal

A _. -.S

This interpolation is a convolution between the scaling function φ and

This particular interpolation represents an example m going from a representation cf the signal at one scale to an approximation of the signal at the scale

BAD ORIGINAL

immediately above it. If we wish to go from a representation of the signal at, say, a scale of 2 ^" to an approximation of the signal at some scale 2 ^* - , wnere 1>1, it is necessary to dilate tne scaling func ion appropriately.

The appropriate dilation may be performed as follows: if the scaling function of Equation \ l ι is at tne resolution ld.e. 2 -) , then it belongs to V .

Since

c i\

one may write

φ (2 - ^" -__) = ∑ Λ(JO φ (..-ic)

(2) This equation may be generalised to relate the scaling function at a scaie 2 to the scaling function at a resolution 2 " ' ^' - for any j . Therefore,

S i n) - 53 Φ (2 ^,*1 n-_c) A. ^' S ( k)

2 ' K

' 2 )

For the sake of simplicity, the above examples have assumed a one-dimensional signal ratner than a two- dimensional image, which of course is what we are really interested in. In dealing with images ratner than signals, it is necessary to use the corresponding two-

BAD ORIGINAL

dimensional discrete wavelet transform (DWT ^N . The two- dimensional transform may again be represented by a repeated function block where, instead cf a pair of outputs, we have four outputs corresponding to the low- frequency image component and the high-frequency vertical, horizontal and diagonal components. The latter three image components correspond to tne detail signals. For details of the two-dimensional decomposi ion, refer to Mallat [1] . It is possible to generalise Ξσuaticn (3) to the two-dimensional case:

r_) 2 - ^'*1 n - k) φ (2- ../ Λ-. _. ^< X,

(4) due to the separability of the two-dimensional decomposition. Figure 5 shows, in schematic form, how the information may be stored in a data file on the server. The lowest resolution image that is expected to be required ⁽ A) is stored, along with the sequences of detail information D- , D _: and D, . If, for example, the lowest resolution image A is an 8 x 8 image, one will automatically obtain from the algorithm three sets c •P detail signals each cf which also has size 3 x 3. The next level up is 16 x 16, and so on. To produce an image of 16 x 16 resolution, the three detail signals D_ are added into the low-resolution image A. Similarly, to create an image of 32 x 32 resolution, the detail images D- are added onto the previously-created 16 x 16 image.

It will cf course be understood that it is not necessary for the data file to be stored exactly as it is shown m Figure 5. Ail that is required is that the low- resolution image can be selected from the file and sent

BAD ORIGINAL

across the network to the workstation followed, if requested by the user, by each of the detail images in order .

In looking at medical images, it is often useful to be able to view a particular region of interest in greater detail. The infrastructure of orthogonal, multiresoiution image decomposition provides an efficient way cf doing this, in terms of bandwidth requirements. Consider a region of interest, such as a heart valve on a cardiac image. By selecting an approximate region of interest on a low resolution, interpolated image, the present infrastructure allows one to view that region at full resolution without needing to increase the resolution outside of the specified region. The user first selects an area of interest (for example using a mouse) , and that area is then redrawn on the screen in greater detail. For example, if the user wishes to view in more detail only the top left-hand corner of the thumbnail image A, he or she simply selects that area and a signal is sent to the server instructing it to send further detail appropriate to that area only. With the file structure shown in Figure 5, the server would send only the detail information contained within the top left-hand corners of each of the three D. images. If the user then wishes to view that enlarged image in yet more detail, the server would send only the top left- hand corners of the D _: images .

In order to refine the detail over a particular user-selected area of the image, we need to determine the range of influence of all samples 'pixels ¹ within the area we are interested m, over all scales of the decomposition process. This range cf influence is clearly a function of the width cf the filters used, that is the impulse response cf the filters used in the

BAD ORIGINAL

decomposition. Since we are typically using filters of very small length, this range of influence is fairly localised throughout all space. However, proper reconstruction can be carried out only if we can correctly identify all of the samples, at each scale, which influence the desired image within the selected area .

A general implementation involves tracing all of the influenced children of a pixel throughout the various components of the image in the transform space.

Although tedious, this is quite within the capabilities of the skilled man. If orthogonal filters are used, as they are in the preferred embodiment, then the range of influence simply becomes the direct children (4) of each pixel.

This allows great savings in bandwidth. Consider the requirement to view a region of microcalcification in a digital x-ray mammogram at a resolution of 256x256 pixels. Mammograms typically have full pixel dimensions of 4096x4096 pixels. In such a situation, immediate enhancement of the region requires ail the coefficients to go in resolution from, say a 512x512 icon to a full resolution region is less than 1% of the total coefficients necessary to rebuild the entire image at 4096x4096 resolution.

The results of our study show that the reconstruction of an image from its full wavelet decomposition represents by selecting an appropriate set of scales represents a valid method of building a multi- resolution framework for progressive transmission of echocardiographic images. We have shown that not only does this framework support the progressive updating cf "thumbnail" images, but also that smooth interpolation cf these small images by an appropriately dilated scaling

BAD ORIGINAL

function lead to low resolution images at magnifications (scales) comparable to the original image. Such blurred images may be used, for example, to select with precision a particular region of interest, which cne wishes to view m greater detail 'i.e. at full resolution' .

The clear advantage of adopting this approach to the clinical viewing cf images m a networked client/server environment, such as a teleconferencing or remote diagnosis application, is that communications bandwidth is used optimally. Where a large number cf clients are involved, and commercial networks are employed, the savings can be significant. Finally, even on local area networks using ethernet technology, the cut m overheads can represent a significant saving in bandwidth.

References

[1] Daubechies, I., "Orthonormal basis of compactly supported wavelets", Commun cations on Pure and Appl ied Ma thematics , vol. 41, pp 909-996, 1988.

[2] Mallat, S., "A Theory for multiresoiution signal decomposition: The wavelet representation", IEEE Transactions on Pattern Analysis and Machine In tell igence, vol. 11, no. 7, pp 674-692, 1988. [3] Burt , P. and Adelson, E., "The Laplacian Pyramid as a compact image code", IEEE Transactions on Communi ca tions , vol. 31, no. 4, April 1983, pp 532-540.

BAD ORIGINAL