Cross-Reference to Related Applications

[0001] The present application claims priority from Australian Provisional Patent

Application No 2019901610 filed on 10 May 2019, the contents of which are incorporated herein by reference in their entirety.

Technical Field

[0002] This disclosure relates to systems and methods for rendering an enhanced contrast greyscale image on a display device and in particular for rendering the greyscale image on a prosthetic vision device.

Background

[0003] Visual prostheses are devices for conveying visual information to visually impaired people. They typically operate by capturing scene data using one or more sensors worn on the body. A display image of the scene is computed from the scene data and communicated to the user via a display device. Display devices include suprachoroidal, subretinal and epiretinal retinal prostheses, cortical prostheses, head-mounted displays or sensory substitution devices (e.g., auditory, tactile, etc).

[0004] For example, consider a suprachoroidal retinal prosthesis device. An image capture device, such as a camera, captures scene data of a user's surroundings. The scene data is processed to generate a stimulation signal which is an electrical representation of the scene data. This stimulation signal is then used to elicit phosphenes through electrical stimulation of various stages of the visual pathway. Phosphenes are perceived bright dots of light which occur in predictable locations within the visual field and when stimulated by the stimulation signal convey information about the scene data to the user.

[0005] Similar schemes are employed for other display devices. Summary

[0006] According to a first aspect, there is provided a method for providing an enhanced contrast greyscale image to a display device, the method comprising: converting an input colour image to a reduced colour image by mapping colour values of the input colour image to palette colour values, wherein the palette colour values represent clusters in a colour space of the input colour image; generating a conversion map by mapping the reduced colour image to a target greyscale image, wherein the conversion map assigns a contrast between greyscale values based on a colour contrast between palette colour values; applying the conversion map to the input colour image to generate an output greyscale image; and providing the output greyscale image to the display device.

[0007] It is an advantage of this aspect that different palette colours in an input colour image are rendered as greyscale contrast on a display device. It is a further advantage that the reduced colour image has a reduced colour range compared to the input data. This reduces the computational requirements for generating the conversion map thereby reducing energy requirements and latency between receiving the input image and rendering the output image.

[0008] The method may further comprise downsampling the input image to a pixel configuration of the display device before applying the conversion map.

[0009] It is a further advantage of this method that the greyscale image data is generated using the downsampled image. This reduces the computational requirements for converting a colour image to a greyscale image, thereby reducing energy requirements and latency between receiving the input image and rendering the output image.

[0010] The downsampling may comprise using Lanczos sampling.

[0011] The pixel configuration may be irregular. [0012] The method may comprise identifying the clusters in the colour space using K-means clustering.

[0013] The method may further comprise a step of enhancing contrast in the greyscale image before the providing step, the step comprising, for each pixel of the greyscale image: determining a neighbourhood intensity mean based on an intensity of pixels having a predetermined spatial relationship to that pixel;

comparing that pixel intensity to the neighbourhood mean;

increasing that pixel intensity if the pixel intensity is greater than the neighbourhood mean; and

decreasing that pixel intensity if the pixel intensity is less than the neighbourhood mean.

[0014] It is an advantage of this method that contrast in the greyscale image is enhanced thereby improving the clarity of the rendered output image data.

[0015] The predetermined spatial relationship may be pixels that are next to that pixel.

[0016] The predetermined spatial relationship may be pixels that are within a predetermined distance of that pixel.

[0017] The method may further comprise the step of suppressing localised noise in the output greyscale image data before the providing step, the suppressing step comprising, for each pixel of the output greyscale image data: determining a regional mean based on an intensity of pixels in a region of that pixel; and adjusting the pixel intensity according to the expression:

where y' is the adjusted pixel intensity, y is the pixel intensity, T _min is a minimum threshold value, T _max is a maximum threshold value and m _y is the regional mean.

[0018] It is an advantage of this method that pixels which only vary by less than a threshold amount from the regional pixels are not enhanced and that extreme individual pixel intensities, being pixel intensities close to either zero or saturation (1) are set to that extreme when the regional pixels are also close to that extreme. This avoids subthreshold pixel intensities which can be perceived as‘flashes’, or noise, in the rendered image. This step therefore can be considered a filter, reducing noise and therefore distractions in the rendered image.

[0019] The pixels in the region of that pixel may be pixels that are adjacent that pixel.

[0020] The pixels in the region of that pixel may be pixels that are within a predetermined distance of that pixel.

[0021] The minimum and maximum thresholds may be set according local or global properties of the received input colour image.

[0022] The conversion map may be generated for an initial input colour image and retained for one or more subsequent input colour image.

[0023] The method repeated for one or more subsequent input images and wherein the step of generating the conversion map may comprise adding hysteresis to the conversion map such that subsequent conversion maps adapt slowly to the subsequent input colour image data.

[0024] The display device may be a prosthetic vision device.

[0025] The prosthetic vision device may be selected from a suprachoroidal retinal prostheses, a head-mounted display or a sensory substitution device.

[0026] According to a second aspect, there is provided a system for rendering a greyscale image, the system comprising: an image capture device for generating an input colour image representative of a scene; a processing module configured to: convert the input colour image to a reduced colour image by mapping colour values of the input colour image to palette colour values, wherein the palette colour values represent clusters in a colour space of the input colour image; generating a conversion map by mapping the reduced colour image to a target greyscale image , wherein the mapping assigns a contrast between greyscale values based on a colour contrast between palette colour values; apply the conversion map to the input colour image to generate an output greyscale image; and

provide the output greyscale image to the display device; and a display device configured to render the output greyscale image data.

[0027] According to a third aspect, there is provided a device for generating output greyscale image data, the device comprising: a processor configured to:

convert the input colour image to a reduced colour image by mapping colour values of the input colour image to palette colour values, wherein the palette colour values represent clusters in a colour space of the input colour image;

generating a conversion map by mapping the reduced colour image to a target greyscale image , wherein the mapping assigns a contrast between greyscale values based on a colour contrast between palette colour values;

apply the conversion map to the input colour image to generate an output greyscale image; and

provide the output greyscale image to the display device. [0028] According to a fourth aspect, there is provided a non-transitory computer readable medium configured to store software instructions that when executed cause a processor to perform the method described above.

Brief Description of Drawings

[0029] Fig. 1 is a schematic illustration of a system for rendering a greyscale image of a scene;

[0030] Fig. 2 is a schematic illustration of an exemplary processing module;

[0031] Fig. 3a illustrates the effect of a prior art conversion map.

[0032] Fig. 3b illustrates the effect of a conversion map as performed by the processing module of Fig. 2.

[0033] Fig. 4 is a flow diagram representing a method performed by the processing module of Fig. 2;

[0034] Fig. 5 is a schematic illustration of an exemplary processing module;

[0035] Fig. 6 is a flow diagram representing a method performed by the processing module of Fig. 5;

[0036] Fig. 7 illustrates the effect of downsampling;

[0037] Fig. 8 is a flow diagram of a method for enhancing contrast;

[0038] Fig. 9 is a flow diagram for suppressing noise;

[0039] Fig. 10 is a flow diagram for providing a subsequent greyscale image;

[0040] Fig. 11 is an exemplary conversion map; and [0041] Fig. 12 illustrates a reduction in dynamic range. Description of Embodiments

[0042] This disclosure relates to methods and systems for displaying image data, representative of a scene, on a greyscale display device having limited resolution such as vision prosthetic devices. Embodiments will be described with reference to wearable devices that aim to enhance or restore a human wearers perception of their environment.

[0043] Typically, a body worn sensor captures scene data and a display device processes and provides the scene data to a user. For the purposes of this invention, a display device could be a suprachoroidal retinal prosthesis, a head-mounted display or a sensory substitution device (e.g., auditory, tactile, etc).

[0044] In real-world scenes, there are typically more visual details than can be

communicated by a restricted display device. For example, a display device may have low resolution which limits how much detail can be conveyed to a user. In the case of a suprachoroidal retinal prosthesis, a user may only be able to see 20 'pixels' or phosphenes.

[0045] Display devices also tend to have a reduced dynamic range, limiting usage to high contrast environments. While many different resolutions and dynamic ranges may be used, some example retinal prostheses provide between 20-1500 electrodes, with between approximately 2-8 levels of brightness. Such limited dynamic range has little adverse effects if the user views a high contrast environment, e.g., black and white only. The display maps the input camera intensity, from black and white, to a black and white output and therefore, the contrast will be preserved. However, if the environment consists of shades of grey, then regions that are different, but have low contrast may be mapped to the same output value. Further, with a colour image, if the colours have the same intensity, but different

chromaticity, then chromatically contrasting regions are likely to be assigned the same level in the output display. Hence the user will not be able to perceive the difference. As a result the user is unable to distinguish regions that perhaps correspond to real-world objects such as a door, or obstacles. OVERVIEW

[0046] Fig. 1 illustrates a system 100 for rendering a greyscale image 101 on a display device 102. Display device 102 is a visual prosthetic for a user 104 having at least a degree of visual impairment. Greyscale image 101 conveys key information about a scene 106, in this case a pair of scissors on a table, to user 104. Scene data is captured by an image capture device 108 captures an input colour image 109 of scene 106 and provides it to processing module 110. Image capture device 108 is typically a sensor, such as a colour camera, worn on the body of user 104. Input colour image 109 is processed by processing module 110 to provide suitable data for display on display device 102 which is greyscale and typically of lower resolution and dynamic range than image capture device 108.

[0047] Processing module 110 is shown in greater detail in Fig. 2. As mentioned, processing module 110 processes an input colour (Red-Green-Blue) camera image into a greyscale image for display on a display device such as a retinal prosthesis, a haptic display or enhancing head-mounted display such as Smart Glasses. Typically, display device 102 is of a lower resolution and dynamic range than image capture device 108. In this case, processing module 110 converts, or maps, input colour image data at a first resolution and dynamic range to output greyscale image data at a second resolution and dynamic range. The output greyscale image has greyscale contrast representative of chromatic contrast in the input colour image. The effect of this mapping is illustrated in Fig. 3b where it is contrasted with a standard mapping technique shown in Fig. 3a. In Fig. 3a, input image 302 is mapped to greyscale image 304. Input image 302 comprises chromatically contrasting stripes 306 and 308, which are of equal intensity but distinguishable based on colour. For example, stipes 306 may be red and stripes 308 may be green. The standard mapping technique 309 maps input image 302 to greyscale image 304 based on intensity. As the intensity of red stripes 306 are equal to the intensity of green stripes 308, both are mapped to the same shade of grey 310.

[0048] In contrast, processing module 110 maps contrasting colours 306 and 308 to contrasting greyscale intensities 312 and 314, as shown in Fig. 3b.

[0049] The operation of processing module 110 will be described with reference to Figs. 2 and 4. Fig. 2 is a schematic illustration of processing module 110 which comprises a transceiver module 202, a colour reducing module 204, a map generation module 206, an image conversion module 208, a memory module 210 and a processing unit 212.

[0050] Modules 202 to 208 may be software applications stored in memory module 210, which, when executed, perform the methods outlined below. For example, modules 202 to 208 could be functions or classes written in a programming language such as C++ or Java.

[0051] In some embodiments modules 202 to 208 are field programmable gate arrays (FPGAs) configured to execute the methods outlined below.

[0052] Memory module 210 could be any suitable form of memory. For example, memory module 210 may be volatile memory, such as cache or RAM, or non-volatile memory, such as an optical disk drive, hard disk drive, storage server or cloud storage or any combination of these.

[0053] Processing unit 212 may be hardware processor, such as a microcontroller or microprocessor, capable of receiving data and executing software applications to process the data. Processing unit 212 may be a single processor or multiple processors. For example, processing unit 212 may be a cloud based processing unit configured to carry out the methods outlined below.

[0054] Processing unit 212 leverages modules 202 to 208 to perform a method 400 illustrated in Fig. 4. Initially, transceiver module 202 performs step 402 of method 400 by receiving input colour image 109 generated by image capture device 108. Input colour image 109 represents scene 106. In some embodiments transceiver module 202 is connected to image capture device 108 via a physical cable and receives input colour image 109 over this cable. In other embodiments, transceiver module 202 is in wireless communication with image capture device 108 and receives input colour image 109 wirelessly. For example, input colour image 109 may be transmitted wirelessly from image capture device 108 to transceiver module 202 using Bluetooth wireless communication protocol.

[0055] Colour reducing module 204 receives input colour image 109 from transceiver module 202 and executes step 404 of method 400 by converting input colour image 109 to a reduced colour image. This is achieved by mapping colour values of input colour image 109 to palette colour values, wherein the palette colour values represent clusters in a colour space of the input colour image. The colour space refers to a representation of input colour image 109 wherein the pixels of image 109 are assigned a position in space based on their colour values and/or spatial location in image 109. The coordinate axes of the colour space are the fundamental colours used to capture and/or store the colour information of input image 109. Example fundamental colours are red, green, blue (RGB) or cyan, magenta, yellow, black (CMYK) etc.. The colour of each pixel of input image 109 is defined as some ratio of the fundamental colours, which can be considered as a vector, or location, in the colour space. Colour clusters are then defined using an appropriate metric for the space. For example, clusters can be defined using a Euclidean distance between pixels in the colour space where pixels that lie within a calculated Euclidean distance form a cluster. As mentioned though, other suitable metrics may also be used. For example, HSV space includes intensity plus colour angle and is not Euclidean. In other words, a distance over the colour space can be defined based on any metric over the space. The metric is a number to measure the distance between points that is always greater if the actual distance in the space is greater, or is always equal if equal. This metric need not be Euclidean. While specific clustering, such as k-means, can be used, other clustering algorithms can equally be applied. All pixels within a cluster are then mapped to/replaced by a common colour representative of that cluster to define the reduced colour image. These clusters define a palette, or range of colours, for the reduced colour image in the sense that each common colour representing one cluster is one of the palette colours. The colours may be stored separately from the image data as a colour palette but this is not necessary because the palette colours can be extracted from the reduced colour image. In this sense, mapping colour values of the input colour image to palette colour values means that there is a reduction to common colours from the full problem of the input image.

[0056] In some embodiments that use the spatial location to define the pixel location, the metric takes into consideration the size of the pixel clusters in the colour space. This helps to reveal isolated pixels of a given colour in image 109 and has the advantage that if there are only a few pixels of a colour in an area, they can be ignored as they are not a significant component of the image. This is particularly advantageous if the resolution of the display device is low compared with image 109, as small patches of a similar colour spread over the image can be ignored and not define a cluster. [0057] Map generation module 206 receives the reduced colour image data and executes step 406 of method 400 by generating a conversion map. The map is a transformation from an input image to an output image. It is generated by finding a transformation which maps the reduced colour image to a target greyscale image. The mapping assigns a contrast between greyscale values based on the colour contrast between palette colour values of the corresponding pixels in the reduced colour image. That is if two clusters have a large difference in colour space, they should have a large distance in greyscale space. Similarly, if two clusters have a smaller colour difference, they should have a lesser greyscale difference wherein the colour difference is determined by the metric over the colour space. Exemplary methodology for generating the conversion map is provided in greater detail below. As discussed below, the map can be stored for later use, and in some embodiments is stored as a matrix or array which specifies the transformation from the reduced colour image to an output image.

[0058] Image conversion module 208 performs step 408 of method 400, converting input image 109 to output greyscale image 101. The conversion is achieved by applying the conversion map to input colour image 109. This process is described in more detail below.

[0059] Transceiver module 202 then performs step 410, providing output image 101 generated by module 210 to image display device 102 where it can be rendered to provide information about scene 106 to user 104. In some embodiments, transceiver module 202 provides the output image data to display device 102 through a wired connection. In other embodiments, the output image data is transmitted to display device 102 wirelessly. For example, the output image data may be transmitted wirelessly using Bluetooth wireless communication protocol.

[0060] Another embodiment of processing module 110 is shown in Fig. 5 as processing module 110'. Processing module 110' is similar to processing module 110 but further comprises a sampling module 502. Aspects of modulel 110' which are in common with module 110 have been given corresponding annotation numbers and will not be described again.

[0061] Processing module 110' is configured to carry out a method 400', illustrated schematically in Fig. 6. Method 400' is similar to method 400 but further comprises step 602. Aspects of method 400' which are common to method 400 have been given corresponding annotation numbers and will not be described again.

[0062] Sampling unit 502 is configured to perform step 602 by downsampling input image 109 to a pixel configuration of the display device. The pixel configuration defines the number and spatial distribution of pixels in the display device. That is, the downsampling selects‘samples’ of the input image from locations which correspond to pixels in the display device. In another example, downsampling calculates an average or other aggregate value of neighbouring pixels to calculate a pixel value of a‘larger’, downsampled pixel. The effect of downsampling is a reduced number of pixels which are also in the spatial configuration of the pixels of the display device. This downsampling process is illustrated in Fig. 7, where high resolution image 702 is downsampled to low resolution image 704. In some embodiments, the pixel configuration is stored in memory 210. In other embodiments, processing unit 212 is configured to determine a pixel configuration from the connected display device. Greater detail of the sampling pattern is provided below in the sampling section.

[0063] This downsampled image is then provided to image conversion module 208 which performs step 408' of method 400'. At step 408', the conversion map is applied to the downsampled image, rather than the input image, to produce the output image. Applying the conversion map to the downsampled image reduces the computational requirements for converting a colour image to a greyscale image, thereby reducing energy requirements and latency between receiving the input image and rendering the output image. The

downsampling also takes into consideration irregular pixel configurations of the display device. The downsampling process is described in greater detail below including with application to irregular pixel configurations.

[0064] In some embodiments, the steps of method 400 or 400' are performed in a different order. For example, in one embodiment, step 602 is performed on input image 109 to produce the downsampled image. Step 406 is then performed, taking the downsampled image as an input. The target greyscale image produced in step 406 is then taken as output greyscale image 101. [0065] In other embodiments, a map is generated directly from input image 109 to produce a greyscale image. Clustering can then be performed on this greyscale image before downsampling to yield output greyscale image 101.

[0066] In some embodiments, image capture device 108 comprises two or more sensors for capturing input image data. For example, in some embodiments image capture device 108 comprises two digital cameras configured as a stereo vision camera. The stereo vision camera allows for depth/distance and other dimensional data from scene 106 to be determined. In some embodiments, this depth/distance data is used to modulate greyscale image 101, such that greyscale image 101 depends, to an extent, on the distance of features in scene 106 from user 104. For example, pixels of greyscale image 101 which represent features of scene 106 that are more than a predetermined distance from user 104 are reduced in intensity or set to zero. For example, in some embodiments, pixels representing features which are more than 4 metres from user 104 are set to zero. Doing this can help improve clarity of features of scene 106 which are closer to user 104 and reduce the chance of them being conflated with features which are further away.

FURTHER CONTRAST ENHANCEMENT

[0067] In some embodiments, a contrast enhancement method is performed on the greyscale image before it is provided to the display device. This process is illustrated in as method 800 in Fig. 8. Method 800 is performed after step 408 or 408’ and can be performed by image conversion module 208 or by a dedicated contrast module (not shown).

[0068] At step 802, a neighbourhood mean is determined for each pixel of the greyscale image produced at step 408 or 408'. The neighbourhood mean is based on an intensity of pixels having a predetermined spatial relationship to that pixel. For example, the

neighbourhood may be defined as pixels that are adjacent that pixel. Or, the spatial relationship may be pixels that are within a predetermined distance of that pixel.

[0069] In some embodiments, a weighted mean is used where pixels that are closer to that pixel are given a greater weighting. [0070] Whatever the spatial relationship used, the intensity of that pixel is compared to the neighbourhood mean at step 804.

[0071] The intensity of that pixel is then adjusted at step 806 to enhance contrast. The adjustment is performed by increasing that pixel intensity if the pixel intensity is greater than the neighbourhood mean or decreasing that pixel intensity if the pixel intensity is less than the neighbourhood mean.

[0072] Specifically, for each pixel y y from the greyscale image produced at step 408 or 408', we compute the contrast- adjusted value y' as:

[0073] where b is the contrast adjustment magnitude weight and the mean value is taken over some local neighbourhood N _y Y of y , where N _y is the set of the largest K pixel intensities nearest to y . This can be represented as:

[0074] Method 800 ensures that the contrast is not less than a just noticeable difference from its surrounding region.

SUPPRESSING LOCALISED NOISE

[0075] In some embodiments, a further method 900 is performed to suppress localised noise. In embodiments where method 800 is performed, method 900 is performed after method 800. In embodiments where method 800 is not performed, method 900 is performed after step 408 or 408’ and before step 410. Method 900 is performed by image conversion module 208 or by a dedicated noise suppression module (not shown).

[0076] At step 902 a neighbourhood mean is determined for each pixel of the greyscale image produced at step 408 or 408' or method 800. The neighbourhood mean is based on an intensity of pixels having a predetermined spatial relationship to that pixel. For example, the neighbourhood may be defined as pixels that are adjacent that pixel. Or, the spatial relationship may be pixels that are within a predetermined distance of that pixel.

[0077] Whatever the spatial relationship used, the intensity of that pixel is compared to the neighbourhood mean at step 904 as well as to maximum and minimum threshold values.

[0078] The intensity of that pixel is then adjusted at step 906, based on the comparison. Each pixel y is adjusted, or thresholded, according to the formula:

[0079] where T _{m n} and T _max are the minimum and maximum thresholds, respectively, and m _y is the mean value taken over some local neighbourhood. If T _{m n} and T _max are set to the same value, a binary phosphene response is obtained. An alternative method is to use adaptive thresholding where the threshold is set for each phosphene, i.e., T _{m n} (y ) and T _max (y ) . That way, the thresholds can be set according to local or global properties of the scene (e.g., large contrast at a distance that would not normally be captured by using universal thresholds).

[0080] The pixel values are thresholded so that small isolated abberations, such as spectral reflections, and local noise are removed. Through any initial image filtering (such as Lanczos2 used here), a spatially small highlight such as a spectral reflection can lead to a somewhat increased value output for a single location. Typical contrast adjustment, including the above, is performed on the basis of a mean computed across multiple neighbours. If a region is quite uniform, but there is a single highlight, this can result in one pixel value being a small amount above its neighbours. However, because it is above all its neighbours, this effect will be emphasised for a single location. On the other hand, if this were part of a large region of consistent intensity spanning several pixels, multiple neighbours may be similar resulting in a smaller mean difference. This is highly problematic in a visual prosthesis. For example, if the background neighbourhood region is all mapped to stimulation values near zero, and one value is mapped to a high, but somewhat sub-threshold level, it can distract the implantee as users will occasionally see sub-threshold stimulation results. Such a result can be highly distracting for the users as background flashes can cause confusion. Method 900 avoids this by suppressing such noise. This is done on the basis that if the maximum difference from any neighbour is small then this is not a significant area of contrast just because it is an isolated pixel, i.e., this suppresses the case where an isolated pixel is enhanced by method 800 that is set up to adjust for local means based on regions of constant brightness.

[0081] In some embodiments, adaptive thresholding is used where the threshold values (i.e., T _{m n} (y ) and T _max (y ) ) are set for each pixel. In this case, the thresholds can be set according to local or global properties of the scene (e.g., large contrast at a distance that would not normally be captured by using universal thresholds).

SUBSEQUENT IMAGES

[0082] In many instances, system 100 is required to periodically provide greyscale images 101 of scene 106. This allows user 104 to monitor and be aware of any changes which may occur in scene 106.

[0083] There are several options for generating subsequent output images. For example, in an embodiment the conversion map is generated for an initial input colour image and retained for one or more subsequent input colour images. This is illustrated as method 1000 of Fig.

10. As mentioned, method 1000 is performed on a subsequent input colour image, which is received at step 402'. Sampling step 602 is as described above.

[0084] At step 408 or 408' the output greyscale image is generated by applying conversion map 1002 to the downsampled image produced by step 602. Conversion map 1002 was produced for an earlier colour input image.

[0085] An effect of method 1000 is that the perceptual parameters of a scene, including white balance and exposure (the latter two are often performed via setting the camera configuration), can be retained or frozen. It provides the algorithm with temporal stability - with the palette changing every frame small changes in the scene could shift the conversion map and result in visually confusing results. For example, consider a situation in which the initial scene, and therefore image, consisted of red and green and the subsequent scene, and therefore image, consisted of only green. Then, the map generated for the initial image would map the red and green to light and dark contrasting intensities respectively. If a new map were generated for the subsequent image, values assigned to the green would change creating a visually confusing result. If the map from the initial image was retained for the subsequent image, then the intensities for the green would remain the same. In this way, the palette can be intentionally frozen as desired to create a stable transformation. This also reduces computational demands thereby reducing power consumption and latency in system 100.

[0086] In some embodiments, hysteresis is added to the map generation process so that the map adapts slowly to the scene. Essentially, hysteresis means that the map is dependent on the previously used map, reducing the amount of change between one map and the next compared to the case where no hysteresis is used. Instead of one threshold on the input defining the seperation between two output values, there may be two thresholds including a higher threshold for changing the output from low to high and a lower threshold for chaning the output from high to low. This allows the map to adapt slowly to the scene.

[0087] This can be performed, for example, by creating a new map for an image after some time, call this new map P2 , and comparing it with the map currently in existance, call this P1 . Compute the pairwise nearest neighbours between P1 and P2. Suppose P1a is the location of one element of P1 and P2a is the corresponding element of P2. Then in the updated palette, this element could be set to location (1-a) P1a + aP2a . As mentioned, this allows the subsequent conversion maps to adapt slowly to the subsequent input colour images. This avoids the situation where the perceptual parameters of the scene change too rapidly and create visually confusing output images.

MAP GENERATION

[0088] As mentioned, a conversion map is used to convert a colour (RGB) image to a greyscale output image. This map is also referred to as a transformation palette. This palette is essentially a lookup table of pairs of (RGB, Greyscale) values that can be used to convert the RGB samples into greyscale values that represent the potential intensity of phosphenes.

An exemplary transformation palette 1100 is illustrated in Fig. 11. Column 1102 is a list of colour values for the reduced colour image produced by step 404 and column 1104 are the corresponding greyscale values. [0089] The reduced colour image is produced by K-means clustering on the input colour image to cluster the image by RGB colour. This allows for the N most distinct colours present in the image to be found. In some embodiments, other clustering methods are used. For example, in one embodiment hierarchical clustering is used. In other embodiments, distribution based clustering is used and in another embodiment, density based clustering is used.

[0090] The result of the K -means clustering is a list of cluster centers. This can be considered to be the N distinct colours that are present in the image. With a suitably chosen N , most, if not all, of the important colours within the image will be represented.

[0091] A colour to greyscale conversion algorithm which maximises contrast between greyscale values for different palette colour values is then applied to the list of RGB clusters in the reduced colour image.

[0092] Methods that take a colour image and map it to a greyscale image, preserving colour contrast are known. For example, this can be performed by mapping the image to a colour space where the difference, as perceived by humans with normal vision, is uniform between colour and greyscale. That is, if a greyscale contrast is perceived as minor, and a colour difference is perceived as minor (e.g., shades of green) the distance between them in the space will be similar and small, whereas red to green contrast and dark grey to light will both have large distances between them. A colour to greyscale algorithm forms a mapping from colour space (3 parameters) to an output single parameter (intensity) output space that aims to preserve this contrast.

[0093] The colour to greyscale conversion algorithm ensures that input colours that have significant chromaticity difference, but low luminance difference, hence would map similar output greyscale values in a standard scheme, are made visually distinctive. For example, a red colour and a green colour when converted to greyscale naively could end up being similar greyscale values and would not be visually distinctive. The transformation method would ensure that those colours that are very visually distinct in RGB, are still visually distinct in greyscale by making adjustments during colour transformation [0094] For example, a suitable conversion algorithm is described in“Color2gray: Salience- preserving color removal," in ACM SIGGRAPH 2005 Papers, SIGGRAPH 05, (New York, NY, USA), pp. 634{ 639, ACM, 2005 by A. A. Gooch, S. C. Olsen, J. Tumblin, and B.

Gooch. Briefly, this connversion algorithm works by determining, for the colour image, a signed distance scalar measurement between each pixel in the colour image and that pixel's neighbours. The signed distance scalar measurement is a quantitative measure based on both the luminance/intensity differences and the chrominance differences between the pixels, and essentially defines the contrast between the pixels in the colour image. As an example, consider that the signed distance scalar measurement between a pixel i and neighbouring pixel j is d.

[0095] A greyscale image, having pixels which correspond to those of the colour image, is then defined. That is, the greyscale image has a pixel gi corresponding to pixel i of the colour image and a neighbouring pixel gj corresponding to the neighbouring pixel j of the colour image. Intensity values are then found for the pixels of the greyscale image such that contrast between a greyscale pixel and its neighbour is similar to the signed distance scalar

measurement between the corresponding pixels in the colour image. That is, intensity values for gi and gj are found such that g _i - g _j is close to d. These intensity values are found using an iterative optimization process.

[0096] After performing the colour to greyscale conversion, pair the original RGB inputs are paired with their greyscale outputs from the algorithm to create a colour to greyscale transformation palette such as that shown in Fig. 11.

DOWNSAMPLING

[0097] As mentioned above, conversion maps are applied to downsampled input images.

The images can be downsampled using any suitable technique. For example, Lanczos filtering can be used.

[0098] Based on the input sampling locations, defined by the pixel configuration, a Lanczos kernel is calculated for each sampling location. The size and properties of this kernel are determined by the distance of this pixel to its nearest neighbours. In the event the phosphene map is regular, this will likely be the same for each phosphene. In the case where the phosphene map is sparse and/or irregular, the Lanczos kernel generated for each phosphene will vary in size and composition based on the spacings. If all sampling locations are at the same location, the phosphene spacing will be equal to zero. If the phosphene spacing is equal to zero, then the Lanczos2 filter kernel degenerates to a lxl matrix equal to 1. Moreover, if required, border pixels are extrapolated by reflecting pixels across the border, e.g.

‘edcba|abcdefgh|hgfed’, where“|” is an image boundary and‘abc’ are pixel values. Each phosphene covers a region of interest in the image such that ideally most of the image's contents will be encoded into the phosphenes that are output by the algorithm.

[0099] When Lanczos2 filtering is applied, a low-pass filter with filter cutoff proportional to the spatial Nyquist frequency of the downsampled image. In other words, the cutoff is set to the Nyquist frequency defined by the resolution of the downsampled image, that is, twice the maximum spatial frequency in the downsampled image. The sampling period, also called the phosphene spacing, is determined by finding the mean of the set of distances between each sampling locations and that sampling location's nearest neighbour (aka mean of min).

[0100] The Lanczos2 filter kernel, lanczos2[x, y](a) , x, , is given by,

[0101] where a is the cutoff fraction, which is the frequency of the filter as a fraction of the Nyquist frequency, i.e cutoff frequency = cutoff fraction × Nyquist frequency, S is the phosphene spacing in pixels and A is a normalisation factor such that the sum of all weights in the kernel is 1.

[0102] Using the kernels generated in the above step, the input RGB image is sampled at each sampling location. This will produce a single RGB colour value for each sampling location representing the Lanczos filtered area that the sampling location/phosphene represents (we will call these RGB Samples). We end up with a list of RGB values, one for each sampling location in the image. The number will depend on the given sampling locations.

[0103] As mentioned above, the methods described here in can be applied to irregular or sparse pixel configurations. If the phopsphene map is irregular, the filter size is calculated by averaging the distance between the current phosphene and all neighbouring phosphenes, up to a threshold. If a phosphene is isolated (i.e., has no neighbouring phosphenes up to a threshold), then the kernel size is assumed to be zero and treated as above.

[0104] The conversion map described above will be applied to this downsampled image.

DYNAMIC RANGE CONVERSION

[0105] A patient map provides the perceivable dynamic range for each pixel of a display device. In some embodiments, the patient map is used to define the final pixel intesity of the greyscale image by converting a high dynamic range input to one that matches the dynamic range for that pixel as specified in the patient map.

[0106] This process is illustrated in Fig. 12. A high dynamic range image is represented by ‘steps’ 1202. The steps represent distinct intensity values. In this example, the dynamic range is illustrated as having 64 distinct possible values for intensity, or 6 bits of dynamic range.

[0107] The patient map provides the number of distinct intensity values which can be represented by a given pixel of the display device. In this example, the patient map indicates that a given pixel is only capable of representing 8 distinct intensity values (3 bits). This is represented by steps 1204.

[0108] When converting the dynamic range, a range of distinct values from the high dynamic range greyscale image are mapped to a single intensity value for the display device based on the patient map. For example, all intensity values in the range covered by bracket 1206 will be mapped to level 1 of the output device. Similarly, all intensity values in the range cover by bracket 1208 will be mapped to level 6 of the output device. [0109] Even if the dynamic range for a given pixel is only two levels (on and off), if the region is greater than its surrounds it will be mapped to maximum whereas its neighbouring areas that are not high will be mapped to zero. The overall effect on the image will be consistent throughout even when there are differences in dynamic ranges between pixels.

Such differences are common as the perception threshold frequently varies across the display device.

[0110] It should be understood that the techniques of the present disclosure might be implemented using a variety of technologies. For example, the methods described herein may be implemented by a series of computer executable instructions residing on a suitable computer readable medium. Suitable computer readable media may include volatile (e.g. RAM) and/or non-volatile (e.g. ROM, disk) memory, carrier waves and transmission media. Exemplary carrier waves may take the form of electrical, electromagnetic or optical signals conveying digital data steams along a local network or a publically accessible network such as the internet.

[0111] It should also be understood that, unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as "estimating" or "processing" or "computing" or "calculating",

"optimizing" or "determining" or "displaying" or“maximising” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that processes and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0112] It will be appreciated by persons skilled in the art that numerous variations and/or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.