Field of the Invention

The present invention relates to endoscopy and more specifically to polarimetric endoscopy in which the polarisation of light is used, as well as or instead of spectral intensity, to image the subject.

Background to the invention

Surgical endoscopy plays a key role in the diagnosis and intervention of cancer, for example laryngeal cancer. Typically, surgeons rely on standard white light endoscopy (WLE) to detect tissue pathologies based on colour contrast. However, this method provides unsatisfactory sensitivity and miss rate.

Tissues have polarimetric properties and thus will interact differently depending on the polarisation of the incoming light source, offering better tissue contrast as opposed to using typical colour contrast. The primary polarimetric properties of an imaged sample are depolarization, retardance (linear and circular) and diattenuation. For a linear optical system, the polarization of incident light are characterised by the input Stokes vector. The polarisation state of the light is modified following interaction with the sample. The polarization of the transmitted or reflected light is characterised by the output Stokes vector, which can be written as a product of the input Stokes vector and a 4 x 4 ‘Mueller’ matrix. A Mueller matrix is a complete mathematical description of the polarization characteristics of the object that interacts with light. Thus, Mueller polarimetric imaging measures Mueller matrices over a field of view and allows for the visualisation of the object’s polarization characteristics.

However, Mueller imaging polarimeters have their technical and economic challenges for applications in real-time operation procedures since the imaging process involves 16 sequential radiometric images which normally require long acquisition times. Stokes polarimetry on the other hand only requires measurement of the polarization state of the emergent light from samples and with circularly polarized light illumination, can be used to approximate two tissue polarization properties: circular depolarization and linear retardance.

Summary of the Invention

The present invention provides an endoscope system comprising: a shaft, the shaft defining an illumination channel and an imaging channel and having a first end which forms a tip and a second end; a light source arranged to direct light into the illumination channel; polarizing means arranged to polarize the light from the light source so that it is emitted from the tip as polarized light; a camera arranged to receive light from the imaging channel, the camera comprising a photodetector array and a polarizer array whereby the camera can generate a respective detector output signal indicative of the level of light having each of a plurality of different polarizations for each of a plurality of pixels; and processing means arranged to process the detector output signals to generate the value of a polarization parameter for each of the pixels, thereby to generate an image data set.

The polarized light may be circularly polarized, elliptically polarized, or linearly polarized.

The light source may have a fixed wavelength. Alternatively the light source may be switchable between a plurality of different wavelengths.

The imaging channel may be polarization maintaining. Alternatively the processing means may be arranged to apply a correction to the detector signals to compensate for the polarizing properties of the imaging channel.

The polarizing means may comprise a circular polarizer, an elliptical polarizer, or a linear polarizer. The circular polarizer may be located at the tip of the shaft. The circular polarizer may be arranged to convert light from the illumination channel to circularly polarized light. The circular polarizer may comprise a linear polarizer and a quarter wave plate.

The polarizer array may comprise a plurality of groups of polarizing elements. Each group of polarizing elements may comprise four polarizing elements each having a respective polarizing direction. The polarizing directions of the four polarizing elements may be at 45 degree intervals.

The processing means may be arranged to process the output signals to generate at least one of a retardance image data set and a depolarization image data set. The endoscope system may further comprise a quarter wave plate. The quarter wave plate may be switchable between an operative state in which it is in the light path from the imaging channel of the shaft to the camera and an inoperative state in which it is effectively removed from the light path. For example, it may be movable between an operative position, in which it is in a light path from the imaging channel of the shaft to the camera, and an inoperative position in which it is physically removed from the light path. Alternatively the quarter wave plate may be fixed in the light path, and it may have switchable polarization properties so that the quarter wave plate can be switched between polarizing and non-polarizing states, allowing for its effective removal from the light path but with more rapid switching than using a physically movable quarter wave plate. The processing means may be arranged to generate the depolarization image data set from detector signals output when the quarter wave plate is in the operative state. The processing means may be arranged to generate the retardance image data set from the detector signals output when the quarter wave plate is in the inoperative state.

The processing means may be arranged to generate the retardance image data set or the depolarization image data set as a snapshot image data set.

The depolarization image data set and the retardance image data set may be obtained directly from the detector output signals via a data driven method.

Alternatively the processing means may be arranged to determine a plurality of Stokes parameters for each of the pixels and to generate the image data set from the Stokes parameters. The plurality of Stoles parameters may comprise a partial set, i.e. an incomplete set, of Stokes parameters. The Stokes parameters may be So, Si and S2 and the image may be a retardance image. The processing means may be arranged to determine a value for the retardance AS for each of the pixels given by: A5=^(Si ^A 2+S2 ^A 2 )/So.

The stokes parameters may be So, and S3 and the image may be a depolarization image. The processing means may be arranged to determine a value for the depolarization Ad _ep for each of the pixels given by: Ade _P = 1-| S3/S0I .

The endoscope system may further comprise, in any workable combination, any one or more features of the preferred embodiments of the invention as will now be described with reference to the accompanying drawings.

Brief Description of the Drawings

Figure 1 is a perspective view of an endoscope according to an embodiment of the invention;

Figure 2 is a schematic view of the divisional of focal plane camera of the endoscope of Figure 1;

Figure 3 is a schematic view of the system of Figure 1 in use; and

Figure 4 is a flow diagram showing the processing steps performed by the system of Figure 1. Detailed Description

Referring to Figure 1, a polarimetric endoscope comprises a shaft 10 supported at one end 12 by a body 14. The body 14 also supports a polarization state analysing detector which may comprise a division-of-focal -plane linear polarimetric (DoFP-LP) camera 16, and may further comprise an objective lens 18 and may further comprise a quarter wave plate 20. The shaft 10 defines an imaging channel 22 and an illuminating channel 24. For example the imaging channel 22 may comprise a central core of the shaft, which may be of a circular cross section, and the illuminating channel 24 may be formed around the imaging channel 22, and may have an annular cross section. Either or both of the illuminating channel 24 and the imaging channel 22 may be formed from a bundle of optical fibres. A light port 25 may be provided in the shaft 10 and arranged to be connected to a light source to allow light to be injected into the illumination channel 24, for example using an angled mirror. A circular polarizer 26 may be arranged over the end of the illuminating channel 24 so that light transmitted down the illuminating channel 24 will be converted to circular polarized light at the tip of the shaft for emission into the subject. The shape of the circular polarizer is arranged to match the cross section of the illuminating channel, so if the illuminating channel is of annular cross section, then the circular polarizer is annular in shape. A tip component holder 27 may be provided to hold the circular polarizer 26, and any other components at the tip of the shaft 10, in place. For example the tip component holder may comprise a tubular body 27a arranged to fit onto the end of the shaft 10 and having an internal groove 27b or other retaining means arranged to hold the circular polarizer 26. As is well known, a circular polarizer typically comprises a linear polarizer and a quarter wave plate. Whilst one or both of these components could be located remote from the tip of the shaft 10, it is advantageous to provide them as close to the tip as possible so that the circular polarization is maintained as much as possible at the position where the illuminating light strikes the subject.

The imaging channel 22 defines an imaging light path 28 and the camera 16 is supported by the body 14 so that it is aligned with the imaging light path 28. Referring also to Figure 2, the camera 16 comprises a photodetector array 30 comprising an array of detector elements 32, which may conveniently be arranged in a planar square array. The camera 16 further comprises linear polarizer array 34 which comprises an array of linear polarizing elements 36. The polarizing elements 36 are arranged in groups 38, with each group comprising a plurality if polarizing elements, each with a respective different orientation. For example, each group 38 may comprise four polarizing elements with their polarization directions 45° apart as shown in Figure 2. Each of the polarizing elements is aligned with one of the detector elements 32, and each of the detector elements 32 is arranged to generate an output signal indicative of the intensity of light reaching the detector element 32. Therefore, for each group 38 of polarizing elements 36, four separate detector output signals are generated, each indicative of the intensity of a different polarization component of the detected light. Each group 38 of polarization elements may be treated as a pixel and the detector output signals for each pixel may be used to determine a parameter of the polarization of the light received at that pixel, whereby an image data set can be generated comprising a value of the parameter of polarization for each pixel of the image. Alternatively, for each of the four polarization directions, data for each of the detector elements 32 can be derived by interpolation from the measured output signals for the respective polarization directions, of which there is one in each group 38 of polarization elements. This can provide a higher resolution image. It will be appreciated that, in modifications to the embodiment shown, each polarizing element may be aligned with more than one detector element. Also different combinations of polarizations could be used for each pixel, although typically this would make processing of the signals more complicated.

The quarter wave plate 20 may be supported in the imaging optical path between the shaft 10 and the camera 16, and may be movable between an operative position in which it is in the imaging optical path and an inoperative position in which it is moved out of the imaging optical path. In the embodiment shown the quarter wave plate 20 has a clip-in frame which allows it to be clipped in place in the imaging optical path as shown, and manually removed from the optical path. Other suitable methods of achieving insertion and removal of the quarter wave plate 20 will be apparent to the skilled person, such as a manually or electrically operated sliding plate or a rotatable support. The quarter wave plate 20, when in the imaging optical path, can allow the generation of a depolarization image, and when removed from the imaging optical path, can allow the generation of a retardance image as will be described in more detail below. In an alternative arrangement, the quarter wave plate 20 may be fixed in the light path, and may be an electro-optic device having switchable polarization properties. The electro-optic device may be connected to a manual input switch, for example on the body 14. The quarter wave plate can then be switched between polarizing and non-polarizing states, allowing for its effective removal from the light path but with more rapid switching than using a physically movable quarter wave plate. Examples of suitable electro-optic devices include photo-elastic polarization modulators and liquid crystal polarization modulators.

Further optical components may also be supported in the body 14, or integral to the camera 16, for example the objective lens 18. Referring to Figure 3, the system may further comprise a light source 40 which may be connected to the light port 25 by means of an optical guide 42. The light source is preferably a narrow band source, for example it may include a narrow band fdter. The light source 40 may for example comprise a high-pressure mercury lamp with a narrow band fdter centred on a suitable wavelength, such as 546nm. The fdter or the type of light source may be switchable or adjustable so as to enable different illuminating wavelengths to be used. The light guide 42 may be liquid light guide, or a solid light guide. The system may further comprise a processor 44, for example forming part of a computer 46, which is connected to the camera 16 by means of a data cable 48. The processor 44 may be arranged to generate image data sets from the raw data generate by the detectors 32. The system may further comprise a display screen 49 so that images defined by the image data sets can be displayed for viewing.

Referring to Figure 4, the processor 44 may be arranged to process the output signals from the detectors 32 of the camera 16, which form raw detector data, using one or more of the following steps to generate one or more image data sets. In particular one value for each of the detector elements 32 (or each group of four detector elements 32 aligned with a group 38 of polarization elements) can be used to calculate the value of retardance for each pixel, and a value of depolarization for each pixel, thereby generating a retardance image data set and a depolarization image data set. These images may each be in the form of a ‘snapshot’ image using a single measurement for each pixel, all of the measurements being made simultaneously.

Polarimetric correction

The polarizer array 34 of the DoFP-LP camera 16 may comprise an aluminum nano-wire grid on the top of the photodiode array 30. The diattenuation of the micro-linear polarizers 36 is generally variable due to the fabrication processes and may result in artefacts in the reconstructed polarimetric images. This may be addressed through a calibration procedure that determines the diattenuation of each of the micro-linear polarizers 36, and uses a correction procedure, for example as specified in Powell SB, Gruev V. Calibration methods for division- of-focal-plane polarimeters. Opt Express. 2013;21(18):21039-55. The calibration result is camera-specific but is normally only performed once for a particular DoFP-LP camera. This correction procedure is therefore applied to the raw image data 50 to generated corrected image data 52.

Demosaicing

The DoFP-LP camera 16, as described above, has interlaced linear polarizers 36 aligned at 90°, 45°, 135° and 0° from an arbitrary reference direction on top of the photodiode array 30. This means that, for each polarization direction, the raw image data only includes one intensity value for each group 38 of polarization elements. This data could be used to provide a sub-image for each polarization direction having one pixel for each group 38 of polarization elements. However to increase the resolution, bilinear interpolation may be used to demosaic the polarimetric corrected image data and generated four sub-images for example of 1384x 1208- pixels. The sub-images are referred to as I90, 145, 1135 and Io respectively 54, 56, 58, 60.

Reconstruction of partial Stokes polarimetric images

For the retardance mode, the quarter waveplate 20 is removed, and the processor is arranged to obtain the first three elements of Stokes parameters So, Si and S2 from the four sub-images:

For the depolarization mode, the 45° quarter waveplate 20 is placed in the imaging optical path 28 and the sub-images Io and I90 effectively correspond to the left and right circularly polarized light images. The processor is arranged to obtain the first and fourth elements of Stokes parameters So and S3 from the sub-images:

So = Io + loo

(2)

S3 = Io — loo

Reconstruction of retardance, depolarization and their intensity-references

The reconstruction method performed by the processor may, for example, but based on the method described in Qi J, He H, Lin J, Dong Y, Chen D, Ma H, et al. Assessment of tissue polarimetric properties using Stokes polarimetric imaging with circularly polarized illumination. J Biophotonics. 2018;l l(4):e201700139. Tissue Mueller polarimetric studies indicate that depolarization and linear retardance are found to be the main polarization characteristics of interest and utility, and the magnitude of diattenuation for the majority of tissue types is typically very small, with only a small number of exceptions such as skeletal and heart muscles. For tissues with very low diattenuation like the larynx, the main mechanism to convert circular to linear polarized light is tissue retardance. Given that fully circularly polarized light is used for illumination, the magnitude of retardance Ag is therefore determined by the linearly polarized components within the emergent light, defined as where 5L is the linear phase retardance (in radian), and So, Si and S2 are obtained from Equation 1. Ag is a dimensionless quantity with minimum value 0 (non-retarding) and maximum value 1 (highly retarding) and can be used to reconstruct the retardance images. The So image generated from Equation 1 serves as the intensity-reference image for the retardance mode.

The circularly polarized illuminating light can maintain its polarization after one, or a small number, of scattering events, while multiple scattering randomises the polarization. The magnitude of depolarization Adep can thus be characterized by the proportion of the randomly polarized backscattered light, as follows where RP, PM and T refer to the randomly polarized, polarization maintaining, and total backscattered light intensity respectively. Note that the total and the polarization maintaining intensity are given by the first and fourth element of the Stokes parameters So and S3 obtained from Equation 2 respectively. Adep is a dimensionless quantity with minimum value 0 (nondepolarizing) and maximum value 1 (fully depolarizing) and was used to reconstruct the depolarization images. The So image generated from Equation 2 is the intensity-reference image for the depolarization mode of the SPE. Adep may overestimate depolarization for strongly retarding turbid media, since the emergent light may contain linearly polarized components, resulting in RP may be slightly smaller than T-PM in Equation 4.

It is noted that the retardance and depolarization obtained from this simplified approach based on partial Stokes polarimetry is not as rigorous as that based on Mueller polarimetry.

Optional processing of intensity-reference image

The intensity-reference images can be dark, especially for the areas at the periphery of the field- of-view due to inhomogeneous endoscope illumination, slight vignetting, presence of strongly absorbing objects like blood in addition to generally lower illumination power of the system compared with standard endoscopy. An optional correction, referred to herein as gamma correction, may be applied to the intensity-reference image according to Equation 5 to improve the visibility of these regions.

So_in and So_out are the intensity-reference before and after the gamma correction. A gamma value of 1/2 for example may be suitable. To address degradation of image contrast caused by gamma correction, the intensity-reference images may be enhanced with a built-in image sharpening function in MATLAB (imsharpen function). Additionally, under-exposed (caused by e.g. blood on the tissue surface) and over-exposed (caused by e.g. specular highlights) areas, referred to as low image quality areas, may be detected by thresholding the intensity-reference image before gamma correction, where bitdepth refers to the bit-depth of the raw output images from the LP DoFP camera 16 which may for example be 8. Low image quality areas may not be imaged by the system reliably and may therefore be identified, for example by colouring, to avoid misleading viewers.

The processing described above assumes that the endoscope itself does not affect the polarization of the light in the imaging channel 22, i.e. that the endoscope is fully polarization preserving. For some, though not most, commercially available endoscopes this is the case. However, if the endoscope is not polarization preserving, then the processing may include a further correction step to correct for the polarization effect of the endoscope. Provided that the Mueller matrix image of the rigid endoscope is non-degenerate (and ideally with negligible depolarization and diattenuation) and invariant with working distance, the Mueller matrix can be determined by measuring the detector signals for a number of uniformly polarized light sources, and the inverse of that Mueller matrix image can be post-multiplied by the Stokes parameter measurements, used for the retardance and depolarization imaging methods described above, to recover the polarization state at the distal tip of the endoscope.

It will be appreciated that, while the use of circular polarized light and polarization elements (polarizers) with four equally spaced polarizations in each group provide for ease of calculation of the retardance and depolarization, it is possible to use elliptical or even linear polarized illumination, and other combinations of polarization plates and still obtain useful values of retardance and depolarization.

For example, for a given group of polarizers, the first three elements in the first row of the Mueller matrix of each detection polarizer constitute a row vector. These row vectors in each group of detection polarizers can be stacked vertically to construct a first matrix Instrument Matrix 1 (Al).

The waveplate and the detection polarizers constitute a polarization analyzer to perform polarization analysis under depolarization mode. The first and the fourth elements in the first row of the Mueller matrix of each detection unit in a group (waveplate and an individual detection polarizer) constitute a row vector. These row vectors in each group of detection polarizers can be stacked vertically to construct a second matrix Instrument Matrix 2 (A2).

In order for the values of retardation and depolarization to be obtained, the detection polarizers need to be designed so that the Instrument Matrix 1 and Instrument Matrix 2 are of full row rank.

Under retardance mode, the first three elements of the Stokes parameter constituting a row vector SSI can then be obtained from the pixel values read from the detectors within a group, with those pixel values represented as a row vector Pl,

A1*SS1 = P1

Under depolarization mode, the first and the fourth elements of the Stokes parameter SS2 can be obtained from the pixel values of the detectors within a group that are represented as a row vector P2.

A2*SS2 = P2

With Al, A2 of full row rank, SSI and SS2 can be obtained by solving the equation above, via matrix inversion, pseudo-inversion, optimization, or other suitable methods.

The retardance and depolarization can then be obtained from SSI and SS2 respectively according to the equations (3) and (4).

Still further methods can also be used to obtain the values of the retardance and depolarization from the detector readings without calculating the Stokes parameters as an intermediate step. In such a method, for a given group of polarization elements comprising a predetermined number of polarization elements with predetermined linear polarization directions, and associated detector elements, and a predetermined illumination polarization such as circular or elliptical or linear, large amounts of detector readings can be obtained for different specific known values of retardance and depolarization, and from that data a direct mapping from the detector readings Pl and P2 to the tissue retardance and depolarization can be generated using data driven methods such as a deep convolutional neural network. This mapping can then be used in the endoscope to derive a pixel value of retardance and/or depolarization from the detector readings thereby to generate images similar to those derived by calculation of the Stokes parameters as described above.