Login| Sign Up| Help| Contact|

Patent Searching and Data


Title:
PATTERN ACTIVATED STRUCTURED ILLUMINATION LOCALIZATION MICROSCOPY
Document Type and Number:
WIPO Patent Application WO/2021/053245
Kind Code:
A1
Abstract:
An optical module for structured light microscopy and a structured light microscopy method are described, the method comprising illuminating an exposure area of a sample comprising photoactivatable fluorescent molecules with an activation light pattern, the activation pattern comprising one or more activation light maxima, the activation light activating one or more of the photoactivatable fluorescent molecules; sequentially illuminating the exposure area with a plurality of excitation light patterns, the plurality of excitation light patterns defining a cumulative excitation light pattern, the cumulative excitation light pattern defining one or more cumulative excitation light minima, the cumulative excitation pattern being positioned relative to the activation pattern such that each maximum of the activation light pattern coincides with one of the one or more cumulative excitation light minima, the excitation light of an excitation light pattern exciting one or more of the activated fluorescent molecules; detecting radiation emitted by the photoactivatable fluorescent molecules which are activated by the activation light pattern and excited the plurality of excitation light patterns; and determining position information of one or more of the photoactivatable fluorescent molecules based on the detected radiation.

Inventors:
SMITH CARLAS SIERD (NL)
CNOSSEN JOHAN PIET (NL)
Application Number:
PCT/EP2020/076337
Publication Date:
March 25, 2021
Filing Date:
September 21, 2020
Export Citation:
Click for automatic bibliography generation   Help
Assignee:
UNIV DELFT TECH (NL)
International Classes:
G02B21/00; G01N21/64; G02B21/06; G02B21/36; G02B27/56; G02B27/58
Domestic Patent References:
WO2014046606A12014-03-27
WO2017153430A12017-09-14
Foreign References:
US20090237501A12009-09-24
US10247672B22019-04-02
Other References:
PALM, BETZIG ET AL.: "Imaging Intracellular Fluorescent Proteins at Nanometer Resolution", SCIENCE, vol. 313, 2006, pages 1643 - 1645
STORM, RUST ET AL.: "Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM", NATURE METHODS, vol. 3, 2006, pages 793 - 795
BALZAROTTI ET AL.: "Nanometer resolution imaging and tracking of fluorescent molecules with minimal photon fluxes", SCIENCE, vol. 355, pages 606 - 612, XP055474328, DOI: 10.1126/science.aak9913
GWOSCH ET AL., MINFLUX NANOSCOPY DELIVERS MULTICOLOR NANOMETER 3D-RESOLUTION IN (LIVING) CELLS, Retrieved from the Internet
J. CNOSSEN ET AL.: "Localization microscopy at double precision with patterned illumination", BIORXIV 554337
CNOSSEN ET AL., LOCALIZATION MICROSCOPY AT DOUBLE PRECISION WITH PATTERNED ILLUMINATION
A.G.YORK ET AL.: "Instant Super-Resolution Imaging in Live Cells and Embryos via Analog Image Processing", NATURE METHODS, vol. 10, no. 11, 2013, pages 1122 - 1126, XP055554918, DOI: 10.1038/nmeth.2687
B.-J. CHANG ET AL.: "Universal light-sheet generation with field synthesis", NATURE METHODS, vol. 16, 2019, pages 235 - 238, XP036725909, DOI: 10.1038/s41592-019-0327-9
J.D. MANTON ET AL.: "Structured illumination microscopy with extended axial resolution through mirrored illumination", ARXIV:1810.04590V1
Attorney, Agent or Firm:
DE VRIES & METMAN (NL)
Download PDF:
Claims:
CLAIMS

1. A structured light microscopy method comprising: illuminating an exposure area of a sample comprising photoactivatable fluorescent molecules with a predetermined, inhomogeneous activation light pattern, the activation pattern comprising one or more activation light maxima and one or more activation light minima, the activation light activating one or more of the photoactivatable fluorescent molecules; sequentially illuminating the exposure area with a plurality of excitation light patterns, the plurality of excitation light patterns defining a cumulative excitation light pattern, the cumulative excitation light pattern defining one or more cumulative excitation light minima, the plurality of excitation light patterns being positioned relative to the activation pattern such that each maximum of the activation light pattern coincides with one of the one or more cumulative excitation light minima, the excitation light of an excitation light pattern exciting one or more of the activated fluorescent molecules; detecting radiation emitted by the photoactivatable fluorescent molecules which are activated by the activation light pattern and excited by the plurality of excitation light patterns; and determining position information of one or more of the photoactivatable fluorescent molecules based on the detected radiation.

2. The method as claimed in claim 1, further comprising: determining a scanning pattern defining scanning positions; repositioning the activation light pattern and the plurality of excitation light patterns relative to the sample in accordance with the scanning pattern; and, for each scanning pattern position, repeating the steps as claimed in claim 1.

3. The method as claimed in claim 1 or 2, wherein the plurality of excitation light patterns comprises a first series of sinusoidal light patterns having a first orientation, wherein the sinusoidal light patterns have the same period and have equidistant phases, the maximum phase difference being in the interval 30 to 150 degrees, preferably in the interval 60 to 90 degrees.

4. The method as claimed in any of the preceding claims, wherein the plurality of excitation light patterns comprises two or more series of sinusoidal light patterns with the same orientation, wherein the orientations of the series are equally distributed over 180 degrees.

5. The method as claimed in claim 4, wherein the activation light pattern defines a regular rectangular, preferably square, grid with a grid distance, and wherein the plurality of excitation light patterns comprises a first series of sinusoidal light patterns in a first orientation and a second series of sinusoidal light patterns in a second orientation, the second series being orthogonal to the first series, and the excitation light patterns having a period equal to the grid distance.

6. The method as claimed in claim 4, wherein the activation light pattern defines a regular triangular grid with a grid distance, and wherein the plurality of excitation light patterns comprises a first series of sinusoidal light patterns in a first orientation, a second series of sinusoidal light patterns in a second orientation, and a third series of sinusoidal light patterns in a third orientation, the angle between the first and second orientations being 60 degrees and the angle between the second and third orientations being 60 degrees, and the excitation light patterns having a period equal to the grid distance.

7. The method as claimed in any of claims 3-6, wherein each of the series of sinusoidal light patterns comprises three sinusoidal light patterns.

8. The method as claimed in any of the preceding claims, further comprising: detecting a background and selecting a plurality of excitation light patterns based on the detected background.

9. The method as claimed in any of the preceding claims, further comprising: determining new activation light patterns and/or excitation light patterns, based on the determined position information of one or more of the photoactivatable fluorescent molecules; and, executing the method steps of at least claim 1 using the new activation light patterns and/or excitation light patterns.

10. A microscope for obtaining super-resolution images of a sample comprising photoactivatable fluorescent molecules, the microscope comprising: a sample holder for holding the sample; a first light source and first beam forming optics configured for illuminating an exposure area of the sample with activation light having a predetermined, inhomogeneous activation light pattern, the activation light pattern defining activation light maxima; a second light source and second beam forming optics configured for illuminating the exposure area with a plurality of excitation light patterns, the plurality of excitation light patterns defining a cumulative excitation light pattern, the cumulative excitation light pattern defining one or more cumulative excitation light minima, the plurality of excitation light patterns being positioned relative to the activation pattern, such that each maximum of the activation light pattern coincides with one of the one or more cumulative excitation light minima, the excitation light of an excitation light pattern exciting one or more of the activated fluorescent molecules; a detector for detecting radiation emitted by the photoactivatable fluorescent molecules which are activated by the activation light pattern and excited by the plurality of excitation patterns; and a processor for determining position information of one or more of the photoactivatable fluorescent molecules based on the detected radiation .

11. A microscope according to claim 10, wherein the second beam forming optics comprise a computer-controllable polarisation controlling element and one or more gratings, configured to generate a plurality of pre-determ ined interference patterns.

12. An optical module for use in a localisation microscope, the optical module being configured for determining position information of one or more photoactivatable fluorescent molecules in a sample, the sample defining a sample plane, the optical module further being configured for: illuminating an exposure area of a sample with a predetermined, inhomogeneous activation light pattern, the activation pattern defining one or more activation light maxima and activation light minima in the sample plane, the activation light activating one or more of the photoactivatable fluorescent molecules; sequentially illuminating the exposure area with a plurality of excitation light patterns, the plurality of excitation light patterns defining a cumulative excitation light pattern, the cumulative excitation light pattern defining one or more cumulative excitation light minima in the sample plane, the cumulative excitation light pattern being positioned relative to the activation light pattern such that each of the one or more activation maxima coincides with one of the one or more cumulative excitation light minima, the excitation light exciting one or more of the activated fluorescent molecules; detecting radiation emitted photoactivatable fluorescent molecules which are activate and excited by the activation light pattern and the series of excitation light patterns; and determining position information of one or more of the photoactivatable fluorescent molecules based on the detected radiation.

13. The optical module according to claim 12, further configured for: illuminating the exposure area with one or more excitation light patterns in a direction orthogonal to the sample plane.

14. A computer comprising a a computer readable storage medium having computer readable program code embodied therewith, and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform the method according to one or more of claims 1-9.

15. A computer program or suite of computer programs comprising at least one software code portion or a computer program product storing at least one software code portion, the software code portion, when run on a computer system, being configured for executing the method according to one or more of the claims 1- 9.

16. A non-transitory computer-readable storage medium storing at least one software code portion, the software code portion, when executed or processed by a computer, is configured to perform the method according to one or more of the preceding claims 1-9.

Description:
Pattern activated structured illumination localization microscopy Field of the invention

The disclosure relates to structured illumination and localization microscopy and, in particular, though not exclusively, to methods and systems for structured illumination, and a computer program product enabling a computer system to perform such methods.

Background of the invention

In conventional (optical) microscopy the resolving power is limited to an amount on the order of the ratio between the wavelength of the light that is used in the image formation and the numerical aperture of the microscope. Different techniques for circumventing this so-called diffraction limit have emerged over the last two decades. These techniques are generally referred to as super-resolution microscopy. An important class of super-resolution microscopy techniques is the family of localization microscopy techniques, such as Photo-Activation Localization Microscopy (PALM, Betzig etal. “Imaging Intracellular Fluorescent Proteins at Nanometer Resolution”, Science 313: 1643-1645, 2006) and STochastic Optical Reconstruction Microscopy (STORM, Rust etal., “Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM)”, Nature Methods 3: 793-795, 2006).

In these techniques the sample of interest is labelled with a set of fluorophores, which may be fluorescent molecules or fluorescent labels. The specimen is illuminated with light of a specific wavelength (or wavelengths) which is absorbed by the fluorophores, called excitation light. This causes the fluorophores to emit light of longer wavelengths (i.e. , of a different colour than the absorbed light). The illumination light is separated from the much weaker emitted fluorescence through the use of a dichroic mirror and/or emission filter.

A special group of fluorophores is photoactivatable fluorophores, i.e. fluorophores that can be switched between a fluorescent state and a non-fluorescent state by illuminating the fluorophores with activation light, respectively deactivation light. Depending on the parameters of the activation light, these fluorophores can be stochastically switched between a light emitting state (“on”-state) and a dark state (“off-state) such that at any instant in time only a sparse subset of all fluorophores is in the on-state. This has the consequence that light from individual fluorophores can be measured by a point or array detector such as a camera. The position of the individual emitters in the plane imaged onto the camera can then be determined with a precision, which is typically on the order of 10 nm depending on photon count, about 10-20 times smaller than the diffraction limit. Repeating this process, i.e. recording many time frames of such sparse images and analysing all frames for the emitter locations, finally gives an image of the sample with details on the order of the localization precision rather than on the order of the diffraction limit. Because of the need to collect and analyse many images, the temporal resolution of these methods is relatively low.

Nevertheless, these super-resolution fluorescence microscopy techniques have gained much traction in medical and biological research, as they are able to provide unique insights that no other microscopy method can. They may be used to study e.g. cells or organisms in vivo, with labelling that is precise enough to follow biological systems at nanometer resolution. One consistent problem with super-resolution microscopy is that the measurement precision and image quality typically depends on the number of recorded photons and therefore requires high- intensity laser illumination. This may lead to so-called photo-toxicity, which disrupts the natural behaviour of the organisms and shortens their life. Localization precision is typically proportional to the square root of the illumination intensity, so reducing the illumination intensity with a factor 10 reduces the resolution with a factor V10 = 3.16. Therefore, there are many efforts in the field to increase the localization precision for a given illumination intensity, or conversely, to reduce the required illumination intensity for a given localization precision.

US10247672 describes a super-resolution method referred to as Pattern Activated Non-Linear Structured Illumination Microscopy (PA-NL-SIM). The method comprises providing patterned activation radiation and patterned (or structured) excitation radiation to a sample that includes photo-transformable optical labels (PTOLs) such as photo-activatable fluorescent molecules, wherein the activation light and the excitation light each have an optical parameter (e.g. intensity) that varies periodically in space, possibly with the same spatial period. The concept of structured illumination microscopy (SIM) exploits the fact that when two patterns are superimposed multiplicatively a beat pattern (similar to a Moire pattern) will appear in the product of the two patterns. One pattern can be the unknown sample structure emitting signal light, and the other pattern can be the purposely structured pattern of excitation light. Because the amount of signal light emitted from a point in the sample is proportional to the product of the local excitation light intensity and the relevant structure of the sample, the observed signal light image that is detected by the detector will show the beat pattern of the overlap of the two underlying patterns. Because the beat pattern can be coarser than those of the underlying patterns, and because the illumination pattern is known, the information in the beat pattern can be used to determine the normally unresolvable high-resolution information about the sample. PA-NL-SIM uses a combination of an activation pattern and a excitation pattern to generate a fluorescence emission pattern comprising additional information.

WO201 7/153430 describes a method for high-resolution imaging of a structure with fluorescent markers, comprising scanning (part of) a sample with a zero point of an illumination intensity distribution, where the zero point is typically surrounded by a ring-shaped or doughnut-shaped intensity maximum. An activation region comprising and surrounding the zero point may be irradiated with activation radiation, and the activation region may be surrounded by and partially overlap with a deactivation region. Using switchable luminescence markers, the whole sample may be scanned by varying the active regions over a grid and within the active regions, varying the scanning zero point. The method disclosed in WO2017/153430 may use (or can be regarded as an extension of) the MIN FLUX method, described in Balzarotti et al., ‘Nanometer resolution imaging and tracking of fluorescent molecules with minimal photon fluxes’, Science Vol. 355, Iss. 6325, pages 606-612. MINFLUX uses the same excitation light zero-point scanning as the method disclosed in WO201 7/153430, but no activation light or deactivation light. Combining MINFLUX with an activation beam is also described in Gwosch et al., ‘MINFLUX nanoscopy delivers multicolor nanometer 3D-resolution in (living) cells’, available at https://www.biorxiv.Org/content/10.1101 /734251 v1.

J. Cnossen etai, ‘Localization microscopy at double precision with patterned illumination’, bioRxiv 554337, disclose a method called SIMFLUX. SIMFLUX may be considered a combination of MINFLUX with Structured Illumination Microscopy (SIM). The structured illumination in SIMFLUX consists of quickly switched (with respect to position and orientation) sinusoidal excitation light patterns to illuminate a sample including fluorescent molecules. The fluorescent photon count is recorded and stored for a number of frames, the number depending on the number of excitation patterns. The information from the frames is combined to determine a molecule position, based on the diffraction pattern, the intensity, and the excitation pattern. In the known SIMFLUX method, the combination of the excitation patterns provides a uniform illumination of the sample leading to a uniform increase in localization precision. This technique results in an image resolution gain of 2.2 or an reduction of the required illumination light by a factor of 4.84 when compared with state of the art methods that use uniform illumination. Despite the progress that is made in the super-resolution microscopy, there is a need for methods and systems for super resolution microscopy that have a further improved spatial resolution, a reduced illumination intensity and/or an increased image acquisition rate.

Summary of the invention

It is an objective of the embodiments in this disclosure to reduce or eliminate at least one of the drawbacks known in the prior art. In an aspect, the invention may relate to a method for structured illumination microscopy comprising: illuminating an exposure area of a sample comprising photoactivatable molecules with a predetermined, inhomogeneous activation light pattern comprising one or more activation light maxima, the activation light activating one or more of the photoactivatable fluorescent molecules; sequentially illuminating the exposure area with a plurality of excitation light patterns, the plurality of excitation light patterns defining a cumulative excitation light pattern, the cumulative excitation light pattern defining one or more cumulative excitation light minima, the cumulative excitation light pattern being positioned relative to the activation pattern, such that each maximum of the activation light pattern coincides with one of the one or more cumulative excitation light minima, the excitation light of an excitation light pattern exciting one or more of the activated fluorescent molecules; detecting radiation emitted by the photoactivatable fluorescent molecules which are activated by the activation light pattern and excited by the plurality of excitation patterns; and, determining position information of one or more of the photoactivatable fluorescent molecules based on the detected radiation.

The position information may be used to e.g. reconstruct an image, preferably a super-resolution image, of the at least part of the sample. The determining of the position information may use information on the activation light pattern and on the plurality of excitation light patterns. An activation light pattern may also be referred to as an activation pattern, and an excitation light pattern may be referred to as an excitation pattern.

Conventional structured illumination methods use illumination or excitation patterns which are selected such that the cumulative or combined (e.g. summed or time-integrated) illumination intensity of a series of illumination patterns to the sample is uniform or as close to uniform as possible. In contrast, the structured illumination microscopy scheme according to an embodiment of the invention uses illumination patterns which are selected such that a cumulative illumination intensity of a series of illumination patterns is inhomogeneous, comprising low intensity areas (minima) and high intensity areas (maxima). That way, the illumination patterns may be selected to maximize the information per photon, in particular the position information. The inhomogeneous cumulative excitation light pattern is combined with an inhomogeneous activation light pattern, comprising low intensity areas (minima) where there is a low probability of activating a fluorescent molecule, and high intensity areas (maxima) where there is a high probability of activating a fluorescent molecule. By aligning the activation pattern to the plurality of excitation patterns in such a way that the one or more maxima of the activation pattern coincide with the one or more minima of the cumulative excitation pattern, only the low intensity areas of the cumulative excitation patterns are used, resulting in a high information per photon and a low photo-toxicity.

Fluorescent molecules generally have a ‘fixed’ photon budget, typically about 1500-5000 photons per molecule (such as e.g. Alexa Fluor 647), before they stop functioning. Therefore, maximising the information per photon not only reduces the sampling time to obtain a given localisation precision, but also increases the maximally obtainable localisation precision. The information per photon can effectively be increased by adding a priori information, e.g. by using a known activation and/or excitation pattern, which may result in a statistical probability distribution of localisations. This process may also be executed in closed loop: once initial information on the position of molecules has been obtained, the activation and/or excitation light patterns may be re-optimized to further increase the information per photon, and hence the localisation precision. This process of re optimisation may be repeated for a number of frames.

The information per photon may be the Fischer information per photon emitted by the photoactivatable fluorescent molecules, where the Fischer information may relate to a position of a fluorescent molecule. The position may be estimated using a maximum likelihood estimation algorithm. In an embodiment, the method may comprise determining maxima of the Fischer information divided by the cumulative excitation pattern, based on at least the plurality of excitation patterns. The method may further comprise positioning the plurality of excitation patterns relative to the activation pattern maxima such that the maxima of the Fischer information divided by the cumulative excitation pattern coincide with the maxima of the activation pattern. This way, the information per photon may be maximised, and consequently, the total available information before the fluorescent molecules go dark.

In a typical embodiment, maxima of the Fischer information per photon may coincide with minima in the cumulative excitation pattern. Thus, positioning the plurality of excitation patterns such that the one or more maxima of the Fischer information divided by the cumulative excitation light pattern coincide with the one or more maxima of the activation light pattern, may comprise positioning the plurality of excitation light patterns such that one or more minima of the cumulative excitation light pattern coincides with the one or more maxima of the activation light pattern.

A minimum in the cumulative excitation light, as defined above, may be obtained by using for example different excitation patterns with coinciding minima for at least some of the minima, or, preferably, by using spatially shifted versions of the same excitation pattern, wherein the shift is smaller than the pattern size. However, many other patterns are also possible, including even random patterns, provided the pattern is known with sufficient precision and comprises intensity gradients and preferably intensity zero points.

In the absence of background fluorescence, the information per photon is highest close to the minima (or ideally: zero-points) of the excitation illumination patterns. The use of excitation patterns selected such that the cumulative excitation pattern is non-uniform, therefore, results in areas with a high localisation precision, corresponding to the minima of the cumulative excitation pattern, and areas with a low localisation precision, corresponding to the maxima of the cumulative excitation pattern. By activating only the fluorescent molecules around the minima of the excitation illumination patterns, only the high-resolution areas are selected for imaging and subsequent data analysis.

In an embodiment, the method may further comprise: determining a scanning pattern defining scanning positions, and, repositioning the activation light pattern and the plurality of excitation light patterns relative to the sample in accordance with the scanning pattern and, for each scanning pattern position, repeating the previously defined method steps, i.e. , illuminating an exposure area of a sample comprising photoactivatable molecules with an activation light pattern comprising one or more activation light maxima, the activation light activating one or more of the photoactivatable fluorescent molecules; sequentially illuminating the exposure area with a plurality of excitation light patterns, the plurality of excitation light patterns defining a cumulative excitation light pattern, the cumulative excitation light pattern defining one or more cumulative excitation light minima, the excitation light pattern being positioned relative to the activation pattern, such that each maximum of the activation light pattern coincides with one of the one or more cumulative excitation light minima, the excitation light of an excitation light pattern exciting one or more of the activated fluorescent molecules; detecting radiation emitted by the photoactivatable fluorescent molecules which are activated by the activation light pattern and excited by the plurality of excitation patterns; and determining position information of one or more of the photoactivatable fluorescent molecules based on the detected radiation.

By scanning a part of the sample with the area(s) from which data are obtained, it is possible to obtain high-resolution data from a large part of the sample, possibly the entire sample. The number of scanning positions may depend on the distance between the high activation light intensity areas relative to the size of the high activation light intensity areas. In some embodiments, 2x2 scanning positions may be sufficient, other embodiments may use more or less.

In an embodiment, the plurality of excitation light patterns may comprise a first series of sinusoidal light patterns with a first orientation, wherein the sinusoidal light patterns have the same period and have equidistant phases, the maximum phase difference being in the interval 30 to 150 degrees, preferably in the interval 60 to 90 degrees.

Such a partial series of sinusoidal light patterns may be defined as: sin(f x — Pc f c ), /? c = 0, ... , L/ c , with L/ c > 1 and 30 L/ C < f c < 150 °/L/ c , and where the activation light pattern has period (2 p / f) in the x direction.

A sinusoidal excitation pattern can be advantageously created by interference of two plane waves. Additional excitation patterns may advantageously be created by phase shifting the interference pattern.

In an embodiment, the plurality of excitation light patterns may comprise two or more series of sinusoidal light patterns having the same orientation, wherein the orientations of the series may be equally distributed over 180 degrees.

For example, a plurality of excitation patterns may comprise two orthogonal series of sinusoidal excitation patterns, or three series of sinusoidal excitation patterns with a mutual angle of 60°.

The activation light pattern and/or the plurality of excitation light patterns may be periodic pattern in one or more directions.

In an embodiment, the activation light pattern may define a regular rectangular, preferably square, grid with a grid distance, and the plurality of excitation light patterns may comprise a first series of sinusoidal light patterns in a first orientation and a second series of sinusoidal wave patterns in a second orientation, the second series being orthogonal to the first series, and the excitation light patterns having a period equal to the grid distance.

A small number of excitation patterns per orientation increases the imaging rate, allowing a higher temporal resolution. A small number of orientations likewise reduces the temporal resolution, and may additionally simplify the optical set-up. In a different embodiment, the activation light pattern may define a regular triangular grid with a grid distance, and the plurality of excitation light patterns may comprise a first series of sinusoidal wave patterns in a first orientation, a second series of sinusoidal wave patterns in a second orientation, and a third series of sinusoidal wave patterns in a third orientation, the angle between the first and second orientations being 60 degrees and the angle between the second and third orientations being 60 degrees, and the excitation light patterns having a period equal to the grid distance.

Such an embodiment may be advantageous when e.g. an existing, conventional SIM-setup using three pattern orientations is adapted to make use of elements of this disclosure.

In an embodiment, each of the series of sinusoidal wave patterns may comprise three sinusoidal wave patterns.

Three excitation light patterns per series of excitation light patterns may be a suitable number to strike a balance between increase in localisation precision, exposure time and cumulative exposure luminosity. This may result in, on average, a high information to photon count ratio.

In an embodiment, the method may further comprise: detecting a background and selecting a plurality of excitation patterns based on the detected background. At least the optimal phase distance between sinusoidal waves in a series of excitation patterns may depend on the detected background.

In an embodiment, the method may further comprise: determining new activation light patterns and/or excitation light patterns, based on the determined position information of one or more of the photoactivatable fluorescent molecules; and, executing the method steps of one or more of the methods described above using the new activation light patterns and/or excitation light patterns.

Based on initially determined localisation information, the activation light patterns and/or excitation patterns may be (re-)optimised to further increase the information per photon. An example of such an optimisation is a (phase) shift of excitation patterns relative to an activation pattern.

In a second aspect, the invention relates to a microscope for obtaining super-resolution images of a sample comprising photoactivatable fluorescent molecules, the microscope comprising: a sample holder for holding the sample; a first light source and first beam forming optics configured for illuminating an exposure area of the sample with activation light having a predetermined, inhomogeneous activation light pattern, the activation light pattern defining activation light maxima; a second light source and second beam forming optics configured for illuminating the exposure area with a plurality of excitation light patterns, the plurality of excitation light patterns defining a cumulative excitation light pattern, the cumulative excitation light pattern defining one or more cumulative excitation light minima, the plurality of excitation light patterns being positioned relative to the activation pattern, such that each maximum of the activation light pattern coincides with one of the one or more cumulative excitation light minima, the excitation light of an excitation light pattern exciting one or more of the activated fluorescent molecules; a detector for detecting radiation emitted by the photoactivatable fluorescent molecules which are activated by the activation light pattern and excited by the plurality of excitation patterns; and a processor for determining position information of one or more of the photoactivatable fluorescent molecules based on the detected radiation.

The processor may use the position information for generating an image of at least part of the sample on the basis of the detected radiation.

In an embodiment, the second beam forming optics may comprise a computer-controllable polarisation controlling element and one or more gratings, configured to generate a plurality of pre-determined interference patterns.

In a further aspect, the invention may relate to an optical module for use in a localisation microscope, the optical module being configured for determining position information of one or more photoactivatable fluorescent molecules in a sample, the sample defining a sample plane, the optical module further being configured for: illuminating an exposure area of a sample with a predetermined, inhomogeneous activation light pattern, the activation pattern defining one or more activation light maxima and one or more activation light minima in the sample plane, the activation light activating one or more of the photoactivatable fluorescent molecules; sequentially illuminating the exposure area with a plurality of excitation light patterns, the plurality of excitation light patterns defining a cumulative excitation light pattern, the cumulative excitation light pattern defining one or more cumulative excitation light minima in the sample plane, the plurality of excitation light patterns being positioned relative to the activation light pattern such that each activation light maxima coincides with one of the one or more cumulative excitation light minima, the excitation light of an excitation light pattern exciting one or more of the activated fluorescent molecules; detecting radiation emitted photoactivatable fluorescent molecules which are activated by the activation light pattern and excited by the plurality of excitation light patterns; and, determining position information of one or more of the photoactivatable fluorescent molecules based on the detected radiation.

In an embodiment, the optical module may further be configured for illuminating the exposure area with one or more excitation light patterns in a directional orthogonal to the sample plane. Such a module may comprise a second light source for illuminating an exposure area of a sample with excitation light and beam-forming optics for generating an excitation pattern in the sample in the axial direction. Such beam-forming optics may comprise means for generating a self interference pattern in the excitation light in the axial direction, orthogonal to the sample plane.

In an embodiment, the invention may relate to a structured light microscopy method comprising: illuminating a sample comprising photoactivatable fluorescent molecules with an axial activation light pattern, the axial activation pattern comprising one or more activation light maxima in an axial direction of the sample, the activation light activating one or more of the photoactivatable fluorescent molecules; sequentially illuminating the sample with a plurality of axial excitation light patterns, the plurality of axial excitation light patterns defining a cumulative excitation light pattern, the cumulative excitation light pattern defining one or more cumulative excitation light minima in the axial direction of the sample, the plurality of cumulative excitation light patterns being positioned relative to the activation pattern such that each maximum of the activation light pattern coincides with one of the one or more cumulative excitation light minima, the excitation light of an axial excitation light pattern exciting one or more of the activated fluorescent molecules; detecting radiation emitted by the photoactivatable fluorescent molecules which are activated by the activation light pattern and excited the plurality of axial excitation light patterns; and, determining axial position information of one or more of the photoactivatable fluorescent molecules based on the detected radiation.

In an embodiment, the invention may relate to a microscope for obtaining super-resolution images of a sample comprising photoactivatable fluorescent molecules, the microscope comprising: a sample holder for holding the sample; a first light source and first beam forming optics configured for illuminating the sample with activation light having an axial activation light pattern, the axial activation light pattern defining one or more activation light maxima in an axial direction of the sample; a second light source and second beam forming optics configured for illuminating the sample with a plurality of axial excitation light patterns, the plurality of axial excitation light patterns defining a cumulative excitation light pattern, the cumulative excitation light pattern defining one or more cumulative excitation light minima in the axial direction of the sample, the cumulative excitation light pattern being positioned relative to the axial activation pattern, such that each maximum of the axial activation light pattern coincides with one of the one or more cumulative excitation light minima, the excitation light of an axial excitation light pattern exciting one or more of the activated fluorescent molecules; a detector for detecting radiation emitted by the photoactivatable fluorescent molecules which are activated by the activation light pattern and excited by the plurality of axial excitation patterns; and, a processor for determining axial position information of one or more of the photoactivatable fluorescent molecules based on the detected radiation.

In an embodiment, the second beam forming optics may be configured for illuminating the sample from a first direction with a first beam of plane waves of coherent excitation light, and for illuminating the sample from a second direction, the second direction being the counter-direction of the first direction, with a second beam of plane waves of coherent excitation light, the first beam and the second beam interfering to generate an axial excitation light pattern.

In a further aspect, the invention may relate to a computer comprising a computer readable storage medium having computer readable program code embodied therewith, and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform the method steps according to any of the process steps described above.

In a further aspect, the invention may also relate to a computer program product comprising software code portions configured for, when run in the memory of a computer, executing the method steps according to any of the process steps described above.

In a further aspect, the invention may also relate to a non-transitory computer-readable storage medium storing at least one software code portion, the software code portion, when executed or processed by a computer, is configured to perform, executing the method steps according to any of the process steps described above.

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system”. Functions described in this disclosure may be implemented as an algorithm executed by a microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied, e.g., stored, thereon.

Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non- exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including a functional or an object oriented programming language such as Java(TM), Scala, C++, Python or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer, server or virtualized server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

Aspects of the present invention are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor, in particular a microprocessor or central processing unit (CPU), or graphics processing unit (GPU), of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer, other programmable data processing apparatus, or other devices create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. For example, and without limitation, illustrative types of hardware logic components that may be used include Field-Programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application- Specific Standard Products (ASSPs), System-on-a-Chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by special purpose hardware- based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The invention will be further illustrated with reference to the attached drawings, which schematically will show embodiments according to the invention. It will be understood that the invention is not in any way restricted to these specific embodiments.

Brief description of the drawings

Fig. 1A and 1B illustrate a known structured illumination microscopy method;

Fig. 2 depicts a schematic of the formation of an activation pattern for a structured illumination method according to an embodiment of the invention;

Fig. 3A and 3B illustrate activation patterns for a structured illumination method according to an embodiment of the invention;

Fig. 4 illustrates a schematic exposing a sample to an activation pattern for a structured illumination according to an embodiment of the invention;

Fig. 5 depicts a block diagram of a patterned activation structured illumination microscopy method according to an embodiment of the invention;

Fig. 6 schematically depicts a system for a pattern-activated structured illumination method according to an embodiment of the invention;

Fig. 7A and 7B schematically illustrate the gain in localization accuracy that may be achieved using the embodiments described in this application;

Fig. 8 illustrates a patterned activation structured illumination microscopy method according to an embodiment of the invention;

Fig. 9 schematically depicts part of a structured illumination microscopy system according to an embodiment of the invention;

Fig. 10A-10C depict the localization improvement in the axial direction for a pattern-activation structured illumination microscopy scheme according to an embodiment of the invention; and

Fig. 11 is a block diagram illustrating an exemplary data processing system that may be used for executing methods and software products described in this application.

Detailed description Fig. 1A depicts a known structured illumination microscopy method. In particular, the figure illustrates a schematic on the working of the so-called SIMFLUX method as described by Cnossen et al. in their article ‘Localization microscopy at double precision with patterned illumination’] only a single orientation is shown. A sample comprising a fluorescent molecule 102 is illuminated with excitation light with a series of three sinusoidal wave intensity patterns 104 I-3 with 3 phase shifted patterns with equal phase steps of 120° per orthogonal orientation of the line pattern. The excitation pattern has high intensity areas 108 (dark) and low intensity areas 110 (light). For each excitation pattern, the fluorescent radiation is detected and an image is recorded, resulting in three images 114 I-3 for a first orientation of the line pattern. Each image comprises pixels, where the pixel value represents the photon count. A higher photon count is represented with a lighter colour. Even in the absence of a fluorescent molecule in a focal plane, some radiation may be detected, resulting in background noise. First localization information may be obtained by estimating the centroid of the bright spots, i.e. the regions with a photon count higher than the background noise. The estimated centroid location is indicated by a circle 112 I-3 , with the size of the circle proportional to the uncertainty in centroid location.

Second localization information may be obtained by correlating the photon count 120 I-3 with the excitation pattern 122 as shown in graph 124. A high photon count is associated with a high excitation light intensity, and a low photon count with a low excitation light intensity. The photon counts for the three images per orientation are known to be spaced (in this case) 120° apart. As the position of the excitation pattern is known, this provides additional a priori information (modulo the wavelength of the excitation pattern). The first localization information and the second localization information may be combined to effectively increase the localization precision. The same procedure may be executed with a second series of excitation patterns orthogonal to the first series of excitation patterns, to increase the localization precision in both directions. The measurement may be repeated several times. The measurements may then be combined 126 into a localization image 128, in which the estimated location 130 of the fluorescent molecule is shown with an “X” and a circle representing the uncertainty. This method results in an improvement of localization precision with a factor of about 2.2, compared to conventional position estimation with uniform illumination, i.e. Single Molecule Localisation Microscopy (SMLM).

Fig. 1B depicts a one-dimensional cross-section of illumination intensities of a structured illumination microscopy method as described with reference to Fig. 1 A. It is not desired to have the localization precision vary over the field of view (FoV), therefore a substantially uniform localization precision is obtained by phase-shifting sinusoidal excitation patterns with equal phase shifts as shown in the figure. A series of excitations patterns 132 comprises three sinusoidal 120 degree phase-shifted excitation patterns 134 I-3 , that correspond to the patterns 104-i_ 3 in Fig. 1. The local minima 136-i_ 3 of the excitation patterns are equally spaced over a sample extending along the horizontal axis. The total received excitation light 138 is the same for each point in the sample. In the SIMFLUX method the average improvement drops if non-equal phase steps are used because the improvement varies non-uniform ly over the FoV.

The inventors have surprisingly found that a non-uniform intensity distribution of the excitation light in the FoV may be used to increase the performance of a structured light microscopy scheme, such as the SIMFLUX method. In particular, the invention exploits characteristics of certain excitation light patterns that have a non-uniform cumulative intensity distribution, wherein in some areas of the excitation light pattern the improvement in localization drops compared to known structured microscopy methods (such as the SIMFLUX scheme), while in other areas of the excitation light pattern the improvement in localization increases compared to known structured microscopy methods.

Further, the inventors have surprisingly found that predetermined activation light patterns may be used to increase the performance of the structured light microscopy scheme. The activation patterns will stochastically activate photo- activatable fluorescent proteins at targeted positions over the FoV. This creates an additional source of a priori information for improving the localization of the fluorophores, e.g. fluorescent proteins. The additional a priori knowledge regarding the position of fluorophores in the FoV effectively increases the amount of information per photon. Combining the use of the activation and excitation light patterns will provide a substantial improvement in the performance compared to state of the art structured light microscopy schemes.

Fig. 2 depicts one-dimensional cross-section of an example of activation and excitation light patterns according to an embodiment of the invention.

A sample comprising photoactivatable fluorescent molecules and defining a sample plane may be illuminated with activation light, wherein the activation light has a wavelength configured to activate photoactivatable fluorescent molecules in the sample. As shown in the figure, the activation light 202 has a spatial intensity pattern defining one or more activation areas 204, wherein an activation area in the sample plane may define an area with a high activation light intensity surrounded by regions with low activation light intensity. An activation area may e.g. be defined as the FWHM region or twice the FWHM region of a local maximum in an activation light pattern, or as e.g. the region with a >50 % or >95% activation probability. Within a FoV, the sample may be exposed to a plurality of activation areas wherein each activation area may be shaped according to the spatial intensity pattern of Fig. 2.

The plurality of activation areas may be arranged in a 2-dimensional plane forming an activation pattern.

In the activation areas, the activation light intensity is sufficient to activate photoactivatable fluorescent molecules. The probability that a photoactivatable fluorescent molecule is activated, is proportional to the intensity of the activation light at the position of the photoactivatable fluorescent molecule. Due to the shape of the spatial intensity pattern, this probability is negligible for the area outside the activation areas. The activation light pattern may have any sort of structure, and may for instance be structured as pin points in a regular square grid or triangular grid. Activation light patterns may be regular or irregular; examples of regular patterns are the aforementioned square or triangular grids or sinusoidal waves; in principle, any pattern can be used as activation pattern or excitation pattern in a series of excitation patterns, provided the activation pattern can be configured to activate primarily or exclusively photo-activatable molecules in the high-precision regions of the series of excitation patterns, in particular photo-activatable molecules in the regions with a relatively low cumulative excitation light dose defined by the series of excitation patterns.

Activation light patterns may also be formed by light sheets. An advantage of using engineered light sheets is that activation may be limited to a single layer or small region in the axial direction, thus reducing background noise and reducing phototoxicity. One or more parallel light sheets may be engineered for providing activation patterns and/or excitation patterns in the sample plane with a limited axial extent.

After illumination with the activation light, the sample may be illuminated with a consecutive series of excitation light patterns 206. In the embodiment of Fig.

2, the series may comprise a plurality (e.g. three) of excitation light patterns having a sinusoidal shape along an axis of the image plane (defining one orientation in the image plane). In a typical embodiment, excitation light patterns may be used in two orthogonal orientations or three 60° spaced orientations. Other arrangements however are also possible. The sample may be consecutively illuminated with the excitation patterns, i.e. the excitation patterns are used one after another.

In an embodiment, the series may alternate between the plurality of orientations. For example, the sample may be illuminated using a first sinusoidal excitation light pattern in a first orientation. Thereafter, the sample may be sequentially illuminated by the first sinusoidal excitation light pattern in a second orientation, a second sinusoidal excitation light pattern in the first orientation, the second sinusoidal excitation light pattern in the second orientation, etc. Here, the second sinusoidal excitation light pattern may be a phase-shifted version of the first sinusoidal excitation light pattern. The excitation light may excite one or more of the activated fluorescent molecules with a probability that is proportional to the intensity of the excitation light.

As shown in Fig. 2, the excitation light pattern may have low-intensity regions, typically local minima 208 I-3 and high-intensity regions. The excitation light pattern and the activation light pattern are configured such that low intensity regions of the excitation patterns 208 I-3 are located in or near the activation areas 204. An activation area may be defined as the area where the intensity of the activation light is greater than or equal a predetermined percentage of a maximum activation light intensity, e.g. at least 50 % or at least 25 % of the local maximum activation light intensity. An activation area may also be defined as the area where the probability of activating a light-activatable molecule is at least a predetermined number, e.g. an activation probability of at least 50 % or at least 10 %. Hence, the excitation patterns may have the same period as the activation pattern, or an integer multiple thereof. Preferably, each minimum in the excitation patterns is associated with an activation area, to minimize phototoxicity. The cumulative excitation light 210 exhibits a pattern with local minima and local maxima, wherein the local minima approximately coincide with maxima of the activation pattern. This way, the amount of information per photon is optimised.

Similar to the activation light pattern, the excitation light patterns may have any sort of structure, preferably selected depending on the activation pattern used. Excitation light patterns may be regular or irregular; examples of regular patterns are the sinusoidal waves shown in Fig. 2, or square or triangular grids. If use is made of an irregular activation pattern, the excitation pattern may be similarly irregular, the excitation patterns and/or the cumulative excitation pattern preferably having minima at or near the maxima of the activation light pattern, as this way, the high information per photon regions of the activation pattern coincide with the high information per photon regions of the series of excitation patterns.

Like activation light patterns, excitation light patterns may be formed by light sheets, providing similar advantages, e.g., limiting excitation to a single layer or small region in the axial direction, thus reducing background noise and reducing phototoxicity. Activation light sheets and excitation light sheets may be used in combination or separately.

In a typical embodiment, excitation light patterns in a consecutive series of excitation light patterns of the same orientation may be phase-shifted copies of each other. This way, an excitation light pattern may be reused by a pattern generating device. The increase in localization precision increases with the frequency of the excitation light pattern. Therefore, in an embodiment, the maximum possible frequency for each excitation pattern may be used. The phase step (phase difference) between such phase-shifted copies may be selected based on other parameters, e.g. the influence of the background. In general, the phase step should strike a balance between a large enough phase step that different excitation patterns lead to a detectably different response, and a small enough phase step that a sufficient increase in localization precision is obtained.

In an embodiment, the activation light pattern and/or the excitation light patterns may be adjusted, e.g. shifted, to maximise the information per photon. For example, after illumination with an activation light pattern and a first cycle of excitation light patterns, information from the detected photons may be used to adjust the position of the excitation patterns in a second cycle. The excitation light patterns may e.g. all be shifted with the same amount, or the phase difference between excitation light patterns may be adjusted. For example, if there are many detections close to activation light maxima, the phase difference between excitation light patterns may be reduced, lowering the cumulative excitation radiation near activation light maxima.

The radiation emitted by the activated and excited fluorescent molecules may be detected by (an array of) point detectors, such as a camera, and processed to obtain a localization information improvement, which may in turn be used to reconstruct an image, preferably a localization image, of at least part of the sample.

In an embodiment, the excitation light patterns may be selected such that a cumulative or combined, e.g. summed or time-integrated, excitation intensity 210 of a series of excitation patterns 210 is inhomogeneous, comprising low intensity areas and high intensity areas. Such a minimum in the cumulative excitation light may be obtained by e.g. using different excitation light patterns with coinciding minima for at least some of the minima such as a sinusoidal light pattern with (odd) integer multiple periods of a certain base frequency, or, preferably, by using spatially shifted versions of the same excitation pattern, wherein the shift is smaller than the pattern size.

The aforementioned localization precision enhancing properties of the combined activation and excitation light patterns are strongest close to the minima (or ideally: zero-points) of the excitation light patterns, as the localization precision enhancement, or increase in information per photon, relative to standard Single- Molecule Localization Microscopy depends on the ratio between the gradient of the intensity of the excitation pattern and the intensity of the excitation pattern itself. The use of excitation patterns which are selected such that the cumulative excitation pattern is non-uniform, therefore results in high-resolution areas, corresponding to the minima of the cumulative excitation pattern, and low-resolution areas, corresponding to the maxima of the cumulative excitation pattern. Activating the fluorescent molecules around the minima of the excitation illumination patterns, will result in the selection of areas with a high information content per photon. Moreover, as the location of the activation area is known, additional a priori information is added which may be used to increase the accuracy of the localization estimate. By shifting the activation and excitation patterns relative to the sample, the entire sample, or at least a relevant portion of it, may be scanned. The use of an activation pattern has two additional benefits besides a localisation improvement. The fluorescent molecules are more likely to be activated in or near areas of optimum localisation precision. As a non-limiting example, this region may be of the order of 0.1 micron to 5 micron, preferably about 0.5 to 1.5 micron, for instance about 850 nm; this way the overall exposure of the fluorescent molecules is reduced and bleaching effects can be suppressed. Furthermore, the background radiation (“noise”) may be reduced by limiting the number out of focus of fluorescent molecules that contribute to the background.

The aforementioned information per photon, i.e. per photon emitted by a fluorescent molecule, may be referred to as the Fischer information, which is a well- established concept to define how much we know about a specific estimated variable. The photons may be detected by a camera with, typically, a two- dimensional array of detector elements corresponding to pixels in an image obtained by the camera. The detection of photons, and hence the intensity values in the image are generally distributed according to a Poisson distribution, also known as shot- noise.

A fluorescent molecule gives only a limited number of photons before going into a dark state, and as a result, it is advantageous to maximize the amount of information per photon. In localization microscopy, an aim is to determine spatial coordinates (x, y, z) of the molecule in 2 or 3 dimensions by determining spatial information ( q c , q n and, optionally, q z ) based on the detected photons, e.g. using a maximum likelihood estimation algorithm.

Determination of the Fisher information is generally based on a model /½ , /(#) representing the expected value of a pixel k in a camera frame / for a given set of molecule parameters Q, e.g. location parameters 0 x y z , fluorescent molecule photon count parameters q,, and parameters for a model of the fluorescence background b k (9 b ). The model m ¾ (0) may be defined as: where b k (9 b ) is the number of background photons, PSF fc (0) is the point spread function of the optical system, A k is the area of pixel k and P f (9) defines the excitation light intensity at frame /. Thus, in general, the expected number of detected photons from the fluorescent molecule is proportional to the excitation light intensity.

The Fischer information matrix 1(0) of the set of parameters Q for a measurement over F frames, each frame comprising N pixels, may then be given by: where I(0) i; - is the (i,j) element of the matrix. That is, the Fischer information depends on a sum over all pixels in all frames of the relative derivative of the number of detected photons and is generally highest where a high gradient is combined with a low value of p kj {9). As mentioned above, m ¾ (0) may depend on the variables that are estimated, which may include e.g. location information and a photon count 9 h and other aspects of the measurements, such as fluorescent background radiation and properties of the point spread function.

In an embodiment, the method may comprise determining maxima of the Fischer information divided by the cumulative excitation pattern, based on at least the plurality of excitation patterns. The method may further comprise positioning the plurality of excitation patterns relative to the activation pattern maxima such that the maxima of the Fischer information divided by the cumulative excitation pattern coincide with the maxima of the activation pattern. This way, the information per photon may be maximised, and consequently, the total available information before the fluorescent molecules go dark.

For example, for estimation of the x coordinate of a molecule, the combination of excitation patterns may be chosen such that is maximized at or close to a maximum of the activation pattern, is the expected number of photons from the emitting fluorescent molecule. When multiple dimensions and/or multiple fluorescent molecules are considered, a trade-off can be made between the estimation precision of the parameters.

In the example depicted in Fig. 2, maxima of the Fischer information per photon may coincide with minima in the cumulative excitation pattern 210. Thus, positioning the plurality of excitation patterns 208 I-3 such that the one or more maxima of the Fischer information divided by the cumulative excitation light pattern coincide with the one or more maxima of the activation light pattern, may comprise positioning the plurality of excitation light patterns such that one or more minima of the cumulative excitation light pattern coincides with the one or more maxima of the activation light pattern.

Fig. 3A and 3B depict examples of two-dimensional scanning patterns according to various embodiments of the invention. Scanning patterns 302,304 may be used to scan a part of a sample with an activation pattern and corresponding excitation patterns. Other embodiments may use different scanning patterns, preferably scanning patterns that are periodic in one or more directions.

As shown in Fig. 3A, a first activation pattern 302 covering part of a sample (the field of view or FoV) may comprise activation areas 306-i_ n , depicted as solid spots. In an embodiment, a sinusoidal light pattern along the main directions of the grid 316,318 may be used to excite fluorescent molecules in the sample. The sinusoidal waves may be selected to have the same period as the activation pattern. Flence, the distance between two minima of an excitation pattern may be substantially equal to the distance between to neighbouring activation areas. Preferably, the position of the activation areas relative to the position of the minima of the sinusoidal light patterns may be known with great precision, e.g. better than 1 nm.

After illuminating the sample with a first activation pattern, e.g. the shown activation pattern, the sample may be illuminated with excitation light in a series of excitation patterns, and measurements associated with the first activation pattern may be obtained. Sometime after the illumination with the activation pattern, most activated fluorescent molecules may have become deactivated. The sample may then be illuminated with a shifted activation pattern, e.g. shifted along a first principal axis 318, such that e.g. activation area 306 n is shifted to a new position 308. Localization measurements associated with the shifted activation pattern may then be obtained. The same procedure may be repeated, shifting the activation pattern along the principal axes 316,318 such that activation area 306 n is shifted to additional positions 310,312, for which additional localization measurements may be obtained. This way, high-resolution images from practically the entire sample region may be obtained in a limited number of data acquisition steps (in this example four). In the depicted embodiment, the diameter of the activation area may be approximately half the period of the activation pattern. In a different embodiment, the diameter of the activation area may be smaller than half the period of the activation pattern. In such an embodiment, more scanning positions may be required. Reducing the size of the activation area may require longer measuring times, but may provide a higher localization precision or less interference from nearby activation areas.

Fig. 3B illustrates a second activation pattern 304 comprising activation areas depicted as solid circles. Similar as explained above, the activation pattern may be shifted to effectively scan a sample; e.g. activation area 320 may be shifted along axes 326,328,330 to positions 322,324, thus covering most of the sample. Measurements may be obtained by illuminating the sample with e.g. sinusoidal excitation patterns in the directions of the axes 326,328,330 for each position of the activation pattern. An advantage of a triangular pattern 304, compared to a square pattern 302 may be that less activation pattern positions are required to cover the sample, and that the method may be more easily combined with e.g. a structured illumination microscope generating sinusoidal light patterns in three directions. Different embodiments may use different activation patterns.

Fig. 4 depicts a schematic illustration of a plurality of time instances -A , in this case four time instances, of a single region 450 of a sample, wherein at each time instance a particular part of the region is exposed to activation and excitation light. In an embodiment, the depicted region may have a size of about 220 nm by 220 nm. At a first time instance, the sample may be illuminated with activation light defining an activation area 452. At a second time instance, the sample may be illuminated with activation light defining an activation area 460, which corresponds to area 454 in the depiction of the first time instance. In a third time instance, the activation pattern may be shifted to create an activation area in position 462, corresponding to area 456, and in a fourth time instance, the activation area may be shifted to position 464, corresponding to area 458. In a different embodiment, the positions may be probed in a different order. The positions 452,454,456,458 may also be referred to as first, second, third and fourth scanning position.

For each activation area 452,460,462,464, the sample may be illuminated with a series of excitation patterns. Cross-sections of an activation light pattern 472 and excitation light patterns 470-i_ 6 are schematically shown for a horizontal direction 466 and for a vertical direction 468. The cross-sections 472 of the activation light pattern may show (for each time instance) a single activation pattern. For the excitation light patterns 470 I-6 , only one dimension of each cross-section is shown, the cross-section in the other direction may be essentially constant.

Excitation light corresponding to the six excitation patterns may be used to illuminate the sample, one pattern after another. The excitation patterns corresponding to one activation area may be used in any order. Together, the four time instances may form an example measurement sequence, where the activation pattern and excitation patterns may be moved around the field of view together to measure all molecules with optimal localization precision.

Fig. 5 depicts a block diagram of a structured illumination microscopy method according to an embodiment of the invention. In a first step, a sample may be provided 502, preferably to a sample holder of a fluorescence microscope. The sample may comprise photoactivatable fluorescent molecules. At least part of the sample may be illuminated 504 with activation light that has a spatial activation pattern. The activation pattern may define one or more activation areas, preferably a plurality of activation areas in a grid such as a square or triangular grid. In the activation areas, the intensity of the activation light is sufficient to activate one or more photoactivatable fluorescent molecules. The sample is subsequently illuminated with excitation light in a consecutive series of spatial excitation patterns. The excitation patterns may be selected based on the activation pattern. In an embodiment, the excitation patterns may have, in one or more orientations, the same spatial period as the activation pattern. In an embodiment, the excitation patterns may be selected such that, in a certain orientation, the period of the excitation pattern is 1/n, n = 1 , 2, ... , times the period of the activation pattern in the same orientation.

In another embodiment, the activation pattern may form a square grid, and the series of excitation patterns may comprise two orthogonal partial series of light patterns, as explained in more detail with reference to Fig. 4.

A partial series of excitation light patterns having the same orientation may comprise phase shifted copies of a single excitation light pattern. In an embodiment, the excitation light patterns may be selected such that a cumulative or combined excitation pattern, e.g. a sum or time integral, may have intensity minima that at least approximately coincide with the maxima of the activation pattern. In an embodiment, the excitation patterns may be chosen such that a minimum of each excitation pattern overlaps with an activation area. The excitation light may excite one or more of the activated fluorescent molecules. The different excitation light patterns may be generated one after another. At least a full series of excitation light patterns may expose part of the sample during an activation time of the photoactivatable fluorescent molecules.

The fluorescent radiation emitted by the activated and excited fluorescent molecules may be detected 508 by a photodetector, e.g. an EMCCD camera, a CMOS camera or any suitable imager. During or after each time interval during which an excitation pattern is radiated, the fluorescent radiation may be detected and the photon count per detector pixel may be stored in association with the excitation pattern. The steps of illuminating the sample with an excitation pattern and detecting the fluorescent radiation may be repeated for at least one full series of excitation patterns. In an embodiment, measurements may be repeated over a plurality of series of excitation patterns. In an embodiment, excitation patterns may be re-optimised 516, based on the detected radiation. Re-optimising may give a scaling proportional to x W2 where k is the number of iterations as described in Gwosch et al. In an embodiment, after one more full series of excitation patterns, the activation and excitation patterns may be shifted 512 relative to the sample. In an embodiment, the sample may be shifted, in another embodiment, the patterns may be shifted by e.g. manipulation of the light travel path.

The stored photon counts and excitation pattern information may be used to reconstruct 510 a localization image of the sample, for example according to the method explained by J. Cnossen et al. (discussed in the background section and with reference to Fig. 1). In an embodiment, a priori information about the activation pattern may be used to localize fluorescent molecules in the sample and/or to reconstruct a localization image, for instance using maximum likelihood estimation and/or using the method detailed by J. Cnossen et al. In an embodiment, the background radiation may be detected 514 to optimize the excitation patterns, e.g. to optimize the phase shift between excitation patterns in a partial series of excitation patterns sharing a same orientation.

Fig. 6 schematically depicts a structured illumination microscopy system according to an embodiment of the invention. The system may comprise a first optical module, an excitation module, for generating a sinusoidally varying excitation light pattern in the plane of the sample and a second optical module, an activation module, for generating an activation light pattern. The first and second optical modules may be controlled by a computer to enhance the lateral localization precision of the system.

The excitation module may generate a sinusoidally varying excitation light pattern using a first light source 602 and computer-controlled spatial light modulator 612. The spatial light modulator may be configured as a grating that is capable of producing excitation light patterns of a desired phase and orientation. To that end, the first light source 602, preferably a single mode coherent source, e.g. a 640 nm laser, may be used to generate an excitation light beam, which may be expanded using a telescope 604,606 and guided via a polarizing beam splitter (PBS) 608 and a half wave plate (HWP) 610 towards the SLM.

The polarizing beam splitter and a half wave plate (HWP) may be used to match the chief ray to the normal axis of the spatial light modulator. Each pixel of the SLM behaves as a half-wave plate with a fast axis either vertical or at 45 degree depending on the state at which each pixel is programmed. As a result, a plane wave reflected off the SLM is phase modulated by pi between the pixels in two different states. Modulated light reflected from the SLM may be projected via the PBS to a mask 616 which may be used to transmit the +/- 1 diffraction order and block (higher) stray orders. This way, the modulation contrast may be improved. To align the polarization with that of the grating, a zero-vortex half wave plate (ZV-HWP) 618 may be placed after the mask. This way, the polarisation may be aligned with the direction of the interference pattern to increase contrast and reduce background light, resulting in a zero-point in the excitation light pattern where the intensity is as close to zero as possible.

The activation module may include a second light source 632, e.g. a 405 nm laser, for generating an activation light beam. The activation light beam may be expanded using a suitable optical system 634,636 and guided towards a microlens array 638, which may be configured to generate an array of focused light spots (foci) 640. The generation of the array of light spots is described in more detail in e.g. A.G.York et al., Instant Super-Resolution Imaging in Live Cells and Embryos via Analog Image Processing" , Nature Methods 10:11 (2013) pages 1122-1126.

The contrast of the focused light spots may be increased using an activation mask 642 comprising an array of holes that match the array of light spots. Precise positioning of the generated activation light pattern relative to the excitation light pattern may be accomplished using a suitable optical system 644,646 in combination with an actuator-controlled mirror 648, e.g. a piezo-electric MEMS mirror. The use of the microlens array in combination with an actuator-controlled mirror allows accurate positioning (< 1 nm) and high contrast (>80%). This way, it is possible to position the activation areas relative to the position of the minima of the sinusoidal light patterns with great accuracy.

The light paths of the excitation light beam 619 and the activation light beam 645 may be combined using a first dichroic mirror 620 and may be further guided to a sample holder 624 using a second dichroic mirror 622. Fluorescent light emitted by a sample on the sample holder may travel through the second dichroic mirror to a camera 626, which may be used to record a fluorescent image of the sample.

A computer 600 connected to the system may execute a software program executing program codes for controlling the different elements of the SIM system. In particular, the computer may control the first light source and the SLM of the excitation light module, the second light source and the actuator-controlled mirror of the activation light module and the camera for recording the sample during the exposure of a sample to activation patterns and excitation patterns. In particular, the computer may execute a program for executing the steps as described with reference to Fig. 5. In an embodiment, the first and second light sources may be digitally modulated (on and off) at a frequency between 100 and 400 MHz, for example 250 MHz. In subsequent modulation cycles, different parts of the sample are exposed to an activation light pattern and a series of phase shifted excitation light patterns and images of the exposed parts of the sample are captured as described with reference to the embodiments in this application. Accurate alignment and positioning of the patterns are controlled using the MEMS mirror.

It is noted that Fig. 6 only depicts an exemplary non-limiting embodiment of a pattern-activated SIM system and many other implementations and variants are possible without departing from the inventive concept. For example, instead of an SLM configured as a grating, it is also possible to generate an excitation light pattern based on exposing two gratings (e.g. an x grating and y grating) using a Pockels cell and two polarizing beam splitters as described in detail in the article of J. Cnossen etal. at mentioned in the background of this application.

Fig. 7A and 7B schematically illustrate the gain in localization accuracy that may be achieved using the embodiments described in this application. For comparison, Fig. 7A shows a gain in localization precision of the known SIMFLUX method as discussed in more detail with reference to Fig.lA and 1B, compared to conventional Single Molecule Localisation Microscopy (SMLM). According to the SIMFLUX method, a series of phase-shifted excitation light patterns 702 I-3 may be used to illuminate a part of the sample and excite fluorescent molecules in that part. Equidistance phase shifts may be selected such that the total or cumulative excitation light 704 is substantially constant over the exposed sample region. This may lead to an improvement factor (IF) that varies between roughly 2.0 and 2.4, with an average of 2.20. If photoactivatable fluorescent molecules would be used, a homogeneous constant activation pattern would be applied, and this would not affect the improvement factor.

Fig. 7B schematically shows a gain in localization precision for structured illumination microscopy methods according to the embodiments described in this application. The improvement is compared to conventional Single Molecule Localisation Microscopy (SMLM). As show in this figure, moving the minima of the series of excitation patterns 712 I-3 closer together, i.e. decreasing the phase shift, results in areas wherein the total or cumulative excitation illumination 714 has a (local) minimum, thus yielding a non-uniform exposure to excitation light. When the minimum (minima) of the excitation light pattern coincides with the maximum (maxima) of the activation light pattern, a high improvement factor 718 may be obtained. Fluorescent molecules in this low excitation, high precision area may be activated, effectively selecting this region. The final improvement may depend on e.g. the shape of the activation pattern, the shape of the excitation patterns, the distance between excitation patterns, and the background noise. In the depicted embodiment, the improvement factor varies between roughly 2.2 and 7.8, with an average improvement factor of 3.14.

Fig. 8 schematically depicts the pattern activated structured illumination microscopy method according to an embodiment of the invention. A sample comprising a photo-activatable fluorescent molecule 802 is illuminated with activation light in a first illumination pattern 806. Only one activation area is visible in the depicted frame 804. The activation pattern has high intensity areas (light) and low intensity areas (dark). After illumination with the activation light pattern, an image 810 may be recorded, but this image does not comprise localisation information.

Subsequently, the sample is illuminated with excitation light in a series of three sinusoidal wave intensity patterns 808 I-3 with 3 phase shifted patterns with equal phase steps of 45° per orthogonal orientation of the line pattern (only one orientation is shown). The excitation pattern has high intensity areas (light) and low intensity areas (dark). For each excitation pattern, the fluorescent radiation is detected and an image is recorded, resulting in three images 812 I-3 for a first orientation of the line pattern. Each image comprises pixels, where the pixel value represents the photon count. A higher photon count is represented with a lighter colour. Even in the absence of a fluorescent molecule in a focal plane, some radiation may be detected, resulting in background noise. First localization information may be obtained by estimating the centroid of the bright spots, i.e. the regions with a photon count higher than the background noise. The estimated centroid location is indicated by a circle 814 I-3 , with the size of the circle proportional to the uncertainty in centroid location.

Second localization information may be obtained by correlating the photon count with the excitation pattern. A high photon count is associated with a high excitation light intensity, and a low photon count with a low excitation light intensity. The photon counts for the three images per orientation are known to be spaced (in this case) 45° apart. As the position of the excitation pattern is known, this provides additional a priori information (modulo the wavelength of the excitation pattern).

Third localisation information may be obtained by correlating the photon count with the activation pattern. A high cumulative photon count is associated with a high activation light intensity, and a low photon count with a low activation light intensity. As the position of the activation pattern is known, this provides additional a priori information (modulo the period of the activation pattern).

The first localization information and the second localization information and the third localization information may be combined to effectively increase the localization precision. The same procedure may be executed with a second series of excitation patterns in a different direction, e.g. orthogonal to the first series of excitation patterns, to increase the localization precision in several directions. The measurement may be repeated several times. The measurements may then be combined into a localization image 816, in which the estimated location of the fluorescent molecule for each measurement is shown with a dot 820 and a circle 818 represents the combined uncertainty. For comparison, the localisation uncertainty as may be obtained with the SIMFLUX method discussed with reference to Fig. 1 is displayed with a dashed circle 822. The Pattern Activated SIMFLUX method may result in an improvement of localization precision with a factor of about 1.4-3.4, compared to the known SIMFLUX method, and with a factor of about 3.14-7.5 compared to conventional position estimation with uniform illumination, i.e. Single Molecule Localisation Microscopy (SMLM).

Fig. 9 schematically depicts part of a structured illumination microscopy system according to an embodiment of the invention. In particular, the figure depicts a further (third) optical module that may be used for axial localization precision enhancement. This module may be used separately or may be (coherently) combined with the optical modules described with reference to Fig. 6, which are configured for lateral localization precision enhancement. This module generates a third illumination pattern, which may be an orthogonal pattern relative to the sample plane (with maximal pitch in the z direction) using two counter-propagating plane waves. When this third optical module is combined with a previously described optical module, a 3D pattern-activated (PA) SIM system can be realized, which is adapted to generate 3D orthogonal illumination and activation patterns in a sample and to use these patterns to achieve localization precision improvement in the lateral and axial directions according to the general teaching of this application.

As shown in the figure, the optical module may include a light source 902, preferably a single mode coherent source, e.g. a 640 nm laser, which may be used to generate an excitation light beam, which may be directed to a first polarizing beam splitter (PBS) 904i, which splits the excitation light beam in two coherent beams, a first excitation light beam 906i and a second excitation light beam 906 2.

The first excitation light beam may be guided towards a second polarizing beam splitter 904 2 which relays the first beam towards a first optical lens system 907i for expanding the beam to predetermined cross-sectional dimensions and further guiding the beam via a third polarizing beam splitter 904 3 toward the sample 914.

This way, a first side of the sample may be exposed with plane waves of coherent light. Similarly, the second excitation light beam may be guided towards a fourth polarizing beam splitter 904 4 which relays the second beam 906 2 towards a second optical lens system 907 2 for expanding the beam to predetermined cross-sectional dimensions and exposing the sample with plane waves from a second direction, which is the counter-direction of the first beam. This way, the beams may interfere at the position of the sample 914 forming a sinusoidal light pattern in the axial direction of the sample.

Two actuator-controlled mirrors 910 I , 2 may be used to shift the pattern in the axial direction with high accuracy (< 0.1 nm). Alternatively, electric optical modulators may be used. Further, first and second actuator-controlled mirrors 912 1 2 , e.g. MEMS mirrors, may be configured to accurately position the first light beam relative to the second light beam. The first light beam may be directed to the sample 914 using a dichroic mirror 916; the second light beam may be directed on the sample from a direction opposite from the first light beam. The first light beam and the second light beam create an interference pattern 922 in the axial extent of the sample, the dashed lines indicating the depth of focus.

In addition to the patterned excitation light pattern, small volumes (e.g. 850 nm) may be illuminated with an activation laser (405 nm) over the depth of focus. In some embodiments, the same activation pattern may be used as in 2D pattern activated structured illumination localization microscopy, while in other embodiments, a different activation pattern may be used. For example, a so-called 3D “woodpile” pattern may be used. Such a 3D woodpile patterns may e.g. be formed interfering two oblique plane waves with a counter-propagating axial plane wave. The excitation light source and the actuator-controlled mirrors may be connected to a computer which is configured to execute a 3D pattern-activated SIM scheme. In an embodiment, the activation patterns and/or excitation patterns may be three- dimensional patterns. Three dimensional activation and excitation patterns may be implemented using a single objective or multiple objectives. 3D activation and excitation patterns may also be generated using light sheets, for example as shown in B.-J. Chang et al., ‘Universal light-sheet generation with field synthesis’, Nature Methods, volume 16 (2019), pages 235-238.

Alternatively, in an embodiment, the upper beam path of the set-up shown in Fig. 9, comprising second beam 906 2 and the optics along the second beam path, up to and including actuator-controlled mirror 912i, may be replaced with a piezo-mounted mirror, reflecting the first light beam after it has traversed the sample. A low numerical aperture objective and a tube lens may be added to the beam path between the sample and the mirror to improve the optics. A similar set-up is known from J.D. Manton et al., ‘Structured illumination microscopy with extended axial resolution through mirrored illumination’, arXiv:1810.04590v1 , Fig. 2d (part below the sample). A difference with the set-up as shown by J.D. Manton et al. is that the mirror is mounted on a piezo stage to shift the pattern in the axial direction.

Fig. 10A-10C depict the localization improvement in the axial direction for a pattern-activation structured illumination microscopy scheme compared to a conventional microscopy scheme, in particular a conventional single-molecule localization microscopy scheme. The figures show a 12-time improvement when compared to conventional astigmatism. As shown in Fig. 10A, the counter propagating waves may be shifted with 3 different phase steps, to create three phase-shifted sinusoidal excitation light patterns. Fig. 10B depicts a conventional 3D localization method is using an astigmatic lens which is configured to generate a point spread function (PSF) that changes shape in the axial direction. Fig. 10B depicts the SIMFLUX axial theoretical localization precision 1002i (calculated using the Cramer Rao Lower Bound (CRLB)) and the CRLB of a 3D localization 1002 2 using an astigmatic lens. Fig. 10C depicts the improvement factor 1006 (the ratio of CRLBs) together with the activation intensity 1004 as function of axial position. Interpreting the activation intensity as the probability of activating a molecule, we can multiply the improvement factor and the activation intensity. This shows an average localization precision improvement of 12 over conventional 3D localization using an astigmatic lens.

Fig. 11 is a block diagram illustrating an exemplary data processing system that may be used in embodiments as described in this disclosure. Data processing system 1100 may include at least one processor 1102 coupled to memory elements 1104 through a system bus 1106. As such, the data processing system may store program code within memory elements 1104. Furthermore, processor 1102 may execute the program code accessed from memory elements 1104 via system bus 1106. In one aspect, data processing system may be implemented as a computer that is suitable for storing and/or executing program code. It should be appreciated, however, that data processing system 1100 may be implemented in the form of any system including a processor and memory that is capable of performing the functions described within this specification.

Memory elements 1104 may include one or more physical memory devices such as, for example, local memory 1108 and one or more bulk storage devices 1110. Local memory may refer to random access memory or other non- persistent memory device(s) generally used during actual execution of the program code. A bulk storage device may be implemented as a hard drive or other persistent data storage device. The processing system 1100 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code in order to reduce the number of times program code must be retrieved from bulk storage device 1110 during execution.

Input/output (I/O) devices depicted as input device 1112 and output device 1114 optionally can be coupled to the data processing system. Examples of input device may include, but are not limited to, for example, a keyboard, a pointing device such as a mouse, or the like. Examples of output device may include, but are not limited to, for example, a monitor or display, speakers, or the like. Input device and/or output device may be coupled to data processing system either directly or through intervening I/O controllers. A network adapter 1116 may also be coupled to data processing system to enable it to become coupled to other systems, computer systems, remote network devices, and/or remote storage devices through intervening private or public networks. The network adapter may comprise a data receiver for receiving data that is transmitted by said systems, devices and/or networks to said data and a data transmitter for transmitting data to said systems, devices and/or networks. Modems, cable modems, and Ethernet cards are examples of different types of network adapter that may be used with data processing system 1100.

As pictured in Fig. 11, memory elements 1104 may store an application 1118. It should be appreciated that data processing system 1100 may further execute an operating system (not shown) that can facilitate execution of the application. Application, being implemented in the form of executable program code, can be executed by data processing system 1100, e.g., by processor 1102. Responsive to executing application, data processing system may be configured to perform one or more operations to be described herein in further detail.

In one aspect, for example, data processing system 1100 may represent a client data processing system. In that case, application 1118 may represent a client application that, when executed, configures data processing system 1100 to perform the various functions described herein with reference to a "client". Examples of a client can include, but are not limited to, a personal computer, a portable computer, a mobile phone, or the like. In another aspect, data processing system may represent a server. For example, data processing system may represent a server, a cloud server or a system of (cloud) servers.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.