Field of the Inventions

[0001] The inventions described below relate to the field of minimally invasive surgery.

Background of the Inventions

[0002] Retractors with cameras mounted on components of the retractors have been used for various surgeries, including spine surgery and abdominal surgery. Tesar, U.S. Patent 9,492,065 (November 15, 2016) discloses a retractor for abdominal surgery, with cameras mounted on the circular frame, or blades of the retractor. Chhit, U.S. Pub. 20200261071 (Aug. 20, 2020) discloses a retractor for spinal surgery, with camera assemblies mounted on a retractor frame or a retractor blade. In each of these systems, the entire camera assembly may be rotated relative to the retractor system to point the camera to different areas of a surgical field at the distal end of the retractor blades. In our U.S. Patent 11/723,641 (U.S. Application 17/444,328) , entitled Camera System for Use with Retractors, we disclose a camera assembly configured for mounting on the proximal end of a retractor system in which the camera assembly includes a rotatable reflector mount disposed within the camera assembly and configured to rotate a reflecting element to direct images from the surgical working channel to the image sensor. The reflector mount and reflector may be rotated to point the viewing axis of the system toward different points in the surgical field at the distal end of the retractor. These systems can be enhanced with the provision of 3D imaging with images provided in a 2D display screen, a 3D display or a 3D head mounted display. Summary

[0003 ] The devices and methods described below provide for improved visualization of a surgical work space held open with a retractor and ef fective 3D display of images obtained from a camera assembly conf igured for mounting on the proximal end of a retractor system, where the camera assembly includes a distal -most optical surface which, in the preferred embodiment , is disposed at the proximal end of the cannula system such that the entirety of the camera assembly is disposed at the proximal end of the retractor system . The distal -most optical element in the camera assembly may be disposed at the proximal end of the retractor system, and may be disposed proximal to the proximal end of the retracting element (blade or cannula tube) , and may be disposed such that the distal -most optical element overhangs the working channel established by the retractor . The camera assembly includes a ref lector mounted on a rotatable ref lector mount , and mechanisms to rapidly move the ref lector back and forth between at least two dif ferent positions to obtain at least two dif ferent views of the surgical space , and a control system operable to operate the camera system and a display screen to obtain alternating images from the dif ferent view points , process the images and present the images from the dif ferent viewpoints alternatingly on a display screen .

[0004] The alternating nature of the display can be accomplished at low frame rates (as low as one alternation per second or lower) or a high frame rate , but in any case the images from the two viewpoints should be alternated at a rate suf f icient to provide useful 3D images from which a surgeon operating through the retractor can discern the relative depth of anatomical structures in the surgical site . The presentation of a pair of stereoscopic images in alternating succession on a two dimensional display screen to provide a stuttering image conveying depth perception is referred to as a wigglegram . The presentation of a pair of stereoscopic images in alternating succession on individual screens of a pair of shutter glasses may also be used with the systems described below to provide 3D images conveying depth perception .

[0005] "Stereoscopic image pairs" refer to two photographs , or two sets of video images , of the same obj ect taken at slightly dif ferent angles are viewed together , creating an impression of depth and solidity . In the systems described below that use a 2D display and shutter glasses , video images may be viewed "together" in the sense that one set of images from a f irst viewpoint are transmitted to a display screen visible to a user followed by a second set of images from a second viewpoint transmitted to the display placed, while the user wears shutter glasses which are transparent to the right eye but opaque to the left eye while the right -eye image is displayed, and vice-versa , at any rate that produces acceptably smooth images (which need not be cinema or broadcast quality) . In systems describe below which use a 2D display and wigglegrams , "together" may mean one viewpoint image displayed immediately after the second viewpoint image , perhaps several times in succession (a single wigglegram stereoscopic pair , displayed in alternation) . I f implemented in a heads mounted display, "together" may mean that one set of images from a f irst viewpoint are transmitted to a display screen visible to one eye of the user while images from a second viewpoint are transmitted to a display screen visible to the other eye of the user . [0006] The f irst mode presented below is based on wigglegrams . Wigglegrams are typically created using a camera assembly comprising two or more distinct imaging assemblies which in turn comprise at least a lens and imaging sensor . The systems described below provide wigglegram stereoscopic image pairs using a single imaging assembly .

Brief Description of the Drawings

[0007 ] Figure 1 is a coronal cross section of a patient illustrating a typical placement of a retractor system and camera assembly in which the rotatable mirror may be used to direct the viewing angle of the camera assembly to dif ferent areas of the surgical workspace at the distal end of the retractor blades .

[0008] Figures 2 and 3 illustrate camera assemblies conf igured for adj ustment of the viewing axis of the camera assembly .

[0009] Figures 4 and 5 illustrate swiveling operation of the camera assembly useful for obtaining stereographic image pairs of the surgical workspace .

[0010] Figures 6 and 7 illustrate trucking operation of the camera assembly useful for obtaining stereographic image pairs of the surgical workspace .

[0011] Figure 8 and 9 illustrate tilting operation of the camera assembly useful for obtaining stereographic image pairs of the surgical workspace .

Detailed Description of the Inventions

[0012 ] Figure 1 illustrates one environment of use of the retractor system and camera assembly, and shows a coronal cross section of a patient 1 , taken through the midsection at the level of the lower back and lumbar spine , illustrating a typical placement of a retractor system and camera assembly in which the rotatable mirror may be used to direct the viewing angle of the camera assembly to dif ferent areas of the surgical workspace at the distal end of the retractor blades . The anatomy shown in Figure 1 includes the vertebra 2 , an intervertebral disc 3 , lamina 4 and spinous process 5 , and the articular j oint 6 to an adj acent vertebra , each of which might need to be addressed surgically to treat a variety of conditions .

[0013 ] To gain access to any of these structures , a surgeon may approach the spine through a retractor system 11 placed in one of several pathways . For a laminotomy or laminectomy, for example , the retractor will be placed posteriorly to the spine , and inserted into an incision in the back . I f the retractor is an expandable multi -blade retractor , the surgeon will separate the blades to create a large working channel . I f the retractor is a cannula retractor or trocar , the lumen of the cannula will serve as the working channel . The working channel also serves as a viewing channel , such that the surgeon can view the surgical workspace through the same path used to insert tools into the surgical space .

[0014] In addition to bone structures of the spine , the surgical f ield may include arteries , veins , nerves and ligaments which may need to be avoided, manipulated, retracted, cauterized, etc . A 3D image that conveys information regarding the relative depth of these structures can facilitate proper access to or avoidance of each of these structures . [0015] To provide viewing of the surgical workspace , the camera assembly 12 may be f ixed to the proximal end of the retractor . The camera assembly may be f ixed to a retractor blade of an expandable retractor or to the proximal end of a tube of a cannula retractor , or to the frame of a multi -blade retractor , or to the proximal end of a single blade retractor , or other retracting element .

[0016] Figures 2 and 3 illustrate a camera assembly conf igured for adj ustment of the viewing axis of the camera assembly by rotating a ref lector without rotation or translation of the entire camera assembly, and without movement of the imaging sensor within the camera assembly . As shown in Figure 2 , the camera assembly 12 comprises a housing 13 with an imaging sensor 14 disposed at a f irst end of the housing , a lens assembly 15 , a rotatable ref lector mount 16 and a ref lector 17 mounted on the ref lector mount at the second end of the housing . The rotatable ref lector mount is disposed between the imaging sensor and the ref lector , and is rotatably disposed within the housing such that it can tilt the mirror in planes intersecting the central viewing axis 18 of the imaging sensor ( typically perpendicular to the f lat face of the imaging sensor) . The ref lector mount , when comprising an element disposed between the ref lector and imaging sensor , is optically transmissive , such that light beams from the surgical workspace and ref lected by the mirror will pass through the ref lector mount to the lens 15 and imaging sensor 14 . The ref lector establishes a central viewing axis 19 of the camera assembly, and the rotatable ref lector mount provides a means for rotating the mirror and thereby altering the viewing axis relative to the imaging axis of the sensor to provide views of the surgical workspace at the distal end of the retractor blades or retractor tube , without need to rotate the entire cameral assembly . The ref lector may be the distal -most optical component , but may be covered with a lens , protective transparent plate or f ilter or other optical component which serves as the distal -most optical component . Such a lens , protective transparent plate or f ilter or other optical component which serves as the distal -most optical component may be disposes at the obj ective aperture of the bore of the rotatable ref lector mount or an optical opening of the camera assembly . The distal -most optical component (or surface thereof ) preferably is disposed, when the camera assembly is f ixed to a retractor , at the proximal end of the working channel established by the retractor blades , or proximal to , and overhanging without occluding the working channel established by the retractor blades .

[0017 ] The rotatable ref lector mount 16 of Figures 2 and 3 may be provided in the form shown, with a generally spherical body 20 (a ball head, for example) , or a body comprising a portion of a sphere , with a bore 21 with a f irst segment 21A passing through the body along a line 22W between the workspace and the ref lector and a second segment 21B passing through the body along the line 22R of light ref lected by the ref lector toward the image sensor ( the ref lector ref lection axis ) . The bore segments establish optical apertures in the spherical body 20 , including an aperture closest to the surgical workspace (an obj ective aperture) and an aperture closest in the optical pathway to the image sensor (an ocular aperture) . The bores may be open, empty bores , or they may be f illed with optically transmissive or transparent material ( if so , the surface of any transmissive material in the obj ective aperture may constitute a distal -most surface of the camera assembly) . The spherical body may be truncated, as shown, to provide a flat surface at the objective aperture, and truncated to establish a flat surface along a cord set at an angle to the reflector reflection axis. In this geometry, the truncated flat surface along the cord corresponds to the flat plane of the reflector, and is set at a 45° angle to both the axis of the bore first segment 21A and the second bore segment 21B when the spherical body 20 is in a neutral position. This angle may be varied to accommodate retractors of various configuration, to avoid interference with other structures on the retractor. The reflector, or the aperture closest to the surgical workspace (an objective aperture) in the reflector mount/spherical body, or an optical element disposed with the aperture may be the distal-most optical element, or the optical element closest to the objective, and preferably is disposed proximal to the proximal end of the retractor elements, or the working channel defined by the retractor elements .

[0018] Figures 4 and 5 illustrate an operation of the camera system to obtain varying images of the same surgical workspace in alternating fashion to provide stereoscopic image pairs which can be processed into wigglegrams providing depth of field information on a 2D display screen. In Figures 4 and 5, the reflector mount 16 and reflector 17 are rolled or swiveled about an axis A parallel to the long axis L of the retractor system 11 (which may be the center of the ball, or an edge nearer the image sensor) . (We define this movement of the reflector and reflector mount as swiveling or yawing if accomplished about any axis parallel to the viewing axis (the center of the reflector mount, or closer to the image sensor) , and "rolling" if limited to rotation about the imaging axis 18 ) . Preferably, this motion is a swiveling motion about the center of the reflector mount, which results in small " lateral" movement of the viewing axis . The f irst position of the ref lector mount and ref lector shown in Figure 4 places the viewing axis in a f irst location a short distance from the second position of the ref lector mount and ref lector shown in Figure 5 , in which the second viewing axis position achieved when swiveled as shown in Figure 5 is displaced from the f irst position .

[0019] In Figures 6 and 7 , the camera assembly is provided in conf iguration a which uses an imaging sensor 14 , lens assembly 15 and f ixed ref lector (preferable a prism) 23 mounted on the proximal end of the retractor , with the prism comprising the distal -most optical surface of the camera assembly . In this embodiment , the camera assembly components are translated across the opening of the retractor system (note that the entire camera assembly may be translated inwardly and outwardly) . In Figure 6 , the camera assembly including the prism is conf igured and positioned to point the viewing axis 19 of the camera assembly to a f irst point in the surgical workspace nearer the retractor wall , while in Figure 7 the camera assembly including the prism are positioned to point the viewing axis 19 of the camera assembly to a second point further away from the wall of the retractor blade or tube . The camera assembly and prism are operable to be translated inwardly toward the center to the surgical channel , and back outwardly toward the edge of the surgical channel . (We def ine this motion as trucking or pedestailing , analogous to similar movements of f ilm cameras relative to a f ilm set . )

[0020] Figures 8 and 9 illustrate operation of the camera system to obtain varying images of the same surgical workspace in alternating fashion to provide stereoscopic image pairs which can be processed into wigglegrams providing depth of f ield information on a 2D display screen . In Figure 8 , the ref lector mount 16 and ref lector 17 are positioned to point the viewing axis 19 of the camera assembly to a f irst point in the surgical workspace , while in Figure 9 the ref lector mount 16 and ref lector 17 are positioned to point the viewing axis 19 of the camera assembly to a second point in the surgical workspace . The ref lector mount 16 and ref lector 17 have been rotated about a transverse axis T (orthogonal to the long axis L and the imaging axis 18 ) (We can def ine this tilting toward and away from the retract wall nearest the camera mount , and rotation about axis T as tilting or pitching . )

[0021] Thus , in Figures 2 through 9 , we present various conf igurations of a camera and retractor system operable to obtain stereographic image pairs useful to create wigglegrams to be used in various visualization methods described below .

[0022 ] The camera assembly may be operated in a normal , standard video camera mode , obtaining video images in standard format at standard refresh rates or frame rates , with the viewing axis in a position which may be deemed the f irst position . The camera may be operated through a control system 24 , to obtain images from the camera and process images and operate the display 25 to display images 26 on a screen . When 3D information is desired, an operator may operate the control system which is operable to cause the ref lector mount to move the viewing axis to the second position and obtain at least one video frame from the second position . The camera assembly can be manipulated so that the viewing axis is returned to the f irst position or the viewing axis may be maintained in the second position . The camera assembly may be manipulated by hand ( for example , with a lever operable connected to the ref lector mount and external to the camera housing) from the f irst position to the second position and back again, or may be moved back and forth with a motor or solenoid actuator (or other means for rotating or translating) operably connected to the ref lector mount , or other means for rotating the ref lector mount relative to one of its several axes . With the camera viewing axis pointed in the f irst direction, toward the f irst point , the camera is operated, through a control system, to obtain an image ( still image or short video clip comprising a few frames of video) . The dif ferent obtained images may be compared and key- stoned and trimmed for transmission to the display screen to display stereoscopic image pairs of the same area of the surgical workspace that are at least roughly aligned on the display screen .

[0023 ] The associated control system is operable to obtain both ( 1 ) normal 2D video images from the camera while it is maintained in the viewing axis single direction, and control the display screen to display 2D corresponding video images on the 2D display screen, so that the system can be used as usual to visualize the surgical site while passing the distal end of surgical tools through the cannula and ( 2 ) stereoscopic pairs of images from the camera , with one image of the pair obtained with the camera viewing axis in a f irst position and the other image obtained with the camera viewing axis in a second position, and control the display to display a wigglegram of comprising corresponding stereoscopic image pairs on the 2D display screen, so that the system can be used to visualize the surgical site and discern the relative depths of structures within the surgical site .

[0024] The control system may be operated to provide occasional alternating stereographic image pairs , while otherwise presenting normal video (NTSC or HD or other standard) on the 2D display, in a portion of the display designated for video display, provided on the 2D display in a manner that replaces or interrupts the normal video , or provided in an inset (picture- in-picture) format . The control system may operate the camera assembly to obtain and display the occasional alternating stereographic image pairs upon demand, as indicated by input provided to the control system . Thus , a surgeon may operate the system, and work through the retractor , generally under the guidance of standard video images displayed on the display screen, and provide input to the control system when a 3D information is desired to cause the control system to obtain pair ( s ) of stereographic images and present the pair ( s ) of stereographic images as a wigglegram on the 2D display screen . Once the surgeon has seen the wigglegram and discerned any relative depth information conveyed by the wigglegram, the surgeon can provide input to the control system to cease display of the pair ( s ) of stereographic images .

[0025] The system may be operated to obtain and display a continuing series of pairs of stereoscopic images , to create a wigglegram video , if desired, for example , to guide tool tip toward a structure in the surgical workspace . To generate a video comprising multiple wigglegrams , the control system may generate a f irst pair , display the f irst pair in alternating fashion (at least one exchange to convey depth information) , then generate a second pair , display the second pair in alternating fashion (at least one exchange to convey depth information) , etc . , generating and displaying many sequential pairs in sequence . The result will be a j ittery video that nonetheless provides immediate visualization of movement of tool tips through the surgical space and anatomical structures in the surgical space along with conveyance of depth information to the surgeon .

[0026] A method of visualizing a surgical workspace of a patient from the proximal end of a retractor using the system described above can include the steps of installing the retractor in a patient to provide access to a surgical site rotating the rotatable ref lector mount ( 1 ) about the imaging axis to move the viewing axis between the f irst position and the second position, ( 2 ) about an axis transvers to the imaging axis to move the viewing axis between the f irst position and the second position and/or ( 3 ) about an axis parallel to a long axis of the retractor system to move the viewing axis between the f irst position and the second position ; obtaining an image of the surgical site from the f irst position and obtaining an image of the surgical site from the second position ; and operating the control system to create and display a wigglegram comprising the image of the surgical site from the f irst position and obtaining the image of the surgical site from the second position .

[0027 ] The embodiments described above are conf igured to provide visualization conveying depth information for the surgical site using a typical operating room set -up in which a surgeon operates on a surgical site through a bladed or cannula retractor while looking at the surgical site directly through the cannula and viewing an enlarged video image of the surgical site on a display screen . The system accomplishes this goal with standard display screens without the need for complex head gear , head tracking or eye tracking . However , the operation of the camera assembly to obtain f irst and second images from f irst and second viewpoints can be combined with ( 1 ) shutter glasses and a separate display screen or ( 2 ) a head-mounted display . When used with shutter glasses , the shutters may be opened (made transparent ) and closed (made opaque) in alternating fashion, while the display screen is controlled to display f irst and second images in alternating fashion synchronized with the opening and closing of the shutter glasses to present 3D images to a user wearing the shutter glasses , without regard to the generation of a wigglegram . This may be accomplished at a high frame rate , such as video frame rate or refresh rate of 30 , 60 , 120 Hz ( frame per second) or other standard rate , rather that the very low frame rate of wigglegrams . When accomplished with a head mounted display, the wigglegram may be presented as described above in relation to larger 2D displays , or the stereoscopic pairs may be obtained and presented at a high frame rate (higher than the several frames per second which suf f ice for a wigglegram) to provide true 3D imaging with images from the f irst position of the viewing axis displayed in the line of sight of one eye and images from the second position of the viewing axis displayed in the line of sight of the second eye .

[0028] While the preferred embodiments of the devices and methods have been described in reference to the environment in which they were developed, they are merely illustrative of the principles of the inventions . The elements of the various embodiments may be incorporated into each of the other species to obtain the benef its of those elements in combination with such other species , and the various benef icial features may be employed in embodiments alone or in combination with each other . Other embodiments and conf igurations may be devised without departing from the spirit of the inventions and the scope of the appended claims .