CROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to United States Provisional Patent Application

No. 62/306,615 filed on March 10, 2016, herein incorporated by reference in its entirety.

BACKGROUND

[0002] Adjusting a field of view (FOV) of a scene during an image capture process may be performed in various ways.

[0003] First, an optical zoom lens may be interposed in an optical path between the image capture device and the scene. However, the optical zoom lens cannot change a shape of the field of view as needed and is often more complicated and bulky than a fixed focal length lens.

[0004] Second, multiple images may be captured at different angles by rotating a single camera and stitching the multiple images together. However, physical rotation of the camera is usually slow because a camera can be heavy. As such, this method may not be able to capture dynamic (e.g., quickly changing) scenes or video.

[0005] Third, a single camera may be used to capture a large FOV. For example, a fisheye lens may capture photos of >180 degree FOV and some catadioptric lens designs allow one camera to capture 360 degree FOV within a single shot. However, these large FOV photos often come with much less detail than a traditional camera, because fewer pixels are included in a given angle range within the FOV. Also, these wide FOV designs often introduce strong geometrical distortion, which may degrade image quality.

[0006] Fourth, multiple cameras may be used to capture images of different angles simultaneously. The multiple images from the different cameras may then be stitched together. For example, two fisheye cameras may be arranged back to back, each covering 180 degree FOV. By stitching their photos together, one may get a 360 degree spherical photo. However, a multiple camera system is usually complicated, bulky, and expensive.

SUMMARY

[0007] In an aspect, a system is provided. The system includes a camera, a mirror optically coupled to the camera, and a motor coupled to the mirror. The system further includes a controller having a memory and at least one processor. The at least one processor is configured to execute instructions stored in the memory so as to carry operations. The operations include adjusting the motor to a first position such that the camera has a first field of view. The operations also include capturing, via the camera, an image frame of the first field of view. The operations further include repeating the adjusting and capturing steps at respective positions and respective fields of view until a plurality of image frames have been captured that include a predetermined field of view. The predetermined field of view includes an angle greater than the first field of view. The operations yet further include combining the plurality of image frames so as to form an output image frame.

[0008] In an aspect, a method is provided. The method includes adjusting a motor to a first position. The motor is coupled to a mirror and the mirror is optically coupled to a camera. The adjusting is performed such that the camera has a first field of view. The method further includes capturing, via the camera, an image frame of the first field of view. The method also includes repeating the adjusting and capturing steps at respective positions and respective fields of view until a plurality of image frames have been captured that include a predetermined field of view. The predetermined field of view includes an angle greater than the first field of view. The method yet further includes combining the plurality of image frames so as to form an output image frame.

[0009] In an aspect, a system is provided. The system includes various means for carrying out the operations of the other respective aspects described herein.

[0010] These as well as other embodiments, aspects, advantages, and alternatives will become apparent to those of ordinary skill in the art by reading the following detailed description, with reference where appropriate to the accompanying drawings. Further, it should be understood that this summary and other descriptions and figures provided herein are intended to illustrate embodiments by way of example only and, as such, that numerous variations are possible. For instance, structural elements and process steps can be rearranged, combined, distributed, eliminated, or otherwise changed, while remaining within the scope of the embodiments as claimed.

BRIEF DESCRIPTION OF THE FIGURES

[0011] Figure 1 illustrates a schematic diagram of a system, according to an example embodiment.

[0012] Figure 2 illustrates a system, according to an example embodiment.

[0013] Figure 3 illustrates a system with relay optics, according to an example embodiment.

[0014] Figure 4 illustrates a system without relay optics, according to an example embodiment.

[0015] Figure 5 illustrates a method, according to an example embodiment. [0016] Figure 6 illustrates an operational timeline, according to an example embodiment.

[0017] Figure 7 illustrates an operational timeline, according to an example embodiment.

[0018] Figure 8 illustrates an operational timeline, according to an example embodiment.

[0019] Figure 9 illustrates an operational timeline, according to an example embodiment.

[0020] Figure 10 illustrates an operational timeline, according to an example embodiment.

[0021] Figure 11 illustrates motor patterns, according to example embodiments.

DETAILED DESCRIPTION

[0022] Example methods, devices, and systems are described herein. It should be understood that the words "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as being an "example" or "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or features. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein.

[0023] Thus, the example embodiments described herein are not meant to be limiting.

Aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated herein.

[0024] Further, unless context suggests otherwise, the features illustrated in each of the figures may be used in combination with one another. Thus, the figures should be generally viewed as component aspects of one or more overall embodiments, with the understanding that not all illustrated features are necessary for each embodiment.

I. Overview

[0025] The present disclosure relates to a system and method operable to adjust camera field angles at high speed. The field angle adjusting may be synchronized with a high speed image capture process. As such, multiple frames may be captured within a short time period, avoiding the effects of object movement and/or camera shake due to hand-holding. The present system and method may capture images for use in photo stitching and/or wide FOV (e.g., 360° FOV) video. II. Example Systems

[0026] The present disclosure utilizes time multiplexing to capture photo or video of variable field of view (VFOV) using a single camera. Figure 1 illustrates a schematic diagram of a system 100, according to an example embodiment. The system 100 includes a camera and relay optic that can guide light in various directions and, at the same time, expose a lens stop. The system 100 also includes a rotational mirror that may adjust a field of view (e.g. field angle) of the camera. System 100 additionally includes a motor controller that reads the status of rotational mirror and controls the mirror movements. System 100 yet further includes a camera control and image processing system that a) reads data from the camera and controls the camera and b) reads data from the motor controller and controls the motor controller. The system 100 may include a control panel, one or more buttons, a touchpad, or another type of input device.

[0027] The system 100 may accept one or more control inputs (e.g., via a shutter button or another type of shutter release signal or mechanism) to initiate (or start and stop) an image capture process. In such a scenario, the various steps or blocks described herein may be carried out in response to the control input. That is, a user may push a shutter button once and the system 100 may automatically carry out one or more image capture sequences so as to provide one or more VFOV output image frames.

[0028] Note that in this system, the rotational mirror is the only moving part, and can be made small and light. It is much easier and takes less power to move a small object fast. A small and light rotational mirror makes it possible to incorporate this system into portable consumer devices.

A. Time multiplexing for VFOV Photo Capturing

[0029] As the mirror rotates, the field of view of the camera will scan across the whole

360 degrees. In an example embodiment, the system takes multiple images as the mirror rotates and stitches them into a 360 degree image. Figure 2 illustrates a system 200, according to an example embodiment. As shown in Figure 2, four images may be captured when the mirror rotates to the angle of 45°, 135°, 225°, and 315°, respectively. Each image has a horizontal field of view (FOV) larger than 90°. Then, by stitching the four photos together, a 360° output image may be formed. If the mirror only rotates between two angles, then we can stitch an output image of -180° FOV. By controlling how the mirror is rotated and when images are captured, the system can output images having various FOVs. For image stitching, it may be desirable to make sure there is overlap between consecutive images to avoid gaps in the output image. As such, the system and method herein may capture images based on a predetermined image overlap. Precise control of the motor rotation may minimize the required overlap and the predetermine image overlap may be determined based on a motor control error or mirror position uncertainty.

B. Time multiplexing for VFOV Video Capturing

[0030] In another embodiment, the system can capture images as the mirror rotates and the images may be converted into a 360° video (or any other smaller angle amount). Figure 2 shows a spinning mirror with four image capture positions as one example. With the spinning mirror, the system captures frames for FOV1, FOV2, FOV3, and FOV4 repeatedly; an image processing algorithm stitches every four consecutive frames into one large field of view frames, and so converts a high frame-rate, but narrow FOV video (as shown in the middle diagram of Figure 2), into a lower frame-rate but large FOV video (as shown in the last diagram of Figure 2).

C. Relay optics

[0031] One function of relay optics in the system is to expose the aperture stop of the camera. Since the aperture stop usually defines, or corresponds to, the smallest beam diameter in the optical path, we can minimize the size of the reflective rotational mirror by placing it at or near the position of the aperture stop. To achieve fast rotational speeds, minimizing the size and weight of the rotational mirror is important.

[0032] In addition, without relay optics, the rotational mirror may collide with the camera body, or the reflected FOV may be blocked, at least in part, by the camera body, especially when the camera FOV is large or the camera body is large. Figure 3 illustrates a system 300 with relay optics, according to an example embodiment. The system 300 may include an image capture device, such as a smartphone, tablet, laptop, wearable computer, or another type of camera. Namely, the relay optics may serve to guide an exit pupil to the top of the image capture device, so that when the rotational mirror spins, the camera field of view will not be blocked by the body of the image capture device. In an example embodiment, the relay optics may change a position of an exit pupil of a smartphone so that the rotation mirror is not blocked by the smartphone body.

[0033] In another embodiment, the relay optics can be modified to increase the camera field of view.

[0034] In another embodiment, the relay optics may be removed from the system.

Figure 4 illustrates a system 400 without relay optics, according to an example embodiment. In embodiments where the rotational mirror does not collide with camera body and the reflected FOV is not blocked by camera body, the relay optics may be removed, as shown in Figure 4. The lens in Figure 4 may include an external aperture stop, so that the distance between the mirror and the camera may be maximized to avoid FOV blocking.

D. Rotational Mirror

[0035] In an example embodiment, a flat rotational mirror is used so that the camera field of view will not be altered (e.g. distorted). In such a scenario, the mirror is placed close to the external stop of the optical system.

[0036] In another embodiment, a spherical mirror may be used to increase both horizontal and vertical field of view.

[0037] In another embodiment, a cylindrical mirror may be used to increase horizontal

(or vertical) field of view and keep the vertical (or horizontal) field of view unchanged (e.g. distorted along a first axis, but undistorted along a second axis).

[0038] The camera control module and the motor control module may communicate via a wired connection, or wirelessly by WiFi or BLUETOOTH. Other wireless communication protocols are contemplated.

III. Example Methods

[0039] The rotational mirror and image capture methods described herein may be performed in different ways to adapt to different camera systems, and to achieve different results. Listed below are several non-limiting modes in which this system may operate.

A. Rotate mirror and capture frame in a ping-pong sequence

[0040] Capturing an image while the mirror is still moving may cause motion blur. To minimize motion blur, the mirror movement needs to be minimized during exposure. Figure 5 illustrates a method 500, according to an example embodiment. As shown in method 500, the system synchronizes mirror rotation and camera exposure so that the mirror only moves after an image frame is captured and before the next image capture starts.

[0041] Figure 6 illustrates an operational timeline 600, according to an example embodiment. The operational timeline 600 illustrates an example sequence of motor motion and frame capture. In order to capture and stitch a 360° photo, at least one image frame is captured for every stable position of the mirror as shown in Figure 6. As described elsewhere herein, the method may be initiated or otherwise controlled by pushing a shutter button. Other user interactions are possible to initiate, stop, select from, and/or control the capture modes described herein.

B. Capture frame continuously while synchronize frame exposure and mirror movement [0042] Figure 7 illustrates an operational timeline 700, according to an example embodiment. Operational timeline 700 illustrates another scenario in which the system captures frames in a continuous mode and the mirror motion is synchronized so that one frame starts exposure when mirror stops, and completes exposure before mirror starts to move again. By comparing the frame timestamp and the motor timestamp, captured image frames without image blur may be identified. Alternatively, an image processing routine can be employed to detect and remove blurred frames. The selected frames are shown in Figure 7 as Frame 1 , Frame 2, Frame 3, and Frame 4, which can be stitched to be a 360° photo. If the camera works in a video mode, four image frames without motion blur can be extracted from every consecutive eight image frames, and these four image frames will be stitched into one 360° image frame. As a result, a 360° video may be output at a lower framerate as compared to the actual image capture rate.

C. Capture video and rotate mirror asynchronously (Global shutter image sensor)

[0043] In another embodiment, synchronization between frame capture and mirror rotation need not be required. Figure 8 illustrates an operational timeline 800, according to an example embodiment. In such a scenario, operational timeline 800 illustrates how to rotate the mirror and capture image frames asynchronously so that 360° continuous video can still be produced. In Figure 8, a blue block shows when a frame exposure starts and ends; the orange block shows when a motor starts and ends moving; and the mirror rests at position 0, 1, 2, and 3 between motor movement. The notations are:

Te denotes the exposure time of each frame;

Ts denotes the time between motor movements; and

Tt denotes the time while the motor is moving.

[0044] In the scenario illustrated in Figure 8, the image sensor may have a single global shutter for all pixels of an image sensor. As such, each pixel may start and end exposure at the same time.

[0045] As mentioned above, a frame may suffer from motion blur if the mirror is moving during image frame exposure. To stitch a complete 360 degree photo or video, at least one photo should be captured for every motor position. To capture such 360 degree photos or videos, the exposure time (Te) of each frame (captured continuously) should be less than half of the time between motor movements (Ts). That is, if Ts >= 2 ^χ Te, at least one frame will be captured with no mirror motion for every mirror position. As a result, the speed of motor movement and exposure time may be determined such that Ts >= 2 ^χ Te, and then it is guaranteed that there is at least one good frame for each mirror position. For example, in Figure 8, Frame 1 , 4, 6, and 7 are good (captured with no mirror motion) for Mirror positions 1, 2, 3, and 0, respectively. By stitching these four frames together, the system gets one 360 degree frame of a 360 degree video.

[0046] In an example embodiment, a "good" frame may be determined as follows:

For an arbitrary frame with a global shutter,

a) Read the start timestamp: tO, and compute its end timestamp as tl = tO + Te.

b) Check if mirror is moving during time from tO to tl?

1. Yes: Frame is bad.

2. Otherwise, Frame is good.

[0047] Another way to determine a good frame is by comparing the sharpness of each frame. That is a "good" frame should be sharper than an adjacent, or neighbor, "bad" frame.

D. Capture video and rotate mirror asynchronously (Rolling shutter image sensor)

[0048] Image sensors in many modem smartphones or other cameras have a rolling shutter instead of a global shutter. Figure 9 illustrates an operational timeline 900, according to an example embodiment. For an image sensor with a rolling shutter, each row of pixels starts and ends exposure at different time points, as shown in Figure 9. Accordingly, while one row is correctly captured (e.g., captured with a non-moving mirror), another row in the same frame may be corrupted by mirror motion. Figure 10 illustrates an operational timeline 1000, according to an example embodiment. As shown in Figure 10, Row 0 - 3 in Frame 1 in Figure 10 are bad (as they are captured during mirror movement); and Row 4-5 are good or captured with no mirror movement. Good rows are highlighted in Red in Figure 10.

[0049] In this system, the motor speed and the frame capture time may be configured in such a way that Ts >= 2 ^χ (Te + Tg), where Tg is the time after a given row ends exposure and before the given row starts exposure for next frame. Tg may include, for example, a pixel readout time and a pixel reset time. In such a scenario, it is guaranteed that for each sensor row and each mirror position, there is at least one good row captured. By stitching all rows captured with no mirror movement together, we get a complete image frame for each static mirror position. Then, by stitching all correctly captured frames together for a complete cycle of all mirror positions, we get a correctly captured 360 degree frame. As such, a 360 degree image frame and/or video may be captured.

[0050] One way to determine if one row is correctly captured or not is as follows:

For each frame

a) Read the timestamp of Row 0: tO.

b) For each Row i

i) Compute the start timestamp Ai = tO + i ^χ s, where s is time difference between consecutive rows in rolling shutter.

ii) Compute the end timestamp Bi = Ai + Te, where Te is exposure time.

iii) Is mirror moving during time from Ai to Bi?

1) Yes: Row i is captured during mirror movement ("bad").

2) Otherwise, Row i is captured without mirror movement ("good").

[0051] In this example, each frame covers a horizontal >90 degree field of view, and each mirror position is 90 degrees apart from adjacent mirror positions. By stitching four consecutive frames, we get a 360 degree FOV output image frame. As an example embodiment, a process to produce a 360 degree video out of the proposed system is as follows:

1. Set q = -1

2. Set k = 0

3. Set i = 0

4. While i < total frame number

a) Find all good rows in this frame, copy them to Image H;

b) Let p = mirror position, ranging from 0 to 3.

c) If p is not equal to q

i) if q == 3

1) Stitch F_0, F_l, F_2, F_3 to Image Gk

2) k = k+l

ii) Copy H to F_p

d) i = i + l

5. Encode all images Gk to provide a video.

E. Capture video of various frame rates at different field angles.

[0052] Figure 11 illustrates motor patterns 1100, according to example embodiments.

In the example of Figure 10, the mirror rotates in a pattem shown in Figure 11 as Motor Pattern 1. As a result, if the raw video is captured at 240 frames per second (fps), the final 360 degree video will be 30 fps because one 360 degree image frame is output for every 8 raw frames. The frame rate is uniform for every field angle.

[0053] In an example embodiment, the motor motion and camera image capture may be coordinated so as to capture different frame rates for different field angles. In one embodiment, we can rotate the mirror with a pattern shown in Figure 11 as "Motor Pattern 2". With this pattem, we capture more frames for FOVl , and fewer frames for other field of views. For every consecutive 12 raw frames, we produce 3 good frames for FOVl, and 1 good frame for each of FOV2, FOV3 and FOV4. As a result, if we capture raw frames at 240 fps, we get 60 fps for FOV1 (0° - 90°), and 20 fps for FOV2, FOV3, and FOV4 (90° - 360°).

[0054] In another embodiment, the mirror may remain at FOV1 for 6 ^χ Te, so that we can read 5 good frames for FOV1 ; and then move mirror to FOV2 and stay for 2 ^χ Te, so that we we can read 1 good frame for FOV2; and then continue the process as shown in Figure 11 : Motor Partem 3. With this configuration, we are able to produce 5 good frames for FOV1, 1 good frames for each of FOV2, FOV3, and FOV4, from 12 raw frames. As a result, if we capture raw frames at 240 fps, then we get 100 = 240 * 5 / 12 fps for FOV1 (0° - 90°), and 20 fps for FOV2, FOV3, and FOV4 (90° - 360°).

[0055] By configuring mirror rotation patterns differently, we can achieve various frame rates for different field angles.

[0056] Note that although most examples above assume 4 mirror positions per cycle and the sum of field of view covers the whole 360 degree of field of view, the system may be configured to have more or less mirror positions per cycle. Furthermore, the system may include different rotation angles and cover different fields of view per cycle based on, for example, specific application needs.

F. Move the rotational mirror according to image analysis or user input

[0057] In an example embodiment, the system rotates the reflective mirror at a uniform speed to scan a scene of large FOV (e.g., capture video of large FOV). For each image frame, an event detection may be performed. For example, motion detection can be done by measuring the difference among consecutive image frames; or face detection be done on each image frame by comparing each image frame to an image database. An event detection may include one or more objects. Once an event is detected in one field angle, the system may track this field angle and/or the object(s) at a higher frame rate, and capture other field angles at a lower frame rate, until the event ends or is no longer detectable, or until a new event is detected.

[0058] An example method is as follows:

1. Capture video of large FOV as the abovementioned Mode C or Mode D.

2. For each image frame:

a) run event detections (e.g., motion detection, face detection, obj ect detection) b) adjust the framerate at each field angle according to the detection result, so that field angles with detected events are assigned a higher frame rate than other field angles without detected events.

c) adjust mirror motion to achieve the various framerate capturing as described in Mode E above. d) Loop to Step 2

[0059] In another embodiment, a user may interact with the system. For example, the user may input an desired angle range and/or region of interest:

1. Capture video of large FOV as the abovementioned Mode C or Mode D.

2. For each image frame:

a) display the image frame.

b) a user may indicate one or more regions of interest via certain input devices (e.g., touch screen, mouse, gesture, etc).

c) adjust the framerate at each field angle according to the user indication, so that user-indicated field angles are assigned a higher frame rate than other field angles that are not user-indicated.

d) adjust mirror motion to achieve the various framerate capturing as described in Mode E above.

e) Loop to Step 2

[0060] In an example embodiment, the system need not capture wide FOV video.

Instead, the system may capture a video with a normal FOV. The video may be displayed to users, who may adjust a field angle according to one or more indications.

[0061] The particular arrangements shown in the Figures should not be viewed as limiting. It should be understood that other embodiments may include more or less of each element shown in a given Figure. Further, some of the illustrated elements may be combined or omitted. Yet further, an illustrative embodiment may include elements that are not illustrated in the Figures.

[0062] A step or block that represents a processing of information can correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a step or block that represents a processing of information can correspond to a module, a segment, or a portion of program code (including related data). The program code can include one or more instructions executable by a processor for implementing specific logical functions or actions in the method or technique. The program code and/or related data can be stored on any type of computer readable medium such as a storage device including a disk, hard drive, or other storage medium.

[0063] The computer readable medium can also include non-transitory computer readable media such as computer-readable media that store data for short periods of time like register memory, processor cache, and random access memory (RAM). The computer readable media can also include non-transitory computer readable media that store program code and/or data for longer periods of time. Thus, the computer readable media may include secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, compact-disc read only memory (CD-ROM), for example. The computer readable media can also be any other volatile or non-volatile storage systems. A computer readable medium can be considered a computer readable storage medium, for example, or a tangible storage device.

[0064] While various examples and embodiments have been disclosed, other examples and embodiments will be apparent to those skilled in the art. The various disclosed examples and embodiments are for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the following claims.