RECOGNITION

BACKGROUND

The present invention relates to television delivery systems. More particularly, and not by way of limitation, the present invention is directed to a system and method for searching a Video On Demand (VOD) catalog utilizing image recognition, for example by recognizing an image on a poster or box cover,

VOD systems allow users to purchase and consume multimedia content within their own Multiple System Operator (MSO)/cabie system. The user accesses a catalog of available VOD content and selects a movie or other content for viewing. The selected content is played out through a Set Top Bo (STB) and television (TV) within the user's premises.

In order to search for a movie title within a VOD catalog, the user currently has to enter information into a search form. Alternatively, some systems now enable the user to scan a barcode or other glyph that can be interpreted by a software system to retrieve embedded information such as Universal Product Code (UPC) information about the title.

Using barcodes, however, is limiting in mat the manufacturer must include a barcode on the packaging in order for the title information or ^' UPC code to be retrieved. Movie advertisements such as posters generally do not include this sort of information. When they do, the information is related to the UPC" code for the poster, not for the movie it advertises.

SUMMARY

The present invention enables a user utilizing a digital camera to search a VOD catalog by taking a photograph of an image such as a movie advertisement poster, a cover of a book, a cover of a DVD case, and the like. Using a heuristic system, the invention interprets the image and determines a movie title related to the image. The determined title may optionally be presented to the user for confirmation before searching the VOD catalog to determine whether the determined title is available. In one embodiment, the present invention is directed to a method in a VOD system for searching a catalog of VOD content utilizing automated image recognition. The method inc hides the steps of receiving at an image processing and lookup system, an image from a user's imaging device; comparing by the image processin and lookup system, the received image with a plurality of stored reference images associated with VOD content in the VOD catalog; and determining whether the received image matches a stored reference image associated with a piece of VOD content in the VOD catalog. When the received image matches an image associated with a piece of VOD content, in the VOD catalog, the image processing and lookup system sends to the user, an indication that the matching piece of VOD content is available in the VOD catalog. Whe the received image does not match an image associated with a piece of VOD content: in the VOD catalog, the image processing and lookup system sends to the user, an indication that the matching piece of VOD content is not available in the VOD catalog.

In another embodiment, the present invention is directed to an image processing and lookup system (IPLS) for searching a catalog of VOD content utilizing automated image recognition, The IPLS includes a communication interface configured to receive an image from a user's imaging device; and an image recognition processor configured to compare the received user image with a plurality of stored reference images associated with VOD content in the VOD catalog to determine whether the received user image matches a stored reference image associated with a piece of VOD content in the VOD catalog. When the received image matches an image associated with a piece of VOD content in the VOD catalog, a control processor causes the communication interface to send from the IPLS to the user, an indication that the matching piece of VOD content is available in the VOD catalog. When the received image does not match an image associated with a piece of VOD content i the VOD catalog, the control processor causes the communication interface to send from the IPLS to the user, an indication that the matching piece of VOD content is not available in the VOD catalog.

In another embodiment, the invention is directed to a VOD system that includes a VOD catalog that lists VOD content available for a user to purchase; a database for storing a plurality of reference images associated with the VOD content in the VOD catalog; and an image processing and lookup system (IPLS) for searching the plurality of reference images in the database utilizing automated, image recognition. The IPLS includes a communication interface configured to receive an imase from a user's imaging device, and an image recognition processor configured to compare the received user image with a plurality of stored reference images associated with VOD content in the V D catalog to determine whether the received user image matches a stored reference image associated with a piece of VOD content in the VOD catalog. When the received image matches an image associated with a piece of VOD content in the VOD catalog, a control processor causes the communication interface to send from the IPLS to the user, an indication that the matching piece of VOD content is available in the VOD catalog. Whe the received image does not match an image associated with a piece of VOD content: in the VOD catalog, the control processor causes the coffimunication interface to send from the IPLS to the user, an indication that the matching piece of VOD content is not available in the VOD catalog.

Using this system, users can quickly find title information and determine availability through their VOD system without, the need for a barcode or other glyph. For example, user may simply point a camer at movie poster and schedule an alert notification for when the advertised movie is available in the VOD catalog.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following section, the invention will be described with reference to exemplary embodiments illustrated in the figures, in which:

FIGS. 1A-1 B are portions of a simplified block diagram of an exemplary embodiment of a VOD system in which the present invention has been implemented;

FIG. 2 is a simplified block diagram of an Image Processing and Lookup System (IPLS) in an exemplary embodiment of the present invention; and

FIG. 3 is a flow chart illustrating the steps of an exemplary embodiment of the method of the present invention.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components and circuits have not been described in detail so as not to obscure the present invention. Additionally, it should be understood that the invention may be implemented in hardware or in a combination of hardware and software. For example, one or more computers or processors may perform the steps of the method of the present invention when executing compute program instructions stored in one or more non-transitory program memories.

In one embodiment of the present invention, an unknown image of a movie poster or DVD cover is compared to a plurality of known reference movie posters/DVD box cover images. Heuristics are used to index the reference images. The algorithms for determining a match are described below. The indexing information is stored in a database for reference images. When a comparison to an unknown image is requested, a reference image is uploaded to a server component, which generates comparison data on the uploaded reference image. The server component analyzes the unknown image by comparing certain traits against a known set of images that relate to movie titles. Upon a successful comparison, ail metadata relating to the movie title are conveyed back to the user to be used.

FIGS. ΙΑ-iB are portions of a simplified block diagram of an exemplary embodiment of a VOD system 10 in which the present invention has been implemented. The system includes a user's digital imaging device 1 1 such as a digital camera, cell phone with a camera, smartphone wit a camera, or other digital imaging device. After taking an image of a movie poster or DVD box cover 12, the camer or phone uploads the digital image through an internet Protocol (IP) network 13 to a multi-screen TV system 14. The multi -screen TV system interfaces with an Image Processing and Lookup System (IPLS) 15, and forwards the poster or box cover image to the IPLS. The IPLS has access to a database 1.6 of movie titles and associated poster and/or bo cover images, which it compares to the digital image received from the user.

Tile VOD system JO also includes an Operator Back Office 17, which performs conventional management functions, and also includes a Poster Server i 8 that populates the database 16 with poster and box cover images and associated movie information from VOD content available in the VOD catalog. For completeness, FIGS. 1 A-1 B also show components enabling the user to purchase and view VOD content from the VOD system such as a VOD Server ant! Content Store 1 , a Hybrid Fiber- Coaxial (HFC) access network and Headend (HE) 20, a Set Top Bo (STB) 21, and a television (TV) 22.

FIG. 2 is a simplified block diagram of the IPLS 15 in an exemplary embodiment of the present invention. The users' poster or box cover image is received through an IP communication interface 31 , which passes the image to an image recognition processor 32. in one embodiment, the image recognition processor interfaces with a commercially available image recognition program 33 to compare attributes of the user's poster or box cover image to attributes of images stored either in the database 16 or in a remote database (not shown). The commercially available image recognition program may be executed by the image recognition processor 32, or may be executed remotely as a "cloud service". Suitable commercially available image recognition programs include., for example, the Kooab application programming interface (API) available from the Kooaba Company (www.kooaba.com) or the TiiiEye, Fiximilar, or PixMatch APIs from Idee, Inc. (www.ideeijic.com). The Kooaba API, for example, may reside remotely (i.e., a cloud service) with the IPLS accessing it through Representationai State Transier ( REST), a Hypertext Transfer Protocol (HTTP) style of software architecture. An available "Query API" may be utilized to make requests to ooaba's existing database of millions of images. A "Data API" may be utilized to upload a user's own data for recognition.

When a match is found between the user's poster or box cover image and an image stored in the database, the commercially available image recognition program 33 returns a matching title, which is forwarded to a control processor 34. The control processor may control the operation of the IPLS by executing computer program instructions stored on a non-transitory memory device 35. When the control processor recei ves the matching title., and the commercially available image recognition program used its own database of reference images, the control processor queries the VOD catalog 36 to determine whether the title is currently available in the VOD catalog. The VOD catalog returns an indication of the title availability, which the control processor forwards to the IP communication interface 1 . When the control processor receives the matching title, and the commercially available image recognition program used the title/image database 16 (which is populated with titles and images from the VOD catalog), the control processor does not have to query the VOD catalog, instead, the control processor can immediately forward an indication of the title availability to the ΪΡ communication interface. The IPLS 15 then sends an indication to the user via the multi-screen TV system 14 and the IP network 13 (FIG. 1) indicating that the associated movie is available in the VOD catalog. The IPLS may also send information about, the associated movie title to the user.

In another embodiment, the IPLS 15 performs a three-tiered approach for image recognition as described below. This runctionaHty may reside, for example, in the image recognition processor 32. it should e noted that the first algorithm described below will work for color images; however, for black and white images the first algorithm will not he reliable.

By way of background, if an array of pixels (for example 512x512) is analyzed, and each pixel provides a sample of luminance, the array can be represented as an array of 512x512 values, where each sample is transformed representing a two-dimensional spatial component This was classically done using Discrete Cosine Transform (OCT) for MPEG-2, and then an integer transform for H.264, Other methodologies are also possible. If the values are arranged as a grid with the top left corner representing the lowest spatial frequency, values to the right represent higher horizontal spatial frequencies. An index value (u) may be utilized in the horizontal dimension. Likewise in the vertical direction, an increasing value of an index (v) represents increasing vertical spatial frequency. The (0,0) locatio is a special case and is a DC value (i.e., it is merel the mean of all the samples). Moving to row 1 or column 1 represents the fundamental frequency of the sample array ; row 2 or column 2 is the 2nd harmonic, and so on.

The lower harmonics are more useful from a recognition point of view than the higher ones, so a subset of the lower harmonies may he defined, with the value 'G' being the upper limit of spatial frequencies of interest. It should be noted that spatial frequencies are independent of resolution; i .e., if a spatial transform is done of the same picture with 1 24x1 24 pixels sampled, then the (1 , 1 ) spatial frequency component will be exactly equivalent to the (1,1) spatial frequency component of the 512x512 version, and so on. The only difference would be that there are more transformed values in the grid ~ higher maximum spatial frequency. There are no entries above row or column 512- 1 , because this has no representation in the original data. Hence u,v are limited to range 0... 51:2- 1.

The three-tiered approach for image recognition is as follows:

I) Example index Matching Criteria for image Comparison

If the image is Red, Green, Blue (R..G-8) sampled, then transform from RGB t YPrPb using the transform:

Y - 0.299R + 0.587G + 0. Π4Β

Pr - R-Y

Pb = B-Y

where Y is the luminance and Pr and Pb are chrominance (color difference) values.

Hue e is the angle defined by tan ^" ' (Pr/Pb). The calculation needs to make sore that the correct quadrant is returned and that jPb] and jPrj are compared io select whether to calculate ø or (π/2 - o) to avoid numeric overflows.

Saturation ^:::: (Pb ² + Pr' ) ^1'2

2) Transform Coefficient Ratios

If the sampled image is in a rectangle of dimensions N by M, then the DCT transform is defined below:

Where

fTTj) is the luminance sample Y (which should be a value between 0 and 255 inclusive);

(u,v) are spatial frequency points in the horizontal, and vertical, dimensions; and (ij) are index values utilized to select the pixels that are within the sample rectangle. Apply ibis transform to the spatial frequency points (u.v) in the horizontal and vertical dimensions (for example u, - 0 to G-i inclusive) where G provides a limit for values of u and v, and G-i is the maximum spatial frequency evaluated in each dimension.

F(i ,I) represents a measure of the fundamental spatial frequency in both dimensions and provides a useful reference point for other spatial frequencies.

To provide a signature of the image, select a subset of F(u,v) for u.v ^::: 0 to G-I inclusive, and evaluate the following:

if F{ 1 , 1)00

W(u,v) - F {u,v)/F(i _; l )

else

W(u,v) ^:::: not valid

For a pre-detemiined value G„ typically in the range 3 to 6, the following is stored in the database for the asset as a reference:

a) W(u,v) {either in fractional binary format - e.g., with 8 fractional bits), or log(W(u,v))

b) F(u,v)

|F(u,v)j provides a measure of the confidence achieved with a match with

\V(u,v),

3) Hue Angle Calculation

Define four rectangles A, B, C, and D such that:

A. is the region between ij ^::: {N/8 and 3N/8-1 )

B is the region between i = (5N/8 and 7 /8-1); j ^« (N/8 and 3 /8-1)

C is the region between i ^::: (N/8 and 3N/8- 1 ); j ^::: (5N/8 and 7N/8-1 )

D is the region between i j - (5N/8 and 7N/8- i )

e. _¾ , ΘΒ, e, θο, are the hue angles evaluated for the mean chrominance values for regions A, B. C, D, respectively.

S _A , SB, SC, $D are the saturations for regions A, B, C, D, respectively. The saturation values provide a confidence measure for the associated hue angles. In addition, the ratios (SA/S») and (SB/SD) provide additional metrics, with confidence _ Q _ values that are defined by JSA-SO| and \$&-Sc\ respectively. Either (S _A /S» or log($A S.o) may be used.

Thus, the process results in six values based on chrominance, plus a number of values based on transformed luminance coefficient ratios. For each, the value plus a confidence measure are stored.

Image Index Comparison Between Referenced (databased calculations) and Incoming

(uploaded image calculations)

When comparing images, exact matches to metrics are unlikely due to differences between li¾htin¾. aamma, and so forth, so an index match should use a number of the metrics, with strength and wei ghting values.

A strength value λ _ρ for metric p may be formed by evaluating:

λρ - jCpj / (K _p + |(Rp - C _j ,j), limited to a defined maximum value,

where:

C _v is the candidate image metric strength value for metric p;

R _p is the reference image metric strength value for metric p;

K _p is a weighting factor for metric p determined empirically; and p is an index used to select which of the metrics are being used.

Similarly, μ _ρ for metric p may be obtained by evaluating;

μ _Ρ = jM _p j / (L _p + !(M _P - X _p i),

where:

Mp is the value for the candidate for metric p;

X _p is the value for the reference image for metric p; and

Lp is a weighting factor for metric p determined empirically.

A match score for metric p is then calculated as:

Finally, the overall match value can be calculated as follows:

match =∑ ψ _ρ for all metrics p.

Thus, the match is simply a sum of the metrics that are available, i.e., one or more of the hue matching, transform coefficient ratio matching, saturation ratio matching, and the like. DiiTerent implementations may choose which to use, as each provides some information, however no one metric is good for all circumstances. Therefore a combination of metrics that assess the match for different attributes is valuable.

FIG, 3 is a flow chart illustrating the steps of an exemplary embodiment of the method of the present invention for searching a catalog of VOD content utilizing automated image recognition. At step 41, the IPLS 15 receives a poster or box cover image from a user's imaging device. At step 42, the IPLS compares the received user image with a plurality of stored reference images associated with VOD content in the VOD catalog. At step 43, it is determined whether the received user image matches a stored reference image associated with a piece of VOD content in the VOD catalog. If so, the method moves to step 44 where the IPLS sends an indication to the user that the matching piece of VOD content is available in the VOD catalog. Thereafter, at step 45, the operator back office 17 receives a request from the user to purchase the matching VOD content. At step 46, the VOD system streams the matching VOD content to the user television 22 through the user's STB 21 associated with an account of the user.

However, if it is determined at step 43 that the received user image does not match a stored reference image associated with a piece of VOD content in the VOD catalog, the method moves from step 43 to step 47 where the I PLS 15 sends an indication to the user that the matching piece of VOD content, is not available in the VOD catalog. At step 48, the operator back office 17 receives a request from the user to reserve the VOD movie when it becomes available. At step 49, the operator back office searches for the matching VOD content whenever new VOD content is received. At step 50, upon determining that the matching VOD content has been received, the VOD system notifies the user that the matching VOD content is currently available for purchase.

Use Case for VOD package ingest from Operator's Back Office

1) A VOD package (movie) is pushed into the Operator's hack office.

2) The VOD package contains poster and/or box cover art.

3) The hack office pushes the poster art into the Poster/Box cover Image Processing and Lookup System.

4) The algorithm described above is applied to the reference image.

5) The reference metrics calculated from (4) are stored with this movie. Use Case for Storing an Image with Title Ingest (for reservation, no back office catalog entry)

1 ) An external (ei titer on the Operator's network or the Internet) system is configured on the Poster/Box cover Image .Processing and Lookup System.

2) The Poster/Box cover Image Processing and Lookup System may either pull the image/title information from the external system or services may be set up based on APIs to allow the external systems to push th image/title information into th Poster/Box cover Image Processing and Lookup System.

3) The algorithm described above is applied to the reference image,

4) The reference metrics calculated from (3) are stored with this movie title in the data store.

Image Lookup with VOD Back Office Match (Title Currently in VOD Catalog)

1 ) A user is running an operator's back office application on a smart device with a camera.

2) The user is in a DVD movie retail location and selects a Blue-ray disc which the user is debating to purchase. Since the user has not seen the movie, the user remembers the VOD lookup by image feature.

3) The user runs the application and selects Art image Lookup.

4) The user's smart device switches to camera mode, and the user takes a photograph of the image on the DVD box.

5) The user's smart device pushes the image via IFI or mobile wireless into the operator's network and into the Poster/Box cover Image Lookup System for lookup.

6) The image processing algorithms are applied to the lookup image using the algorithms described above.

7) The three-step process is followed for lookup matches.

8) If the system finds a match or a limited number of matches, they are displayed on the user's smart device as possible choices.

9) The user selects the title for reservation/rental.

10) The user is prompted to watch now or watch later.

Π) If watch now is selected, the title plays on the user's smart device. 1.2 ) If watch later is selected, the title is placed the reserved rentals category for ease Of watching later on my device the operator supports (Set top box play, mobile device play, or web device play).

Image Lookup with VOD Back Office Match (Title Not Currently in VOD Catalog)

1 ) A user is running an Operator's back office application on a smart device with a camera.

2) The user is in a movie theater and watches a movie.

3) After the movie is completed, the user is leaving the theater.

4) The user loved the movie and would like to watch it again as soon as it is released to be viewed in the borne.

5) The user remembers the application running o the user's smart device and remembers the Poster /Box cover art lookup feature,

6) The user walks up to the movie poster displayed in the lobby of the theater.

7) The user runs the operator's smart device application on the user's device and selects Art Image Lookup.

8) The user's smart device switches to camera mode, and the user takes a photograph of the poster image.

9) The smart device pushes the image via WIFI or mobile wireless into die operator's network and into the Poster/Box cover Image Lookup System for lookup.

10) The image processing algorithms are applied to the lookup image using the algorithms described above.

11) The three-step process is followed for lookup matches,

12) If the system finds a match or a limited number of matches, they are displayed as possible choices.

33) The user selects the title for reservation/rental.

14) Since the selected title is still in theaters, it is not in the VOD catalog for purchase.

15) The system notifies the user that the selected movie title is not currently available. T he user is also presented with an option to be notified as soon as it is available. The user selects "Yes". 16) Several raombs pass and the movie is released on VOD and is ingested into the Operator's back office.

.17) The user is notified thai the movie is available for purchase/rental.

18) The user remembers the movie and purchases the movie.

19) The user is prompted to watch now or watch later.

20) If watch now is selected, the movie plays on the smart device.

21) If watch later is selected, the title is placed in the reserved rentals category for ease of watching later on any device the operator supports (Set top bo play, mobile device play, or web device play).

As will be recognized by those skilled in the art, the innovative concepts described in the present application can be modified and varied over a wide range of appii cations. Accordingly, the scope of patented subject matter should not be l imited to any of the speci fic exemplary teachings discussed above, but is instead defined by the following claims.