CROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application claims priority to U.S. Provisional Patent Application No.

62/887,835, filed on August 16, 2019, which is hereby incorporated by reference herein in its entirety.

TECHNICAL FIELD

[0002] The present disclosure pertains to computer technology for selecting items from at least one item database for displaying at a user interface based on inputs received at the user interface. The present disclosure also relates to online shopping recommendations and more specifically to systems and methods for recommending fashion products to shoppers based on machine-learned visual characteristics of the fashion products that are appealing to the shoppers.

BACKGROUND

[0003] Online shopping and online marketplaces are mainstays for retail outlets with most brick-and-mortar retail outfits providing at least a basic online shopping experience. Although online shopping is prevalent in society, e-commerce keyword-based search technologies are suited to finding products with well-defined and well-describable objective features and attributes. Examples of well-defined features include a brand name, a merchant-defined category, color, and other product attributes defined within the product’s metadata. Other product attributes can include, e.g., sleeve length of a product, a model number, and a description assigned to the product by a designer or a merchant. A text search can find these objective features or attributes within previously recorded keywords (or their synonyms). Upon such a search, products matching the text string being searched upon is displayed on one or more static pages. The products can be ordered by relevance to keywords and/or other factors, such as sales popularity.

[0004] Although efficient in finding products with well-defined product attributes, keyword- based searching is not effective when attributes of products on the online stores are not available. For example, an attribute’s description may be vague or can apply to dissimilar products or styles. In these situations, a desired or preferred product may not be presented to an online shopper. [0005] While the function of fashion products (e.g., “dress” or “trousers”) is easily identified by a keyword, the inherently visual and thus subjective nature of fashion is almost impossible to describe with words. In an example, “floral” can mean a style or many different patterns. Similarly, “urban” can mean a use case, a style, or a visual look. Furthermore, both “floral” and “urban” can mean different things to different people. There is a subjective nature to fashion, and the words used to describe a fashion product does not fully capture the essence of the fashion product.

[0006] Visual-based searching can be applied to augment the shortcomings of keyword-based search by having shoppers upload an image of a product (a “seed” or starting image), but current visual similarity -based search technologies are limited to finding only products that the shopper already has an image of, so this requires the shopper to input into the search engine a starting image that reflects already what the shopper is looking for. These conventional technologies do not let the shoppers find or discover new products they have not seen yet, or captured an image of, or perhaps are not even aware exist. Shoppers still need to resort to inefficient text-based search or category-based browsing through all the existing products (often millions) to find the relevant ones, a frustrating, time-consuming, and overwhelming experience that often results in the shopper abandoning the search without finding a product of interest.

[0007] Category-based browsing or text-based search does not prioritize the products shown by visual desirability, which can be important for something as visual as fashion; and visual- similarity based search only prioritizes the visually matching products, but cannot prioritize the products by desirable styles.

[0008] For the shoppers browsing for the products to buy, the order of products displayed is random with regard to visual desirability and visually pleasing items are infrequent. As a result, shopping fatigue can kick in, resulting in the shopper often quitting the shopping process before finding the desirable product to buy. Hence, current e-commerce is more suitable for finding products that fulfill a desired function (e.g., “brown trousers”), but it is not suitable for discovering visually-appealing new products. In contrast, when shopping in physical stores, shoppers often discover appealing products they did not plan to buy when they entered the store, resulting in a much higher per-visit total shopping spending compared to e-commerce. [0009] The aforementioned shortcomings of current search technologies and of resulting browsing activity result in limitations on the data that can be collected about users’ visual preferences (e.g., by products viewed). A low number of pleasing products means less data points. Limited information on products’ stylistic features limits inference. A static display of products limits hypothesis-testing.

[0010] Attribute, text, and tag based searching can be effective if relevant features of products that affect a user’s choice are well-describable and universally defined across shoppers, designers, retailers, manufacturers, etc., so that relevant keywords can be generated beforehand. This implies that the fashion industry should have a universal taxonomy across different cultures and languages. However, as fashion trends change quickly, such universal taxonomy, even if created, would instantly become obsolete. Therefore, attribute-based search is not suitable for finding products where visual features are relevant to user choice. These features can be difficult for the user to describe, too numerous to be pre-generated, and too ambiguous across customers since different customers can describe the same feature in different ways.

[0011] Additionally, visual features are oftentimes not consciously understood by the user. The user can recognize an item she likes when she sees it, but she cannot verbalize what is it about the item that she likes or that is actually pleasing. In addition, too many different visual features can be appealing to the user to describe in search term, even if the user could describe them. These shortcomings to describing products are typical for fashion and design products. Besides fashion and design, other highly subjective and personalized areas such as interior design, home decor and furnishings, floor plan layouts, landscaping, works of art, architecture, or any area where the subjective user’s “look and feel” are invoked, suffer from the same shortcomings.

[0012] Current e-commerce solutions aim to provide visually relevant products with the help of collaborative filtering recommendation techniques. When a user has expressed interest in an item (e.g. clicked onto product detail page), recommendation is served for similar items not based on identifiable features but based on aggregate opinion of other shoppers interested in that item. Collaborative filtering can have several drawbacks. One drawback is large amount of previous user interactions is needed to provide the recommendation, and this is not always available for less popular or very new products or products that have a short burst of life. Fashion is an area with a very “long tail” (i.e. supply of a large variety of products, each produced in small quantities, and each generating too little data per product). Another drawback is that collaborative-based recommendations are less sensitive to individual tastes, as it aggregates over all other users’ opinions, whereas fashion is one of the most individualistic product areas. Another drawback is current systems do not create accurate user taste profile - for each item on an item-by-item basis; rather, recommendations are given only based on the current or historic items viewed. The limitations of current e-commerce and related search technologies do not allow the technologies to assess the shoppers’ potential stylistic interests if they have not already shown specific interest in these styles (i.e. when they have not discovered the desirable items, while discovery has been made impossible by the same e- commerce systems).

[0013] A potentially large number of item records may be held in one or more item databases, which may be accessed via a user interface. The user interface may provide a search function which allows the item records to be searched based on one or more item attributes specified by a user. The item attributes may for example be specified in a text-based search term. Suitable keywords may be assigned to the item records which can then be compared to the text-based search term. Thus, a user may use the search function to search for any item record pertaining to an item having a specified set of attributes.

[0014] An inherent limitation of such search functions is the requirement for a user to specify the item attributes which form the basis of the search. The search function is therefore only useful in circumstances in which the user already has a specific set of item attribute(s) in mind. [0015] An alternative means of locating desirable items may thus be provided in the form of a recommendation engine. One form of recommendation engine links together certain item records based on aggregate user behaviour. Accordingly, when a given user interacts with a particular item record, he or she may be presented with recommendations for one or more other items which have been linked to that particular item record based on aggregate historic behaviour of other users. For example, underlying machine learning algorithms may be trained to attempt to extract common behaviour patterns from historic user behaviour data to allow item records to be selectively linked together based on the extracted common behaviour patterns.

SUMMARY

[0016] Existing recommendation engines rely heavily on the ability to successfully extract common behavioural patterns from user behaviour data so that item records may be linked together in a meaningful way. One problem with approach is that, to be effective, large volumes of user behaviour data are usually required in order to effectively train the underlying machine learning algorithms. Another problem is the inherent assumption that such common behaviour patterns exist in the first place, i.e. the assumption that aggregate user behaviour is consistent and, moreover, provides a reliable indicator of a given user’s preferences.

[0017] These problems are particularly acute when it comes to items having a complex visual appearance. Visual preference varies significantly between users, therefore reliably extracting common behaviour patterns from aggregate user behaviour data - assuming such common behaviour patterns even exist - would require a large amount of behaviour data and the application of complex behaviour pattern recognition algorithms. At best, a conventional recommendation engine would only be able to provide acceptable recommendations after a significant period of time during which the items have been available and a sufficiently large number of user interactions with the items had been recorded and analysed. At worst, such a recommendation would never perform effectively because the required data would be unavailable or not fit for purpose.

[0018] By contrast, the present technology allows a user to locate items in an item database according to individual visual preference, without any reliance on aggregate user behaviour data.

[0019] A first aspect of the present technology provides a computer-implemented method of determining visual preference of a user for selecting item images from at least one item database according to the visual preference of the user, the method being implemented by one or more computer processors and comprising: selecting a display set of item images from the at least one item database; causing the item images of the display set to be displayed at a user interface; detecting engagement events at the user interface between the user and engaged-with item images of the display set; using the detected engagement events to determine a visual preference hypothesis for the user based on visual features extracted from the engaged-with item images; and updating the display set with new item images selected from the at least one item database by comparing visual features extracted from the new item images with the at least one visual preference hypothesis; wherein subsequent engagement events between the user and engaged-with new item images of the display set continue to be detected and used to refine the at least one visual preference hypothesis, and the display set continues to be updated as the visual preference hypothesis is refined.

[0020] Visual preference is a subjective preference for a visual appearance of an item, and can therefore vary significantly between different users. The presenting teachings allow information about any given user’s particular visual preferences to be systematically inferred and applied, in a way that robustly accounts for the uniqueness of those subjective preferences, thus increasing the speed and efficiency with which any given user can locate visually appealing items. [0021] When a user selects or otherwise engages with a particular item image, that does not necessarily mean he or she has a preference for all of the visual features of that item image. [0022] Accordingly, in embodiments, the visual preference hypothesis may correspond to a type of visual feature or visual feature combination, and may bias the selection of the new item images toward item images having visual features or combinations of visual features of that corresponding type which match the visual preference hypothesis but which exhibit variation in at least one additional visual feature type for testing the visual preference hypothesis.

[0023] This provides an effective way of testing the visual preference hypothesis in question: if a user continues to select or otherwise engage with item images which match the visual preference hypothesis in the corresponding visual feature type(s), but otherwise differ visually, that behaviour is reasonably strong evidence that the visual preference hypothesis is correct. [0024] For example, at least a first visual preference hypothesis and a second visual preference hypothesis may be determined using the engagement events. The first visual preference hypothesis may correspond to a first type of visual feature or visual feature combination, and bias the selection of a first subset of the new item images toward item images having visual features or combinations of visual features of that corresponding first type which match the first visual preference hypothesis but which exhibit variation in a second type of visual feature or visual feature combination. The second visual preference hypothesis may correspond to the second type of visual feature or visual feature combination, and bias the selection of a second subset of the new item images toward item images having visual features or combinations of visual features of that corresponding second type which match the second visual preference hypothesis but which exhibit variation in the first type of visual feature or visual feature combination.

[0025] The first and second visual hypothesis may, for example, be determined in response to the user engaging with at least one item image of the display set. The at least one engaged- with item image may have a visual feature or combination of visual features of the first type which is used to determine the first visual preference hypothesis. The at least one engaged- with item image may have a visual feature or combination of visual features of the second type which is used to determine the second visual preference hypothesis. The biased selection of the first and second subsets tests whether the user has a preference for (i) the visual feature or combination of visual features of the at least one engaged-with item image of the first type based on a response of the user to the first subset, or (ii) the visual feature or combination of visual features of the at least one engaged-with item image of the second type based on a response of the user to the second subset. [0026] The engagement events may comprise at least one active engagement event in which the user selects an item image of the display set at the user interface.

[0027] The item images of the display set may be displayed at the user interface in an item array at respective array positions.

[0028] The item array may be zoomable and/or scrollable.

[0029] In response to the active engagement event, at least one item image may be removed from the display set, and replaced with one of the new item images displayed at the array position of the removed item image.

[0030] The removed item image is removed from the display set based on the array position of the removed item image relative to the selected item image. That is, an item image may be selected for removal based on its proximity to the selected item image within the item array. [0031] Alternatively or additionally, the engagement events may comprise at least one passive engagement event which indicates passive engagement with an item image of the display set without the user selecting the passively engaged-with item image.

[0032] The at least one passive engagement event may comprise at least one of: a zoom event, a scroll event and an item inspection event pertaining to the item array.

[0033] For example, the at least one passive engagement event may comprise one or more scroll events which denote at least one of: a scroll speed, a change in scroll speed, a scroll direction, and a change in scroll direction.

[0034] The item images may be arranged in item groups within the item array based on at least one visual feature type, such that item images with similar visual features of that visual feature type are grouped together.

[0035] Item grouping may be used in conjunction with passive engagement events.

[0036] For example, the above at least one passive engagement event may comprise a scroll event and the visual preference hypothesis is determined or updated based on a direction of the scroll event at the user interface relative to one of the item groups.

[0037] Such embodiments recognize that, when item images are grouped according to visual similarity, the direction of scroll events can be used to infer information about the user’s visual preference, without necessarily requiring the user to actively engage with the item images in question.

[0038] For example, the visual preference hypothesis may be determined or updated to indicate: a preference for a visual feature of that visual feature type which matches the item group when the scroll event is directed toward that item group and/or when a scroll speed towards that item group is reduced by the user and/or when a scroll action past or through that item group is reversed (e.g. the user stops or slows a scrolling action in the vicinity of the item group, or scrolls past the group initially but then scrolls back to it), or a dislike of a visual feature of that visual feature type which matches the item group when the scroll event is directed away from that item group and/or when a scroll speed away past, through or away from that item group is increased or maintained by the user (e.g. the user continues scrolling past the item group or actively increases the scroll speed away from it).

[0039] In addition or as an alternative to item image replacement, new item images may selected dynamically in response to zoom events and/or scroll events at the user interface so as to dynamically populate previously unpopulated array positions within the item array based on the at least one visual preference hypothesis.

[0040] As another example, the at least one passive engagement event comprise may comprise an eye tracking event.

[0041] The preference hypothesis may be determined based on one or more visual features of one or more item images of the display set with which the user has not engaged.

[0042] For example, a lack of engagement with at least one item image of the display set may be detected based on at least one negative interaction event at the user interface relating to the at least one item image.

[0043] For example, the negative interaction event may be a scroll event directed away from the item image.

[0044] The display set may be additionally updated with one or more escape item images from the item database having visual features which are selected because they do not match the visual features of the engaged-with item images.

[0045] Each of the item images may be stored in an item record of the item database with associated item details, and the item details associated with each item image of the display set may be accessible via the user interface.

[0046] The visual features extracted from each item image may comprise visual features which have been extracted by applying one or more trained machine learning visual feature extractors to image data of that item image.

[0047] Alternatively or additionally, the visual features extracted from each item image may comprise one or more visual hyperparameters extracted from that item image.

[0048] Visual features of or extracted from item details or metadata associated with the item images (referred to as “predefined” visual features herein) may be used in combination with the extracted visual features. [0049] At least one derived visual preference hypothesis may be determined based on the at least one visual preference hypothesis, by applying a transformation to at least one visual feature of the at least one visual preference hypothesis, the derived visual preference hypothesis pertaining to the transformed visual feature, and the at least one visual preference hypothesis and the derived visual preference hypothesis may be used to update the display set.

[0050] The or each visual preference hypothesis may be stored with associated information about a level of exposure, via item images of the display set, to visual features for testing that visual preference hypothesis and a level of engagement with item images having such visual features, wherein the display set is updated in dependence on the associated information. [0051] For example, the associated information may be embodied in a confidence measure associated with the visual preference hypothesis.

[0052] This recognizes that low levels of engagement with particular features per se is not necessarily indicative of negative preference, because the user may simply not have had sufficient exposure to such features. However, a high level of exposure coupled with a relatively low level of engagement does provide reasonably strong evidence of a negative preference. Moreover, a high level of exposure coupled with a relatively high level of engagement provides reasonably strong evidence of a positive preference.

[0053] In the first aspect of this disclosure and the embodiments set out above, the visual features used as a basis for inferring visual preference comprise visual features extracted from the images themselves. However, other aspects of the present disclosure can apply the same or similar techniques with any form of visual feature (which may or may not have been extracted from the images). Accordingly, using the above terminology, the techniques may be applied with only visual features extracted through image processing (in a pre-processing step and/or in real-time), extracted visual features in combination with predefined visual features, but also with only predefined visual features, i.e. of or extracted from an associated item description or metadata.

[0054] In another aspect, an improved user interface facilitates efficient item selection based on active and/or passive engagement events (of various possible types) with an item array, providing a rich source of visual preference information.

[0055] According to one such aspect (second aspect), a computer-implemented method of determining visual preference information for a user is implemented by one or more computer processors and comprises: selecting a display set of item images from at least one item database; causing the item images of the display set to be displayed at a user interface at respective positions in an item array, wherein the item array is zoomable and scrollable to navigate the item images of the display set; detecting engagement events at the user interface between the user and engaged-with item images of the display set; updating the display set with new item images selected from the at least one item database based on visual features of the engaged-with item images and visual features of the new item images, wherein the updating comprises at least one of: replacing an existing item image of the display set with a new item image at the same array position, and populating a previously-unpopulated array position of the item array with a new item image.

[0056] In embodiments, visual features may comprise visual features extracted from the item images.

[0057] Alternatively or additionally, the visual features may comprise visual features of or extracted from item details or metadata associated with the item images.

[0058] The engagement events may comprise at least one of: an active engagement event in which the user selects an item image of the display set at the user interface, and a passive engagement event which indicates passive engagement with an item image of the display set without the user selecting the passively engaged-with item image.

[0059] The method may comprise electronically storing visual preference information for the user about the visual features of at least some of the item images with which the user has engaged.

[0060] Another such aspect (third aspect) provides a computer-implemented method of determining visual preference of a user for selecting item images from at least one item database according to the visual preference of the user, the method being implemented by one or more computer processors and comprising: selecting a display set of item images from the at least one item database; causing the item images of the display set to be displayed at a user interface; detecting engagement events at the user interface between the user and engaged-with item images of the display set; using the detected engagement events to determine a visual preference hypothesis for the user based on visual features of the engaged-with item images; and updating the display set with new item images selected from the at least one item database by comparing visual features of the new item images with the at least one visual preference hypothesis; wherein subsequent engagement events between the user and engaged-with new item images of the display set continue to be detected and used to refine the at least one visual preference hypothesis, and the display set continues to be updated as the visual preference hypothesis is refined.

[0061] The visual features of the item images may be extracted, predefined (in the above sense) or a combination of both.

[0062] In embodiments of the second and third aspects, any of the features set out above in relation to the first aspect may be implemented. It will be appreciated that all description and embodiments pertaining to extracted visual features of the first aspect apply equally to predefined visual features in the above sense in the present context.

[0063] Further aspects of this disclosure provide a computer system comprising one or more computer processors and computer memory coupled to the one or more computer processors, the one or more computer processor being configured to execute one or more computer programs held in the computer memory which, when executed, cause the one or more computer processors to carry out any of the steps disclosed herein; and one or more computer programs embodied on transitory or non-transitory computer-readable media, which are configured, when executed by one or more computers, to carry out such steps.

[0064] An embodiment of the disclosure provides a system for recommending at least one product to a user device. The system includes a processor and a non-transitory computer readable medium storing instructions thereon such that executing the instructions causes the system to perform the following: (a) receiving A product descriptions, each product description in the A product descriptions including an image; (b) extracting, for each image in the A product descriptions, a plurality of features including machine-understandable visual parameters; (c) arranging a subset of the A product descriptions relative to a surface in a grid-like manner according to the machine-understandable visual parameters; (d) providing, to the user device, the grid-like arrangement of the subset of the A product descriptions; (e) receiving, from the user device, signals indicating an interaction relative to the surface; (f) in response to the signals, rearranging the subset of the A product descriptions relative to the surface; and (g) sending, to the user device, the rearranged subset of the A product descriptions.

[0065] In an aspect, visual preference of a user is determined by selecting a display set of item images from at least one item database and causing the item images of the display set to be displayed at a user interface. Engagement events at the user interface between the user and engaged-with item images are detected and used to determine a visual preference hypothesis for the user based on visual features extracted from the engaged-with item images. New item images are selected from the database(s) by comparing their visual features with the visual preference hypothesis. This is an iterative process, in which the visual preference hypothesis is refined and the display set continues to be updated accordingly. In another aspect, an improved user interface facilitates efficient item selection based on active and/or passive engagement events (of various possible types) with an item array, providing a rich source of visual preference information.

[0066] An embodiment of the disclosure provides a system for recommending at least one product to a user device. The system is configured to: (a) receive N product descriptions, each product description in the N product descriptions including an image; (b) extract, for each image in the N product descriptions, a plurality of features including machine-understandable visual parameters; (c) arrange a subset of the N product descriptions relative to a surface in a grid-like manner according to the machine-understandable visual parameters; (d) providing, to the user device, the grid-like arrangement of the subset of the N product descriptions; (e) receive, from the user device, signals indicating an interaction relative to the surface; (f) in response to the signals, rearrange the subset of the N product descriptions relative to the surface; and (g) send, to the user device, the rearranged subset of the N product descriptions. [0067] The foregoing and additional aspects and implementations of the present disclosure will be apparent to those of ordinary skill in the art in view of the detailed description of various embodiments and/or implementations, which is made with reference to the drawings, a brief description of which is provided next.

BRIEF DESCRIPTION OF THE DRAWINGS [0068] The foregoing and other advantages of the present disclosure will become apparent upon reading the following detailed description and upon reference to the drawings.

[0069] FIG. 1 shows a schematic block diagram of a computer system for selecting item images from an item database;

[0070] FIG. 2 shows a schematic block diagram of an example set of feature extractors for extracting visual features from an item image;

[0071] FIG. 3 schematically illustrates examples of different visual item feature types, which correspond to respective dimensions of a multidimensional feature space;

[0072] FIG. 4 shows a flow chart for a computer-implemented method of iteratively refining a visual preference hypothesis based on interactions between a user and item images of a display set, and updating the display set as the visual preference hypothesis is refined; [0073] FIG. 5 schematically illustrates how item images may be rendered at a user interface in a navigable item array to allow a user to engage with the item images within the item array; [0074] FIG. 6 shows a simplified example of how visual preference hypotheses may be created, tested and refined;

[0075] FIG. 7 shows a schematic example of a scrollable item array;

[0076] FIG. 8 shows a schematic example of a zoomable item array;

[0077] FIG. 9 shows an example of how a scrollable item array may be updated in response to a user selecting a particular image within the item array; and

[0078] FIG. 10 shows a schematic block diagram of an individual visual preference hypothesis. [0079] FIG. 1-1 illustrates a system for recommending a product or products to a user according to an embodiment of the disclosure.

[0080] FIG. 2-1 is a block diagram showing steps for recommending products to a user according to an embodiment of the disclosure.

[0081] FIG. 3-1 illustrates an example interaction on a user device according to an embodiment of the disclosure.

[0082] FIG. 4-1 illustrates an example system for recommending a product or products to a user according to an embodiment of the disclosure.

[0083] FIG. 5-1 shows an example of a mobile device showing a zoomed out view of different products that can be scrolled through, according to an embodiment of the disclosure.

[0084] FIG. 6-1 shows a zoomed in version of a section of the different products shown in FIG. 5-1.

[0085] FIG. 7-1 shows effect of selecting a certain product in the different products of FIG. 5- 1, according to an embodiment of the disclosure.

[0086] While the present disclosure is susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail herein. It should be understood, however, that the present disclosure is not intended to be limited to the particular forms disclosed. Rather, the present disclosure is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure as defined by the appended claims.

DETAILED DESCRIPTION

[0087] Embodiments of the present technology will now be described by way of example only. [0088] With reference to FIG. 1, there is first provided an overview of certain components of a system 100 for selecting item records from an item database (DB) 102 according to a user’s inferred visual preference.

[0089] FIG. 1 shows a schematic block diagram of the system 100. A user interface (UI) 104 controlled by the system 100 is shown to be accessible to a user. The UI 104 comprises at least one display via which item images may be displayed to the user, and at least one input device (such as a touchscreen, mouse, trackpad, gesture recognition component etc.) via which the user may select or otherwise engage or interact with displayed item images.

[0090] Over time, the system 100 is able to infer visual preference information for the user, use that visual preference information to select appropriate item images from the item DB 102 for displaying to the user based on corresponding visual item features, and iteratively refine the visual preference information and update the displayed item images as the user engages with the displayed item images via the UI 104. The inferred visual preference information is captured in the form of one or more visual preference hypotheses 112.

[0091] Visual item features may be referred to simply as visual features or item features for conciseness.

[0092] Although not shown explicitly in FIG. 1, the system 100 is a computer system comprising one or more computer processors (such as CPUs, GPUs or other accelerator processors etc.) programmed to carry out the function of the system 100 described herein. In particular, the system 100 is configured to implement a selection algorithm 106 for selecting item images, a visual preference inference algorithm 108 for determining one or more visual preference hypotheses 112 for the user and one or more feature extractors 120 for extracting visual features from item images. The selection algorithm 106, visual preference inference algorithm 108 and feature extractors 120 are software components implemented according to program instructions stored in computer memory of the system 100 and executed on the one or more computer processors of the system 100.

[0093] The item DB 102 holds item records, i.e. records of visual items. Three such item records are shown and labelled PI, P2 and P3 respectively. However, it will be appreciated that the item DB 102 may hold a much larger number of item records such that navigating the item DB 102 is challenging. Each item record PI, P2, P3 contains item data (labelled Dl, D2 and D3 respectively) together with a set of visual item features (labelled FI, F2 and F3 respectively). The set of visual item features FI, F2, F3 of each item record PI, P2, P3 comprises extracted visual features 121, as extracted by the one or more feature extractors 120 from at least one item image 112 of that item record. That is, by applying automated feature extraction to image data of the item image 124. Examples of extracted visual item features are described below. The automated feature extraction applied by the feature extractor(s) 120 is a form of “computer vision” analysis. Such computer vision analysis is applied to all item records in the item DB 102, as described in further detail below.

[0094] The set of visual item features FI, F2, F3 may also comprise one or more predefined visual item features 127 (i.e. visual features of or extracted from an associated item description or associated metadata, as opposed to features extracted from the image itself though image processing). As depicted schematically in FIG. 1, the item data of each item record comprises the item image 122 together with associated item details 124. The item data 124 is also associated with item metadata 126, which includes the predefined visual item features 127. The item image 122 is an image of an item to which the item record relates, in which extractable visual features of the item are captured. Both the item details 124 and the item metadata 126 are data about the item. However, whereas the item details 124 may be rendered accessible to the user who is browsing that item, the item metadata 126 is generally only used within the computer system 100. It is noted that, although the predefined item features 127 are shown as part of the item metadata 126, one of more of the predefined visual features 127 may in fact form part of the item details 124 (in the sense that those features may be accessible to a user who is browsing that item). The item details 124 may, for example, be accessible via an item information page, which the user can navigate to via the UI 104.

[0095] Examples of predefined features for garments or similar such items include as sleeve length, collar type etc. or affinity to known fashion styles (among many others).

[0096] For conciseness, the following description may refer to visual features of an item image. Unless otherwise indicated, this means the visual item features extracted from that item image together with any predefined visual features 127 associated with that item image in the applicable item record. Such visual item features, individually and in combination, provide a basis on which the system 100 can understand and thus apply analytical reasoning to items (products) as well as users’ visual preferences.

[0097] The above components cooperate to help users discover the most visually pleasing items amongst a large number of items, according to their individual tastes and preferences. The system 100 is designed so as to:

(1) expose the user to as many visually pleasing items as possible;

(2) maximize the collection of user preference information to enable (1);

(3) repeat (1) and (2) to refine the selection of items displayed to match the user’ s preference accurately, (4) allow the user to re-defme their stylistic preferences, and refine their search when they change their mind (e.g. discover new styles they like, but did not know of previously) during an item browsing session; and

(5) reduce the cognitive load placed on the user in an item browsing session. [0098] The selection algorithm 106 may be referred to synonymously herein as a recommendation engine. The recommendation engine is a form of artificial intelligence (AI), noting that the term AI covers a range of software architectures including, for example, machine learning but also rules-based systems.

[0099] In contrast to a conventional recommendation engine, the recommendation engine 106 is not designed to exclusively deliver maximally pleasing options for the user at each round of recommendation. Rather, the recommendation algorithm 106 can deliberately expose the user to items that might not be most pleasing to the user but would contribute the most information about the user to allow the recommendation engine 106 to make better recommendations in the future.

[0100] The recommendation engine 106 does not require the user to enter the search query or formulate specific search criteria. Instead these are inferred from interaction behaviour. The user does not even need to have a well-defined search goal in mind (not even a hard-to-describe mental image) - the system 100 interactively tests the user for conscious and subconscious preferences to direct him or her towards the most visually pleasing items.

[0101] The system can be characterized as a tool for users to communicate their visual style preferences without using words, but rather by selecting images in what is referred to herein as a “Visual Language of Search”.

[0102] Further details of the above components will now be described in the enumerated sections that follow.

[0103] 1. Computer vision analysis of item images

[0104] FIG. 2 shows an example set of such feature extractors 120 which may be used to extract visual item features from an item image.

[0105] One example of a suitable feature extractor is a convolutional neural network (CNN) 202 or other neural network (NN) feature extractor which has been trained on a suitable set of training images. NN feature extractors are known per se therefore further details are not described herein. A NN feature extractor analyses item images to extract a large number of features characterizing the visual quality of the products. That information is typically encoded in the form of a feature vector or feature map. [0106] Features extracted by aNN feature extractor, or another form of machine learning (ML) feature extractor, may be in the form of machine-understandable visual parameters (referred to as ML parameters for conciseness), such as shape of edges at smaller or larger scale, visual patterns within the items etc - typically a thousand or more qualities defined by the neural network and usually lacking a human-defined counterpart. These may be used to determine visual similarity between item images based on a numerical comparison of corresponding ML parameters between the item images.

[0107] In addition, the extracted visual item features comprise one or more visual hyperparameters, such as color, entropy etc. Such visual hyperparameters are extracted by at least one hyperparameter extraction component 204. The hyperparameter extraction component 204 may also be a neural network or other form of ML component, or it may be implemented as a classical (non-ML) image processing algorithm.

[0108] A visual hyperparameter is a characterization of an overall aspect of an item’s visual appearance. Other examples of visual hyperparameter include an overall colour hue or colour saturation, an indication of whether or not the item is patterned (and/or an extent to which it is patterned), darkness/lightness, relative size of the any features; overall entropy etc.

[0109] Other examples of visual features, which are not visual hyperparameters (and may be extracted or predefined) include the presence or absence of specific components such as metal studs, zips etc. and/or the number, prominence etc. of such components (e.g. a particular hyperparameter may denote the presence of several prominent zips), and other measures of overall style such as fabric, fit (e.g. slim vs regular vs loose) etc.

[0110] As described in further detail below, the item features extracted by the feature extractor(s) 120 are used in combination with any predefined visual features 127 as a basis for making individual item recommendations to the user.

[0111] With reference to FIG. 3, the full set of visual features (predefined and extracted) are defined in a multidimensional feature space. Each dimension of the feature space corresponds to a visual feature type (such as sleeve length, colour or a particular ML parameter). The value of each feature dimension for given image constitutes a visual feature of the corresponding type according to the terminology used herein. Item features of an item image are represented numerically as a vector point within the multidimensional feature space, such that geometric relationships between different vector points mirror visual relationships between item images represented by those points.

[0112] By way of example, FIG. 3 shows first, second and third images 302, 304 and 306. Dimension v of the feature space corresponds, in this example, to a predefined “sleeve length” parameter, feature y to an extracted “colour” hyperparameter (e.g. a dimension in a defined colour space) and feature z to a specific ML parameter (which may lack a straightforward human-interpretable counterpart) as extracted by the ML feature extractor 202.

[0113] The values xi, xi and V3 of dimension v for the first, second and third images 302, 304, 306 indicate, respectively, sleeve lengths of respective garments visible in those images. As depicted schematically in FIG. 3, the garments of the second and third images 304, 306 have similar and relatively long sleeve lengths, whereas the garment of the first image 302 has relatively short sleeves. Hence, xi and .« are relatively near to each other in dimension x of the feature space, as compared with x\. Similarly, the values y i and \ ₃ of dimension (colour) for the first and third images 302, 306 are similar because the garments visible in those images are similarly coloured, in contrast to the value yi of dimension y for the second image 304 which depicts a different coloured garment.

[0114] Visuals similarity is similarly captured in geometric relationship between features in other dimensions, including those which correspond to ML parameters.

[0115] FIG. 4 shows a high-level flow chart for an iterative method of selecting item images from the item DB 102 for displaying to the user.

[0116] Within the system 100, a “display set” 107 is defined. The display set 107 contains item images selected by the recommendation engine 106 from the item database 102 for displaying to the user at the UI 104. Note that, not every item image in the display set 107 is necessarily displayed simultaneously at the UI 104, however the user has the option of navigating to any of the item images in the display set 107 via the UI 108.

[0117] At step 402, the display set 107 is created initially by the recommendation engine 106 selecting a number of initial item images from the item DB 102. At this point, it is assumed that no visual preference information is known for the user. Accordingly, the initial item images may be selected at random or in a strategic manner based on the visual features of the item images so as to maximize the user’s exposure to as many different visual features or combinations of visual features as possible. Alternatively, if a certain amount of information is already known about the user’s visual preferences (e.g. a basic level of visual preference information may be explicitly provided by the user), this may be used to assist the selection of the initial item images.

[0118] The user is thus free to inspect and otherwise interact with item images of the display set 107 via the UI 104. [0119] As the user interacts with particular item images of the display set, those interactions are monitored and recorded (404) in a user profile denoted by reference numeral 110 in FIG. 1

[0120] Interactions can include both positive interactions (“engagement events”), i.e. which indicate a preference for a particular item image as well as negative interactions, i.e. which indicate that the user actively dislikes’ a particular item image. Engagement events can include both active engagement events, in which a user specifically selects a particular item image, as well as passive engagement events which indicate a preference for a particular item image without the user actively selecting it. Examples of engagement events and other forms of interaction are described later.

[0121] The interactions recorded in the user profile 110 are used, by the visual preference inference algorithm 108, to determine (406) the visual preference hypotheses 112 for the user. Once determined, the visual preference hypotheses 108 can, in turn, be used to update (408) the display set with new item images. This commences an ongoing iterative phase of the method, in which the visual preference hypotheses 112 are refined as the user continues to interact with the updated display set 107, and the display set 107 continues to be updated as the visual preference hypotheses 112 are refined.

[0122] 2. Item array layout

[0123] FIG. 5 shows a highly schematic depiction of a dynamic “item array” 500 in which item images of the display set 107 are displayed within an available display area 501 of the UI 104. For example, at least one item browsing page may be provided within the available display area 501, within which the item array 501 is displayed.

[0124] The item images are displayed at respective array positions within the item array 500. That is, the item images of the display set 107 are displayed in a uniform arrangement within the available display area 501 at defined array positions. By way of example, two of the item images in the item array are denoted by reference numerals 502 and 512 respectively and are displayed at array positions 504 and 514 within the item array 500 respectively.

[0125] The item array 500 is both scrollable, i.e. the user can scroll through the item array to render different parts of the item array 500 visible (and hence view different item images which currently form part of the display set 107), and zoomable, i.e. the user can zoom in (resp. out) to show a lesser (resp. greater) number of item images within larger (resp. smaller) respective portions of the available display area.

[0126] FIG. 5 shows additional details of an example item browsing session, which are described in due course. [0127] In this example, the item array 500 takes the form of a very large two-dimensional grid that can be scrolled via the UI 104 both horizontally and vertically. The item browsing page supports a wide range of zoom levels, from hundreds of very small images per page up to few very detailed images per page.

[0128] For example, a “pinch zoom” gesture may allow user of a touchscreen device to effortlessly zoom in on an area of interest in the grid. The products are arranged in the grid 500 so that a large amount of information on user preference can be captured by following the user’s actions on the grid 500. This can include areas high on specific parameters, or mixture of parameters at different zoom levels. More generally, zoom events can be instigated by a pinch gesture or any other relevant gesture, such as mouse wheel movement, virtual reality (VR) or game controller buttons etc.

[0129] On lower zoom levels, many more products than in typical navigation can be shown at once. This exposes the user to more products and thus, on average, a greater number of visually pleasing products. As many stylistic decisions can be taken on low detailed view, this allows the user to quickly make an initial selection from large variety of stylistic options. At the same time, the user can further refine interest by zooming in on products that are potentially relevant. The low-detail zoom levels allow the user to easily travel between different fashion dimensions, such as category, style, color, shape etc.

[0130] This is in contrast to existing item browsing layouts which expose users to a limited number of products (failing to allow the user to see other products that may match their style interests), often already filtered by a functional category (failing to expose the user to products outside the chosen category).

[0131] Using zoomable grid navigation, the user generates large amount of preference data quickly via interaction events at the user interface 104. Micro-actions, such as - but not limited to - a relatively longer examination of a particular area (inspection event), zooming in on a particular portion or he item array (zoom event), how quickly did she zoom out again. These are all examples of passive interaction events, which may be monitored together with active engagement events (such as selecting or liking a product) to understand user preferences. [0132] One form of passive engagement event is an item inspection event, which may be defined relative to other events. A user may be considered to inspect an item for a duration between two other navigation events (e.g. zoom or scroll events), and that may be referred to herein as an interaction event of that duration.

[0133] Specific examples of passive engagement include a user zooming in to a specific product or product group, indicating positive interest (and zooming out to indicate disinterest) or the user scrolling towards or away from any product or product groups. The speed of the scroll event may be taken into account, for example when the user stops or slows the scrolling motion to observe a product or group of products (hesitating if such a product/group is of interest, or not); or a faster scroll could indicate a lack of interest in group of products. It will be appreciated that these examples are not exhaustive, and that other forms of passive engagement with the item array may be used to infer visual preference.

[0134] Another example of a passive engagement event is an eye tracking event, which can be detected using one or more eye tracking sensors (such as image capture device(s)).

[0135] Each area of the grid displays products with known extracted features 121, hence as the user interacts with the item images of the display set 107, the system 100 can start to infer visual preference information for the user, in the form of the visual preference hypotheses 112. [0136] Whilst the grid of products displayed initially may appear to include random set of products, these are preferably chosen strategically by the recommendation engine 108 to maximize exposure to a number of styles, which in turn helps the user to find the closest visual matches to their desired product styles.

[0137] The grid 500 can also be arranged into areas (item groups) of stylistically similar products, based on category, color, style or other important feature dimension. This functions as a navigation aid to the user and allows to preferences to be inferred from scrolling actions. [0138] In other words, item images of the display set 107 may be grouped into item groups, within the item array 500, according to one or more visual feature types (dimensions), so as to group together item images with similar visual features of that type. With item images grouped in this way, user preference can start to be inferred based on scroll event direction. For example, if item events are grouped according to a particular type of style hyperparameter, when a user scrolls toward a particular item group, this may indicate a preference for the corresponding style. Conversely, if a user scrolls away from a particular item group, this may indicate a dislike of the corresponding style.

[0139] Considering this in terms of the feature space, by grouping item images together based on a particular dimension(s) of the feature space, such that each item group corresponds to a particular subspace of the feature space, scroll events can be mapped to subspaces of the feature space based on directionality of the scroll event, and therefore used as a basis for “dividing up” the feature space according to a user’s hypothesised individual visual preferences.

[0140] As the system 100 becomes more confident in the visual preference hypotheses 112, the grid 500 will be mostly populated with items the recommendation engine 106 is confident the user will like. However, it may continue to populate the grid with a number of “escape items” that represent a different style and can be used to indicate saturation with current choice and desire to explore radically different options. Escape items are strategically chosen on the basis that they have visual features which do not match images the user has engaged with, in order to provide avenues for exploring quite different visual preference hypotheses.

[0141] Each of the visual preference hypotheses 112 imposes a biasing effect on the selection algorithm 106. That is, each visual preference hypothesis has the effect of biasing the selection of item images toward items images suitable for testing that hypotheses, i.e. so as to increase the likelihood of such images being selected for the display set 107. The strength of the biasing effect imposed by a given hypothesis may be increased as the confidence in that hypothesis grows, i.e. as the confidence increases, item images which match that visual preference hypothesis are increasing likely to be selected for the display set 107.

[0142] 3. Interactive funnel of interest

[0143] Upon clicking or otherwise engaging with a product image at the UI 104, a number of product images surrounding it will be changed to new product images that:

• match more closely the user’s stylistic preference, by analyzing the user’s stylistic preference from their action and micro-actions (passive and active) at the zoomable and scrollable grid 500;

• are selected to display a variation of products that help the user further refine their stylistic preference - i.e. to provide additional insight into user preferences.

[0144] This process can be done repeatedly to either refine the selection around a certain visual style, by clicking on mostly similar items, or to expand possibilities and explore by clicking on items along a certain visual direction.

[0145] This provides what is referred to herein as an “interactive funnel of interest”. The interactive funnel of interest allows the system to actively infer user preferences. When a user clicks on (or otherwise selects) an item image, it typically expresses likability. However, the user may not like the engaged-with item(s) in its entirety but just some of its features. The item may also be in the desired visual direction but not far enough (or too far). A single click on an item does not tell which features the user liked. With the interactive funnel of interest, desirability of different features of the product can be tested by displaying products emphasizing one or another feature of the liked product and observing further engagements. [0146] A variant of the system 100 augments the zoomable grid layout 500 with processes from the interactive funnel of interest, in order to provide a dynamic map layout. In this variant, when the user scrolls the map, newly visible portions of the map are dynamically generated based on the information gathered on user intent and preferences so far, i.e. based on the visual preference hypotheses 112. The data can include micro-actions such as direction of ongoing scroll (when items are placed on the map not randomly but based on their stylistic or visual features), zoom-in areas, and others; or actions like previously collected item clicks. Items displayed in the newly generated portions are chosen to be either most pleasing or most informative. Only previously unseen areas are dynamically generated while the areas that the user has seen at sufficiently detailed zoom level will stay unchanged. Dynamic map layout can also have the click-based interactive funnel of interest.

[0147] Returning to FIG. 5, an example of a possible item browsing session will now be described.

[0148] Each item image of the display set 107 is associated with the array position at which it is to be displayed in the item array 500, and the item array 500 is updated (at least in part) by removing item images from the display set 107 and replacing them with new item images at the array positions of the removed item images.

[0149] To illustrate how the user may choose navigate the item array 500, FIG. 5 shows the item array 500 in a sequence of states, denoted by numbered circles 1-4.

[0150] In state 1, the item array 500 is rendered at a relatively low zoom level. That is, with a large number of item images displayed in relatively small portions of the available display area

501. This is useful for identifying candidate item images within the item array 500, which the user can the zoom in on.

[0151] In state 2, the user has zoomed in on item image 512 in order to more closely inspect it.

[0152] In state 3, the user has just selected item image 512, causing nearby item images to change (i.e. be replaced with new item images) in response (state 4). This includes item image

502, which is removed from array position 504 and replaced with a new item image at the same array position 504.

[0153] In this particular example, all of the item images at array positions surrounding (i.e. immediately adjacent) to the array position 514 of the selected image 512 are changed. More generally, item images may be selected for replacement based on their array positions relative to the array position of the selected item.

[0154] Applying the iterative method of FIG. 4, the selection of the item image 514 may cause at least one visual preference hypothesis 112 to be newly created and/or an existing visual preference 108 to be updated. The visual preference hypothesis is created or updated based on one or more visual features of the selected image 512, as described later.

[0155] 4. Hypotheses and recommendations

[0156] A core function of the selection algorithm 106 is the selection of new item images from the item DB 102 in response to passive and active engagement events at the UI 104. Such new images may be selected to replace existing item images, or be selected to populate the item array dynamically as the user navigates to previously unseen array positions.

[0157] Continuing the example of FIG. 5, the new item images which replace the surrounding item images (including the image 502) are selected by comparing their visual features with the newly created or updated visual preference hypotheses 112. Hence, that selection takes into account relationships between the visual features of the item image 512 which the user has just engaged with and the visual features of the new item images. This provides a highly engaging experience for the user.

[0158] The visual preference extraction algorithm 108 is made up of a set of component algorithms that infer user preferences from the stream of actions and micro-actions during an item browsing session at the UI 104. These, in turn, may be used by the recommendation engine 106 to determine an optimal layout of products to be shown to the user - both with the intention of selecting items it thinks the user will visually prefer (i.e. which match a given visual preference hypothesis), as well for refining understanding of user preferences (i.e. to allow existing visual preference hypotheses to be refined or new such hypotheses to be explored, even if that means showing a few item images for which there is no direct indication of preference in the user’s action to date).

[0159] The main source of user preferences is the set of visual item features extracted from the item images and user’s behaviour towards these products, though as noted predefined visual features may also be taken into account. When a user clicks on an item image with a certain feature set (e.g. Dress, red, Gucci, country-style, bell-shaped etc.), it strengthens hypothesized user preference towards these features.

[0160] A sequence of interactions over time may taken into account - for example, persistent selection of a certain feature type (clicking dress; dress; dress) can indicate strong preference whereas intermittent selection (clicking dress; trousers; dress) can indicate indifference to that dimension of the feature space (in the example: category).

[0161] Negative information will also be taken into account - when user is shown items with certain features and chooses to ignore these (by not selecting it, scrolling away, zooming out etc), it weakens hypothesized preference towards these features or leads to negative preference. Many different such criteria can be used, and the above is only intended to serve as examples of the general principle that user’s visual preferences will be inferred from her sequence of actions towards products with known visual features (both extracted and predefined).

[0162] By way of further illustration, FIG. 6 shows an example of a set of visual preference hypotheses 112 which may be created and tested in response to a user engaging with an item image.

[0163] FIG. 6 shows the item array 500 at a zoom level at which nine images (3 x 3) are visible. An item image II from item record PI is shown at a central array position. As described previously, the item image II is associated in the item record PI with a set of visual item features F 1. When the user selects item image II from within the array, this indicates a possible preference for one or more visual features of the item image FI. However, at this point, the system 100 does not know which of those features (or which combination of those features) the user has a preference for.

[0164] Purely by way of example, the hypothesis extraction algorithm 108 is shown to have created first, second and third visual preference hypotheses 112a, 112b and 112c in response to the selection of the item image II.

[0165] The first visual preference hypothesis 112a pertains to dimension x of the feature space, which in this example corresponds to a “sleeve length” visual feature type. The first hypothesis 112a speculates that it is the sleeve length of item II that the user likes. In order to test the first hypothesis 112a, the recommendation engine 106 selects a first subset of new item images 602a having similar sleeve lengths, i.e. similar values in dimension v of the feature space, but which exhibit variation in one or more other visual feature types, i.e. in one or more other dimensions of the feature space. This particular example considers a colour component (dimension y), such as colour saturation, as a dimension in which the item images of the first subset 602a exhibit variation, in order to test the first hypothesis 112a. In this example, the item images of the first subset 602 are of garments with similar sleeve lengths but in different colours (or, more specifically, different colour saturation levels).

[0166] The value of dimension v for the item image II is denoted by xl and the value of dimension by yl. The first subset of item images 602a are shown to have values x5 , x22 and x58 in dimension v of the feature space, which are all similar to the value xl. The values of dimension y are denoted y5, y22, and y58 and those images are chosen in part because those values significantly different from each other and from yl in order to effectively test the first hypothesis 112a. [0167] The second visual preference hypothesis 112b pertains to dimension y of the feature space, i.e. colour saturation in this example. Hence, a second subset of item images 602b is selected for testing the second hypothesis 112b on the basis that those images have similar colour saturations, i.e. similar values in dimension of the feature space, but exhibit variation in one or more dimensions other than y; in this example, they exhibit variation in at least dimension v of the feature space (sleeve length). This materialises in the selection of item images which have different sleeve lengths but similar colours for the second subset 602b. [0168] The values of dimension v for the second item image subset 602b are denoted by xl9 , x31 and x94. The values in dimension y are denoted yl9 , y31 and y94. The feature yl of item image II forms the basis of the second hypothesis 112b, and consequentially the second test images 102b are selected so that the values y!9 , y31 and y94are all similar to yl, but the values xl9 , x31 and x94 differ from each other and from xl.

[0169] The third hypothesis 112c is, by contrast, derived from the first and second hypotheses 112a, 112b. The intention of the derived hypothesis 112c is to test whether, in fact, the user might prefer a more “extreme” colour of an otherwise similar item. For example, a more saturated version of the colour of the selected item image II, but otherwise similar.

[0170] In this case, a transformation is applied to the feature yl in dimension y of the second hypothesis 108b in order to obtain a transformed feature yl' = yl + D (i.e. increasing the level of saturation) which forms the basis of the third hypothesis 108c, along with xl which is incorporated into the third hypothesis 108c unchanged (such that the cumulative effect is to show garments with similar sleeve lengths but in more saturated colours).

[0171] As shown at the bottom left of FIG. 6, the array positions surrounding the central array position of the selected image yl are updated so as to replace the existing images with the new item images 602 and 602b. In addition, a third new item image 602c is shown which is selected for testing the third hypothesis 108c, as a garment with similar sleeve length xl but more extreme colour saturation yl' .

[0172] Now that the item array 500 contains a range of images suitable for testing the visual preference hypotheses, as the user continues to interact with the item array 500, the visual preference hypotheses can be refined based on the user’s response to the new item images 602a-602c. For example, should the user select an image from the first subset 602a but not he second subset 602b, that will generally have the effect of increasing the confidence assigned to the first hypothesis 108a relative to the second hypothesis 108b. This, in turn, may strengthen the biasing effect of the first hypothesis 108a relative to the second hypothesis 108b, which in turn increases the relative likelihood of item images being selected so as to match the first hypothesis 108a.

[0173] As will be appreciated, this is a highly simplified exampled intended to aid understanding of certain principles underlying the creation, testing and refinement of visual preference hypotheses. In practice, those same principles may, for example, be extended to a greater number of hypotheses, more complex hypotheses (e.g. pertaining to combination of multiple visual feature types), and to take into account a potentially much greater number of engagement events (passive and active) as well as negative interaction events. Moreover, whilst this example considered a “positive” hypothesis i.e. about visual item features which the user is hypothesised to have a preference for, “negative” and “neutral” hypotheses are equally viable. A negative hypothesis supposes that the user has an active dislike of a certain visual feature or a certain visual feature combination. A neutral visual preference hypothesis supposes a user is neutral about, i.e. neither likes nor dislikes, a certain visual feature type of types. [0174] 5. Hypothesis confidence

[0175] When it comes to testing a visual preference hypothesis, both engagement and exposure are pertinent.

[0176] If the user has been exposed to a large number of item images which are suitable for testing a given visual preference hypothesis, and has responded to those item images in a manner consistent with that visual preference hypothesis, then the system 100 can be relatively confident that the visual preference hypothesis is correct.

[0177] On the other hand, if the user has been exposed to a large number of such item images, but has not responded to them in a manner consistent with the visual preference hypothesis, the system 100 can be reasonably confident that the visual preference hypothesis is not correct. [0178] When the user has only been exposed to a limited number of item images suitable for testing a given visual preference hypothesis, the system 100 will be less sure either way. [0179] Confidence may for example be captured in a confidence measure assigned to a given visual preference hypothesis, which takes into account both relevant exposure and relevant engagement.

[0180] With reference to FIG. 1, by way of example, four individual visual preference hypotheses are shown and denoted by reference numerals 122a to 122d respectively. In the present example, each of those individual visual preference hypotheses pertains to a single visual feature but it will be appreciated that a visual preference hypothesis could also pertain to e.g. a combination of multiple visual features. Each individual visual preference hypothesis 113a to 113d is shown to comprise an assigned confidence measure denoted by reference numerals 113a to 113d respectively.

[0181] FIG. 10 shows a schematic block diagram of a visual preference hypothesis 112n pertaining to one or more visual item features 1001. The visual preference hypothesis 112 is shown to comprise an assigned confidence measure 113n, which in turn is shown to comprise an engagement component 1002 and an exposure component 1004.

[0182] The exposure component 1004 indicates a level of exposure to item images for testing the visual preference hypothesis 112n (e.g. the total number of such item images the user has been exposed to). Following the example of FIG. 6, these could for example be item images with visual feature(s) which match the one or more visual item features 1001 of the visual preference hypothesis 112n, but exhibit variation in other visual feature type(s).

[0183] The engagement component 1002 indicates a level of engagement with those images (e.g. the number of such item images the user has selected or otherwise engaged with at the UI 104). Once the level of exposure reaches an exposure threshold, the visual preference hypothesis 112 may, for example, be removed or invalidated if the level of engagement has not also reached an engagement threshold by that point.

[0184] Alternatively, the visual preference may not be removed or invalidated even when the confidence is low. As noted, the strength of the biasing effect that the visual preference hypothesis 112n has on the item image selection process may increase as the system 100 becomes more confident in that hypothesis. In the present example, the strength of the biasing effect may thus depend on both the engagement level and the exposure level, and increase as both the engagement level and the exposure level increase (up to a certain point), but decrease as the engagement level increases without increasing the engagement level (possibly all the way to zero).

[0185] 6 Example item arrays

[0186] FIG. 7 shows a zoomed-out view of large bi-directionally scrollable item array 500 (product map) displayed on a touchscreen display 702 of a mobile device 700. The UI 104 is provided, in this example, as the touchscreen display 702. In this zoomed-out view (low zoom level), a relatively large number of item images are visible on the touchscreen display 702. As can be seen, the item array 500 extends beyond the available display area of the touchscreen display 702 in the sense that item images of the display set 107 have already been selected for array positions within the item array 500 which are not currently visible. In other words, there exist a number of array position within the item array 500 which have already been populated with selected item images but which are not currently visible. The user can instigate scroll events (e.g. scroll gestures) at the touchscreen display 702, in order to navigate to a such array positions, in order to see the item images with which they have been populated. As noted above, unpopulated array positions may also be populated dynamically with new item images selected from the item DB 102 in response to scroll/zoom events at the touchscreen simple 702, either as they become visible or in anticipation of them becoming visible.

[0187] FIG. 8 shows a zoomed in view of the scrollable product map 500 for closer inspection of interesting items (to understand whether the user like the item).

[0188] Note, the zoomed in view does not replace the product information page in which the item details 124 are displayed. Rather, in this example, a separate product information page is provided (not shown), where, for example, multiple views of the item may be available (i.e. one or more additional item images may be included in the item details 124), along with other details such as brand, price, description etc.

[0189] FIG. 9 shows a further example of how the item array 500 may be updated in response to a user selecting a particular item image. In this example, product images around the selected item image - not limited to just those in its immediate vicinity - change to related item images that are either similar to the original item or only partially similar to allow exploration (by applying the item image selection principles set out above).

[0190] Although specific grid arrangements and display areas are depicted in the figures, it will be appreciated that these are illustrative and by no means exhaustive. An item array can be implemented with a display area of any size or dimensions on a wide variety of devices. Accordingly, the number of item images that is visible at a given zoom level and the arrangement of those item images within the available display area will vary depending on the context, and in particular may depend on the size and dimensions of the available display area. [0191] Whilst specific embodiments have been described for the sake of illustration, variations and alternatives will no doubt be apparent to those skilled in the art in view of the teaching presented herein. The scope of the present disclosure is not defined by the described embodiments but only by the accompanying claims.

[0192] Embodiments of the disclosure provide a system for helping users discover visually pleasing items that are relevant to the user’s subjective and individualized preference from among a large number of products. The system can expose shoppers or users to as many visually pleasing items as possible. The system can also maximize acquisition of user preferences to enable the exposing of users to as many visually pleasing items. The system can refine selection of items displayed to match users’ preferences more accurately. The system can allow users to re-defme their stylistic preferences, refine their search when they change their mind (e.g., discover new styles they like, but did not know of previously) during a browsing session. The system can reduce cognitive load placed on users during a shopping process.

[0193] Embodiments of the disclosure provide a system for enhancing a user’s shopping experience by engaging the user as a shopper and as a recommender or recommendation artificial intelligence (AI). The system can propose to the user pleasing options but then include a random item or an item that might or might not be the most pleasing to the user. Based on the user’s interaction or lack of interaction with the random item or item that might not be the most pleasing to the user, the recommendation AI can update its profile on the user or update its perception of the user’s preferences. In other words, the AI can test a hypothesis by displaying a certain item to the user, and then watch or monitor how that user interacts or does not interact with the item (e.g., does the user hover over that item, or select it immediately, return focus to the item multiple times, or zoom in to focus on that item, etc.) to update the learning model.

[0194] Embodiments of the disclosure can reduce cognitive load placed on the user in the shopping process since the user is not required to enter a search query or formulate specific search criteria. Instead user preferences are inferred from interaction behavior. That way, the user does not have to have a well-defined search goal in mind, not even for a hard-to-describe mental image. In some embodiments, the user is interactively probed for conscious and subconscious preferences to direct her towards items that are pleasing. Embodiments of the disclosure provide a tool for users to communicate visual style preferences without using or inputting words, but rather purely by interacting with images (i.e., the “Visual Language of Search”).

[0195] FIG. 1-1 illustrates a system 100-1 for recommending a product or products to a user according to an embodiment of the disclosure. The system 100-1 includes a recommendation server 101-1, one or more data repositories or databases (e.g., product data repository 102-1, user profile repository 110-1, and hypothesis testing repository 112-1), and a user device 108- 1. Each of the recommendation server 101-1, the user device 108-1, and the one or more repositories are computing devices with a processor and a non-transitory computer readable medium. Examples of the user device 108-1 can be a tablet computer, a laptop computer, a smart phone, a smart watch, any computing device having at least a human-machine input interface and an electronic graphical display, and so on.

[0196] The recommendation server 101-1 includes a machine learning engine 104-1, a selection engine 106-1, and a tracking engine 114-1. An engine is a combination of hardware and software for realizing a certain functionality. The machine learning engine 104-1 can be a neural network implementation for receiving product description and images and providing product features. The selection engine 106-1 provides the user device 108-1 with a selection of products. The tracking engine 114-1 tracks user interaction with the selection of products. The tracking can include zooming patterns of the user, the clicking on a product, providing a rating on a product, and so on. The machine learning engine 104-1, the selection engine 106- 1, and the tracking engine 114-1 can be implemented on the same hardware.

[0197] The product data repository 102-1 stores product description and/or product features. Examples of a product description includes an image, a keyword description, a phrase description, metadata, etc. Examples of product features include machine-understandable visual parameters such as shape of edges, visual patterns, color, entropy, and so on.

[0198] The user profile repository 110-1 stores user preferences. The hypothesis testing repository 112-1 stores a next hypothesis for the selection engine 106-1 based on the user preferences in the user profile repository 110-1.

[0199] FIG. 2-1 is a block diagram showing steps 200-1 for recommending products to a user according to an embodiment of the disclosure. At step 202-1, the machine learning engine 104-1 of the recommendation server 101-1 receives product descriptions. The product descriptions can be obtained from the product data repository 102-1. The product descriptions can also be received from another computing product, e.g., through a desktop, laptop, or mobile device of a store employee. The product descriptions can contain, for each product, a product image, keyword description, and other metadata.

[0200] At step 204-1, the machine learning engine 104-1 of the recommendation server 101-1 extracts features in the product images, performing a computer vision analysis of the product images. The machine learning engine 104-1 extracts a large number of features characterizing the visual quality of the products. For example, for fashion products, one or more of the following features can be characterized: pre-defmed features (such as, sleeve length, collar type etc. or affinity to known fashion styles), visual hyper-parameters (such as, color or entropy), or machine-understandable visual parameters (such as, shape of edges at smaller or larger scale, visual patterns within the items, etc.). The machine-understandable visual parameters can be numerous, e.g., having 1000 or more qualities defined by the machine learning engine 104-1, and can lack a human-defined counterpart that is classically used in determining visual similarity. The extracted features, including a combination of the extracted features, will form the basis of understanding products in the system 100-1 as well as user preferences. [0201] At step 206-1, the machine learning engine 104-1 stores the extracted features associated with each product in the product data repository 102-1.

[0202] At step 208-1, the selection engine 106-1 of the recommendation server 101-1 places and arranges products on a large surface to be displayed to the user. The large surface can be a large rectangular or square grid on an electronic display of the user device 108-1. The large surface can also be a large spherical grid with a curvature. The items in the grid can be arranged in a table arrangement as in columns and rows, or in any regular or haphazard arrangement within the grid. The grid does not have to be a regular shape, but rather represents the visual canvas area upon which items are displayed to the user/shopper. Those skilled in computer programming will appreciate that the grid is a program abstraction to instruct a display engine or module where to place an item for display. The large grid is provided to the user device 108-1 for display. The large grid is scrollable on a screen of the user device 108-1 in one or more directions, e.g., vertically, horizontally, diagonally, and so on, using a human-machine interface of or associated with the user device 108-1. The page on which the large grid is shown on the user device 108-1 can support a wide array of zoom levels, from hundreds of very small images per page up to few very detailed images per page. The page can support interactions such as “pinch zoom” that allows zooming in on an area of interest in the grid using a touch interface on the screen of the user device 108-1. In addition to touching, other examples of interactions can further include gestures and mouse clicks.

[0203] The screen of the user device 108-1 is an example of an electronic display device according to some implementations of the disclosure. The electronic display device is configured to display to the user/shopper the large grid. Other examples of the electronic display device include projection devices, holographic displays, and so on.

[0204] The products are arranged in the grid so that maximum amount of information on user preference can be captured by following her actions on the grid. This can include areas high on specific parameters, or a mixture of parameters at different zoom levels. A zoomable grid provides advantages, e.g., on lower zoom levels, many more products than in typical navigation can be shown at once. Showing more products at once can expose the user to more products and thus also more visually pleasing products. As many stylistic decisions can already be taken on low detailed view, this allows the user to quickly make an initial selection from large variety of stylistic options. At the same time, the user can further refine interest by zooming in on products that are potentially relevant. In contrast, the current e-commerce layout exposes the users to: (1) a limited number of products (that does not let the user see products that may match their style interests), and (2) often already filtered by a functional category (that does not expose the user to products outside the chosen category). The low-detail zoom levels allow the user to easily travel between different fashion dimensions, such as category, style, color, shape etc.

[0205] Another advantage includes using zoomable grid navigation, the user generates large amount of preference data quickly. Micro-actions, such as - but not limited to - which areas did the user examine longer, where did she zoom in, how quickly did she zoom out again, clicking, or liking a product, etc., can be monitored to understand user preferences as each area of the grid displays products with known extracted features. Determining whether the user zoomed out all the way or how quickly the user zoomed out is a function of the screen size of the user device 108-1. A large monitor will have more products in a zoomed out state than a screen of a smartphone. Capturing and monitoring a user’s micro-actions, along with generating a time sequence of the micro-actions along a timeline, seek to model how a sales person watches a prospective client in a retail store to understand that shopper’s preferences based on what the shopper focuses on, how long the shopper focuses on a particular item, what they express a disinterest in, and the like. These micro-actions, optionally along with their time sequence (e.g., to see a time period between interactions, or how long the user lingers on a particular item or particular area of the screen), can be fed into and ingested by the AI model to improve its learning and refine its selection of items to be displayed on the screen, and which displayed items to replace with items that have a higher chance of being visually interesting to the user/shopper.

[0206] While the grid of products displayed initially seems to include random set of products, these can be strategically chosen by the selection engine 106-1 to maximize exposing the user efficiently to a number of styles, to help the user find visually the closest match to their desired product styles.

[0207] The grid can also be arranged into areas of stylistically similar products, based on category, color, style or other important dimension. This functions as a navigation aid to the user and allows to indicate preferences by scrolling actions.

[0208] As more user preferences are captured by the tracking engine 114-1, the grid will be adapted to contain mostly items sought by or matching the user’s preferences but can also include a number of escape items. Escape items represent a different style and can be used to indicate saturation with current choice and desire to explore radically different options. In some instances, an escape item is not radically different but is merely an item that includes a feature positively correlating with the user’s preferences but not necessarily within the user’s preferences. [0209] Zooming in and out can be performed on the user device 108-1 by a pinch or air gesture or any other method on the user device 108-1, such as a mouse wheel movement, a virtual reality controller, a game controller, and so on.

[0210] At step 210-1, the tracking engine 114-1 of the recommendation server 101-1 tracks user interaction with the large surface to refine user preferences. The tracking engine 114-1 captures activity on the user device 108-1 and populates the user profile repository 110-1. Activity on the user device 108-1 includes how long specific products are displayed on the user device 108-1, zooming in or zooming out on the large surface, clicking on products displayed on the large surface, rating products displayed on the large surface, scrolling away from products displayed on the large surface, hovering over a certain product on the large surface, and so on. Optionally, the tracking engine 114-1 can include an imaging device such as an iris tracking device that is configured to track the user’s irises to correlate the user’s iris positions with corresponding items or areas on the screen to further learn what the user’ s preferences are. This can augment the model’s learning while the user is not selecting or interacting with the screen, but rather scanning the items with their eyes. While a lingering without taking any action could be indicative of an interest or disinterest, iris tracking aids the model in discerning whether the user is interested or disinterested in a particular item. For example, focusing the iris on a particular item or area of the screen can be indicative of an interest in the item, so the lingering can be fed into the model as indicative of an interest level. If the user’s iris is scanning around the entire display area, without focusing on any particular item or area, this can be indicative of a disinterest in any of the items displayed, so the selection engine 106-1 can learn this behavior and replace a significant number or even all of the items shown on the screen even before the shopper makes any input relative to the screen.

[0211] At step 212-1, the selection engine 106-1 of the recommendation server 101-1 rearranges products on the large surface based on the user preferences. In an embodiment, upon clicking a product image via the user device 108-1, a number of product images surrounding the product clicked on will be changed to new product images. The new product images chosen to display match more closely the user’s stylistic preference by analyzing her stylistic preference from micro-actions or activities captured at the large surface (or zoomable grid). The new product images shown to display can also be selected to display a variation of products that help the user further refine their stylistic preference, i.e., to provide additional insight into user preferences. [0212] Steps 210-1 and 212-1 can be performed repeatedly to either refine the selection around a certain visual style by clicking on mostly similar items or expand possibilities and explore by clicking on items along a certain visual direction.

[0213] In an embodiment, the system 100-1 actively learns user preferences. For example, when a user clicks on an item, it typically expresses likability. However, the user may not like the item in its entirety but just some of its features. The item may also be in the desired visual direction but not far enough (or too far). Also a single click on an item may not tell which features the user liked. With some embodiments of the disclosure, desirability of different features of a product can be tested by displaying products emphasizing one or another feature of the liked product and observing further clicks on the user device 108-1.

[0214] In an embodiment, the selection engine 106-1 provides a dynamic map layout of the large surface to the user device 108-1. When the user scrolls the map, newly visible portions of the map are dynamically generated by the selection engine 106-1 based on the information gathered on user intent and preferences so far. The data can include micro-actions and activities of the user, such as direction of ongoing scroll (when items are placed on the map not randomly but based on their stylistic or visual features), zoom-in areas, and others; or actions like previously collected item clicks. Items displayed in the newly generated portions are chosen to be either most pleasing or most informative. Only previously unseen areas are dynamically generated while the areas that the user has seen at sufficiently detailed zoom level will stay unchanged. Dynamic map layout can also be combined with rearranging products as a user clicks or interacts with a product as described above in step 212-1.

[0215] In an embodiment, the tracking engine 114-1, the selection engine 106-1 and the machine learning engine 104-1 cooperate to create a recommender AI. The recommender AI includes a set of algorithms that infer user preferences from a user’s stream of actions and micro-actions during browsing and proposes an optimal layout of products to be shown to the user (both for purchase as well for refining understanding of user preferences). A source of user preferences is the set of fashion features extracted from the products and user’s behavior towards these products. When a user clicks on a product with certain feature set (e.g. Dress, red, Gucci, country-style, bell-shaped etc.), the recommendation server 101-1 strengthens hypothesized user preference towards these features.

[0216] In an implementation, a sequence of interactions, rather than only the last click, is taken into account - for example, persistent selection of a certain feature (clicking dress; dress; dress) can indicate strong preference whereas intermittent selection (clicking dress; trousers; dress) can indicate indifference to that dimension (in the example: category). [0217] In an implementation, negative information can also be taken into account - when a user is shown items with certain features and chooses to ignore these (e.g., not click on, scroll away, zoom out, and so on), the recommendation server 101-1 weakens hypothesized preference towards these features. That is ignoring certain products can be interpreted as a negative preference. Many different such criteria can be used, the above serves as examples of general principle - user’s fashion preferences will be inferred from her sequence of actions towards products with known fashion features.

[0218] FIG. 3-1 illustrates an example interaction on a user device according to an embodiment of the disclosure. In FIG. 3-1, item 302-1 indicates a zoomed out version of products arranged on a large surface. Item 304-1 indicates that the user zooms into a specific area within the large surface to inspect products within that area more closely. Item 306-1 indicates that the user clicks on a specific product, which is interpreted by the recommendation server 101-1 as the user liking the product. Item 308-1 shows that in response to the user clicking on the specific product, neighboring products around the specific product are updated to partially similar products.

[0219] In an implementation, instead of updating neighboring products around the specific product, the recommendation server 101-1 presents the user with a new selection of products in response to the user clicking on the specific product. The new selection of products can replace all products on the large surface. The new selection of products can replace some of the products on the large surface.

[0220] In an implementation, in response to the user clicking on the specific product, the recommendation server 101-1 can append the new selection of products to the large surface, enlarging the large surface such that a total number of products arrange on the large surface is greater than the number before the user clicked on the specific product.

[0221] From item 308-1, the user can continue clicking on products displayed within updated product list, which will allow the recommendation server 101-1 to continue refining the user’s preference. Also the user can choose to buy a certain item at item 306-1. Also the user can zoom out of the specific location on the large surface, which returns the user to item 302-1, indicating to the recommendation server 101-1 that the user is changing directions or looking for a different style.

[0222] FIG. 4-1 illustrates an example system for recommending a product or products to a user according to an embodiment of the disclosure. In FIG. 4-1, a product description (Product D 402-1) which includes an image, a keyword description, and metadata is provided to a neural network 404-1 (e.g., the machine learning engine 104-1). The neural network 404-1 converts the Product D 402-1 to product features (Product F) which includes feature A, feature B, ... feature XYZ. Both the Product D and the Product F are stored in the product database 408-1 (e.g., product data repository 102-1). In an embodiment, each product is linked to its respective description and its respective extracted features and stored in the product database 408-1. For example, PI - Dl, FI indicates that product PI is linked to product description D1 and product features FI, and P2 - D2, F2 indicates that product P2 is linked to product description D2 and product features F2.

[0223] The selection algorithm 410-1 (e.g., implemented by the selection engine 106-1) displays products on a large space, and the user can view the products on a display 412-1 (e.g., a screen of the user device 108-1). The display 412-1 can show the products in a grid. The user can click on a product, e.g., click 414-1 on product P5 presented on the display 412-1. [0224] The user’s click 414-1 as well as products within the product database 408-1 are used, e.g., by the tracking engine 114-1, to update the user’s preferences, storing the preferences in a user profile. Positive preferences and/or negative preferences can be tracked by the tracking engine 114-1. An example of the user’s preferences is provided in user profiling 416-1. The click 414-1 can be interpreted to indicate that one or more features F5 in product P5 is desirable or preferred, one or more features F4 in product P4 is undesirable, one or more features FI 8 in product PI 8 is undesirable, and so on. Preferred features stored in the user profile indicate the user’s preference.

[0225] The tracking engine 114-1 in cooperation with the selection engine 106-1 can then take the information in user profiling 418-1 to adjust certain features in features F5 to depart from features F5. For example, features C, M, X and BL in F5 can be adjusted and stored in the hypotheses 418-1 (e.g., hypothesis testing repository 112-1). Once a next hypothesis is determined, then the selection algorithm 410-1 can find products within the product database 408-1 that meets the features within the next hypothesis. The selection algorithm 410-1 thus updates the displayed products on the display 412-1, and the process continues in an iterative manner.

[0226] FIG. 5-1 shows an example of a mobile device showing different products that can be scrolled through in a vertical and horizontal manner, according to an embodiment of the disclosure. The products are shown on the screen in a zoomed out manner based on the size of the screen.

[0227] FIG. 6-1 shows a zoomed in version of a section of the different products shown in FIG. 5-1. The zoomed in version indicates that clothing with a same user preference style that may look different can be situated next to one another. FIG. 6-1 shows zoomed in view of the scrollable map for closer inspection of interesting items (to understand whether the user likes the item). The map may not replace product detail page where multiple views, brand, price and description are available.

[0228] FIG. 7-1 shows that selecting a certain product updates the products around it to a more similar style according to learned user preferences. Upon clicking an item, product images around it (not limited to just immediate vicinity) change to related items that are either similar to the original item or only partially similar to allow exploration.

[0229] While the examples described herein have been made with reference to fashion and design, the concepts disclosed herein are equally applicable to any other highly subjective and personalized area including interior design, home decor and furnishings, floor plan layouts, landscaping, works of art, architecture, or any area where the subjective user’s “look and feel” are invoked. An emphasis is placed on areas where a purchasing selection is based largely upon visual cues and a visual comparison of multiple possibilities arranged in the same visual display, where the options to be selected are iterated to become relevant to the individual purchaser’s subjective tastes and preferences, which defy conventional categorization or tagging.

[0230] While the present disclosure has been described with reference to one or more particular embodiments and implementations, those skilled in the art will recognize that many changes may be made thereto without departing from the spirit and scope of the present disclosure. Each of these embodiments and implementations and obvious variations thereof is contemplated as falling within the spirit and scope of the present disclosure, which is set forth in the claims that follow.