A COMPUTER IMPLEMENTED PROCESS FOR ANALYSING FOOD PRODUCTS

FIELD OF THE INVENTION

This invention relates to a computer implemented process for analysing food products; it is used in an online grocery app or website to enable consumers to purchase healthier foods.

DESCRIPTION OF THE PRIOR ART

Online grocery apps or website are ubiquitous, and provide simple filtering tools to enable consumers to select products that meet their dietary needs. But these tools are generally inaccurate and lead to consumer frustration.

SUMMARY OF THE INVENTION

The invention is a computer implemented process for analysing food products, including the steps:

(a) automatically ingesting and analysing food product packaging data for a food product; and

(b) using the analysed data as an input to a process that automatically assesses whether the food product matches or conforms to dietary attributes, where the dietary attributes define properties of a food product that are relevant to a customer’s food preferences, or requirements.

Further details are in the appended claims.

BRIEF DESCRIPTION OF THE FIGURES

The invention will be described with reference to the accompanying Figures, which relate to an implementation of the invention called Health Hub.

Figure 1 shows the Health Hub process flow when setting up an online grocery.

Figure 2 shows the Health Hub process flow for a consumer’s shopping experience at the online grocery.

Figure 3 shows the Health Hub mobile app, running on a smartphone, and showing how a consumer selects allergies as some of the dietary attributes that the Health Hub system will use for personalised food recommendations and other purposes.

Figure 4 shows how the Health Hub system has automatically recommended a personalised goal, shown on the mobile app, for this consumer (‘Buy less added sugar’), using an AI-based analysis of the consumer’s previous shopping baskets.

Figure 5 shows how the Health Hub website displays products when the consumer has searched for ‘ketchup’. The first two options are reduced sugar options, relevant to this consumer since they have previously selected the ‘Buy less added sugar’ goal. In the top left of the two first options is a small ‘Health+’ icon, showing that these products each meet that goal. The Health Hub system only recommends products that comply with the customer’s dietary attributes (allergies etc.).

Figure 6 shows how the Health Hub website enables a consumer to filter and search for products against their active goals or other new goals if they wish; in this case, the manual filter option ‘No added sugar’ has been selected.

Figure 7 shows how the Health Hub website enables the consumer to select personalised curated products that match chosen goals, such as ‘Buy less added sugar’. Figure 8 shows how the Health Hub website provides additional recommendations to improve the health and sustainability of the consumer’s basket.

Figure 9 shows how the Health Hub mobile app automatically displays a celebration message when a customer achieves their goals.

Figure 10 shows how the Health Hub mobile app automatically displays insights into the customer’s shopping (e.g. ‘15% increase in fruit bought’).

Figure 11 , 12 and 13 are further examples of how the Health Hub mobile app automatically displays insights into the customer’s shopping.

Figure 14 shows how the Health Hub mobile app gives an overall single ‘Basket Health’ score (in this case 26%), plus recommendations for products (in this case, various types of wholegrains) that will increase that score and that comply with the customer’s dietary attributes and chosen goals.

Figure 15 shows how the Health Hub mobile app automatically recommends products that are appropriate to the customer’s goals.

Figures 16, 17 and 18 shows how the Health Hub mobile app displays a customer’s rewards points and related discounts towards products that support their goals (Figure 16), milestones reached, with badges that celebrate progress (Figure 17) and also compete in challenges shown on a leader board (Figure 18) anonymously for a limited timeframe.

Figure 19 and 20 shows how the Health Hub mobile app incentivises customers with motivational messages (Figure 19) and with relevant products and suitable recipes (Figure 20) to help them achieve their goals.

Figure 21 shows how the Health Hub website displays a customer’s food related goals, milestones, insights and rewards. DETAILED DESCRIPTION

One implementation of the invention is called the Health Hub system; this section will give more details on this system. Health Hub encourages consumers to seek out healthier and more sustainable food products; Health Hub enables retailers to meet the increasing regulatory and shareholder pressure to support more sustainable food supply chains and guide consumers to find and purchase healthier and more sustainable food choices. Using personalised shopping goals, insights, behavioural science, gamification and rewards, the Health Hub solution helps to motivate and reward customers every step of the way. Health Hub is an integrated omni channel experience (including both application functionality designed to be integrated into an online grocer’s grocery ordering app and website) to nudge shoppers into making healthier choices over time through accurate search and discovery, curated product recommendations and healthier alternatives, as well as insights on the overall shop/basket, ultimately encouraging a sustainable behaviour change that lasts.

The emergence of apps as a new way of intervention by combining innovative technology and basic health information is a significant trend; studies have suggested population health outcomes can be improved through the reduction of diet and lifestyle-related NDCs being diagnosed (Hingle and Patrick, 2016). Together with integrated behaviour change techniques, ‘goal planning’, ‘feedback monitoring’, ‘shaping knowledge’ and ‘social support’ studies have shown health related apps can have long term engagement possibilities (Villinger et al, 2019). Health Hub not only supports the consumer through their journey of health, but also incorporates behaviour change interventions, adding enormous value to the retailer to help drive positive influence and build trust.

The aim of Health Hub is to be the bridge between a food retailer’s commercial value driven targets and a consumer’s food purchasing actions, translating those purchasing actions into healthier, more sustainable habits and a more engaged consumer. Apps developed to help population groups make healthier and more sustainable choices have proven to be a cost effective and feasible tool by adding a feeling of connectivity and ‘real life’ (Villinger et al, 2019). Spoon Guru takes it a step further and connect the retailer directly with the consumer to build trust and engagement: References

1. Hingle, M. and Patrick, H., 2016. There Are Thousands o/Apps for That: Navigating

Mobile Technology for Nutrition Education and Behavior [online] Society for Nutrition Education and Behavior. Available at:

<https://pubmed.ncbi.nlm.nih.gov/26965099/> [Accessed 9 June 2021]

2. Vilbnger, K., Wahl, D., Boeing, H., Schupp, H. and Renner, B., 2019. The effectiveness of app-based mobile interventions on nutrition behaviours and nutrition-related health outcomes: A systematic review and meta-analysis [online] Wiley Obesity Reviews. Available at:

<https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6852183/ ?log$=activity> [Accessed 8 June 2021]

3. Schiro, J., Shan, L., Tatlow-Golden, M., Li, C. and Wall, P., 2020. Miealthy: smart digital food safety and nutrition communication strategies a critical commentary. [online] Nature - Science of Food. Available at: <https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7530665/pdf /41538_2020_Article_ 74.pdf> [Accessed 11 June 2021]

We will now explain how the Health Hub system works in practice. High level process flows of one implementation are shown in Figure 1 (the Health Hub process flow when setting up an online grocery and Figure 2 (the Health Hub process flow for a consumer’s shopping experience at the online grocery).

Figure 1: Setting up Health Hub for an online grocery:

• For every product, packaging data is extracted to identify ingredients and nutrition statements.

• Packaging data is normalized to a standardized model using logic, ML, metadata addition.

• Dietary assertions are generated for all products (factual statements about a product that conform to a model, schema or ontology). Based on filter processing or AI based analysis of the packaging data. • Complete list of all dietary attributes (TAGs) is generated - e.g. all allergies, all diet types etc.

• For each product, all dietary assertions are evaluated against each dietary attribute using a dietary rules engine, generating a table that logically and consistently describes each product in terms of all parameters of relevance to consumers.

Figure 2: A Consumer’s shopping experience at the online grocery:

• Consumer defines their personal dietary attributes when on-boarding.

• Online grocery store analyses shopping history to proactively identify (using ML) and suggest personalized Goals, e.g. Buy less sugar. Consumer can also choose Goals from a menu.

• Online grocery store proactively recommends healthier replacements for products regularly bought in the past or in the current basket.

• Consumer searches for product types and is only returned products that comply with the consumer’s personal dietary attributes. List is both more accurate and also more expansive than a conventional online grocery filter-based search.

• Online grocery store tracks the consumer’s shopping baskets over time and gives insights on progress towards Goals, and profiles the nutritional balance of recent baskets.

We will look now at the user experience, showing app and website screens from an online grocery store that implements the invention. After the consumer or customer is on-boarded onto an online grocery store’s app or website that implements Health Hub, the consumer will define (online, via the website or app) the dietary preferences for each family member, as shown in Figure 3.

Food or nutrition ‘Goals’ are then defined by the consumer, as shown in Figure 4, where the goal is to ‘Buy less added sugar’. To encourage selection of this goal, the user interface shows also that 42,122 others have chosen this goal (the system tracks this data); it explains that 2X less added sugar and 2.5X increase in heart health is associated with meeting this goal. More detail on what we mean by ‘Goals’: the system presents personalised nutrition goals to motivate each customer to improve their choices; to do this, the system assesses customer behaviour and food choices - e.g. what has been historically in their online food basket and what is in their current online basket. Understanding the macro and micronutrients in the customer’s baskets is important context for customers to find healthier choices when they shop. The system helps customers by analysing their historic baskets and proactively suggesting the most relevant nutrient based goals for that customer.

The following personalised macro and micro nutrient goals are available for recommendation:

• Buy less sodium

• Buy less sugar

• Buy less added sugar

• Buy less saturated fat

• Buy less total fat

• Buy more lower cholesterol

• Buy more protein

• Buy more fibre (must also not be above x% total sugar)

• Buy more Vitamin D (must also not be above x% total fat)

• Buy more calcium (must also not be above x% total sugar)

In order for the customer to receive a personalised nutrition based goal, we assess the amount of each nutrient per lOOg as a uniform measure within each product across all shopping baskets they have purchased over the past 52 weeks. We compare this analysis against all other customers in the same timeframe to create a benchmark which is used to trigger each nutrient based goal. For example, if the customer’s “added sugar” nutrient levels is above the calculated benchmark, the goal is triggered and proactively presented to the customer as an option to try. Each goal will include context on how many customers have chosen each goal, the impact it typically has and any charity partner context on health benefits. Customers can also view popular trending goals or create their own goals.

The system can also present the following personalised food groups for recommendation to a customer:

• Buy more fruit • Buy more vegetables (excludes legumes)

• Buy more / less dairy

• Buy more lean protein

• Buy more whole grains

• Buy less refined grains

• Buy more enriched refined grains

• Buy less red meat

• Buy more plant rich protein (includes legumes)

We go further than the top level Eatwell groups (fruit, vegetables, grains, protein and dairy fruit & vegetables, starchy carbohydrates, proteins, dairy & alternatives, oils & spreads; Eatwell is a UK government initiative to define healthy eating) by understanding the types of grain and protein based on the guidance. For example, it is better to base your grains % against wholegrains and not refined ones. In order for the customer to receive a personalised goal we assess the % of each top level food group as a uniform measure across all shopping baskets over the past 52 weeks. We use ML to classify compound products such as pizza across all groups to ensure we understand the correct proportions. We compare each customer against a benchmark for the top level groups. If the customer is above/below the benchmark (depending on the group) the goal is triggered and presented to the customer.

The red meat, lean protein, whole grains, refined grains and plant rich protein goals are assessed by looking at the proportion of each top level group % per customer across all shopping baskets over the past 52 weeks. For example, the % of red meat for the protein group is assessed for each customer and compared to all other customers to find the benchmark. The benchmark may be 10% of protein within the overall protein group (20% of the plate) is lean protein so a customer would trigger buy more lean protein if they were under the 10% threshold.

The system can also present the following personalised sustainability goals for recommendation to a customer:

• Buy more organic

• Buy more sustainably sourced fish

• Buy less red meat

• Buy more plant rich protein (includes legumes) • Buy more locally sourced

• Buy more recyclable packaged products

• Buy more higher welfare animal products

• Buy more GMO free

In order for the customer to receive a personalised sustainability based goal we assess the number of products that have sustainability certifications across all shopping baskets over the past 52 weeks. We then compare this measure against all other customers to create a benchmark. If the customer is above the benchmark the goal is triggered and presented to the customer. The buy less red meat and buy more plant rich looks at the % of each across all shopping baskets over the last 52 weeks. We then compare this measure against all other customers to create a benchmark. If the customer is above the benchmark the goal is triggered and presented to the customer.

The Health Hub system can surface the following insights for each goal, tracking how the consumer’s food purchasing behaviour has altered:

Nutrition Goals: We calculate the relative difference in the amount of a nutrient per lOOg over 52 weeks and compare that to the recent 8 weeks to see the proportionate change. For example, I’ve gone from an average of 10 teaspoons of sugar per basket to 9, so reduced by 10%. We show the relative difference in the % of each food group over 52 weeks and compare that to the recent 8 weeks to see the proportionate change. For example, I was buying 20% of vegetables and now buying 25% so an increase of 25% but the target is 40%.

Sustainability Goals: We show the relative difference in the amount number of sustainable products over 52 weeks and compare that to the recent 8 weeks to see the proportionate change. For example, I was buying 8 sustainable products and now buying 10 per basket to an increase of 25%.

For all Goals, we show the number of shopping trips since the start of each goal and how the customer is trending over time. Using AI we assess the active goals for each customer and define proprietary customer segmentations and ML models to match the most relevance products against their shopping behaviour. Every recommendation needs to match product criteria using regulated category specific nutrient thresholds to ensure the products we suggest are better for the customer. We also nudge customers to purchase healthier and more sustainable products against the customers favourites, as they browse and also post shopping trip to further incentivise them.

To recap, there are several approaches to goal selection. The consumer can choose their own food Goals (e.g. “Buy less added sugar”; “Buy more vegetables” etc.) from a pick list displayed on the app/website; this list can be set by the retailer; trending goals (e.g. goals that are proving very popular at the moment) can be flagged or prioritised.

Alternatively, as explained in detail above, the Health Hub back end system can automatically suggest appropriate goals by analysing that consumer’s previous shopping baskets with that retailer, comparing various parameters (e.g. sugar content) to average shopping baskets or accepted benchmark levels, and where the consumer’s historic shopping baskets indicate that they would benefit in health terms from some modification to their normal purchasing behaviour, the system can automatically propose that as a goal. For example, where a consumer purchases an above average (or above nutritionally accepted guidelines) amount of sugar, then the system can automatically ask the consumer if a goal to consider might be to ‘Buy less sugar’. The system can use nudge techniques, such as stating that ’41,122 consumers are currently using this goal’ or similar, to make that goal more appealing. If the consumer agrees with that suggestion, the system will automatically propose reduced sugar equivalents of the consumer’s normal purchases.

During online shopping, the system may proactively propose other, different food items which, through its machine-learning based tracking of potentially hundreds of thousands of other consumers food shopping habits, the system considers will be enjoyed and repeat purchased by the consumer and will help the consumer meet their goals. The consumer can also simply manually filter food products to meet the goal or goals.

When checking out, the system automatically suggests additional products which its data suggests will help the consumer reach and maintain its goals. Over future use and repeated purchases, the system tracks whether the consumer has in fact met their goals, and gives them data to show if they have (e.g. ‘Congratulations; your sugar consumption has reduced by 12% over the last 4 months’). Other gamification techniques can be used, such as leader boards (e.g. who has most reduced their sugar consumption).

We will return now to the Figures. When the customer shops online, the retailer can signpost relevant product recommendations that are aligned to their active goals. For example, as shown in Figure 5, the customer searches for “ketchup” and the results page clearly flags the two products which are best suited to support their active goal: in this case, the first two are flagged with a ‘Health +’ icon: these are both reduced sugar options, relevant to this consumer since they have previously selected the ‘Buy less added sugar’ goal min addition, the customer can view how many products in their current basket align to the dietary guidance: the bottom right shows a graphic that represents the amount of vegetables, grains, fruit, protein and dairy in a curved graphic in the consumer’s basket (or average of recent baskets) the length of the curved line progressively increases towards the target, so it is very clear if for example a specific nutrient type (e.g. dairy or oil and fats) is approaching the recommended limit for healthy eating; the progressive increase in the length of a curved graphic towards a target is known from other different non-nutrition contexts, such as fitness trackers.

The system surfaces or displays dietary guideline insights on shopping behaviour to motivate each customer to help improve their food choices over time. This makes dietary guidelines such as Eatwell come to life for the customer, as well as making it easy to relate to in a real-life setting. Using Eatwell dietary guidelines, we summarise how customers’ purchases match the recommended guidance. We track proportionate changes of each food group overtime as we recognise customers will not always follow the guidance every time they shop. It is important to not negatively highlight specific unhealthy behaviour for specific baskets, but celebrate the positive improvements each customer has made over a longer timeframe to isolate their typical shopping behaviour. We also recognise customers will take their own unique path to improve the health of their basket purchases and sharing incentives and insights is proven to help sustain the change. We calculate the relative difference across each food group over 52 weeks and compare this recent shopping trips to display their proportionate change in shopping habits. For example, this customer’s baskets typically included 26% of grains across the products they buy and it used to be 22%. They are 4% short of the guidance so doing very well. Customers can select a specific food group and understand products they frequently buy to support this food group, products they’ve started buying and also view recommendations to further improve their food purchases.

Figure 6 shows how the customer can also filter and search for products against their active goals or other new goals if they wish; in this case, manual filter options are no added sugar; contains fruit; contains vegetable; wholegrain; plant-based; recyclable; organic.

Figure 7 shows how the system supports the creation of highly personalised curated products that match their active goals or other goals. This is to ensure that the system make the discovery of healthier and sustainable products as easy and seamless as possible; in this case, the system displays: Top picks; Buy less added sugar; buy more vegetables; buy more plant-based; buy less sodium.

Figure 8 shows that when the customer is ready to start the checkout process, the system suggests additional recommendations to improve the health and sustainability of their basket: in this case, one goal is ‘Buy less salt’ and the system has automatically recommended three no salt or low sodium products; another goal is ‘Buy more plant-based’ and the system has recommended three suitable products. In each case, the consumer can readily add the item to their shopping basket by selecting the ‘Add’ button. Further, the system can also highlight products which the customer has actually placed in their basket, and suggest a replacement that better meets their health goals. A ‘Replace’ button is then present. If they do select ‘replace’, they can see the circular graphic in the bottom right alter: for example, replacing say a milk shake with an almond-milk shake will see the curved line associated with diary recede in length. This is an opportunity to further nudge and incentivise customers when they shop.

Figure 9 show how the system automatically celebrates success when customers achieve their goals, notifying the customer of their achievement (“Congratulations, you’ve achieved your 60 day goal Buy less sodium”) and giving them (Figure 10) simple insights that help the customer track their progress and stay motivated, especially when not shopping; in this case, the mobile app displays the messages: ‘ 15% increase of fruit bought’ ; ‘2% wholegrain increase last shop’ ; ‘4/7 challenges won’; ‘87% Average % of Good For You products’. Figures 11, 12 and 13: When reviewing progress against each specific goal, the system shares a summary of their progress after each shop and how behaviour is changing over time. For example, the ‘Buy less sodium’ goal shows (Figure 11) that there has been a 2% sodium reduction, equivalent to 4 teaspoons of salt, over the course of 10 shopping baskets. The consumption of sodium over successive shopping trips is shown in a bar graph, so that the customer gets an easy to understand graphic showing sodium use. In Figure 12, the ‘Buy less added sugar’ goal shows in a bar chart how sugar consumption is dropping over successive baskets. Figure 13 is a high level view on how their shopping behaviour compares to the standardised ‘EatwelT dietary guidelines.

Figure 14 and 15: The system gives an overall single ‘Basket Health’ score (in this case 26%) with the most recent trend (here, -10%). The system automatically analyses the customer’s recent baskets to understand what suggestions are appropriate, and what suggestions are most likely to be taken up (based on its data of how previous customers with similar goals and similar food choices have reacted). In Figures 14 and 15, the system recommends more wholegrains, and suggests suitable products. Customers can drill into more details behind each ‘EatwelT food group to understand how their purchases are broadly changing overtime. The system provides new recommendations to help the customer remain engaged and further improve their food choices: It is important to support engagement every step of the way.

Figures 16, 17 and 18: Rewarding progress and making the program fun is key to the success of the system. The customer can view their rewards, such as rewards points and related discounts towards products that support their goals (Figure 16), milestones reached, with badges that celebrate progress (Figure 17) and also compete in challenges shown on a leader board (Figure 18) anonymously for a limited timeframe.

Figure 19 and 20: After the customer has completed their activation of new goals, we incentivise them with motivational messages (Figure 19) and with relevant products and suitable recipes (Figure 20) to help them achieve their goals.

Figure 21 shows how the desktop, browser based insights are displayed: the system is an omni channel experience where the customer can view all their goals, milestones, insights and rewards in a larger desktop or laptop view, in addition to email and app and instore experiences. Moving on from the examples shown in the Figures, the system functionality can be organised as follows:

Features (core solutions and associated products)

1. Customer Needs Management a. Customer Profile Setup b. Health Hub opt in / out c. Goal Setup

2. Personalized Product Discovery through a. TAGs (allergy, diet, nutrition, sustainability, b. Good For You Badge (Name TBD)

3. Personalized Recipe Discovery through a. TAGs (allergy, diet, nutrition)

4. Intelligent Curation through a. Smart Swaps (substitution)

■ Frequently bought items

■ Flagging substitute products in search and browse pages

■ Active Basket substitutes b. Product Recommendations

■ Curated lists to meet your goals

■ Flagging products that meet your goals in search and browse pages c. Contextual Basket Recommendations

■ Add to basket recommender (to meet your goals) d. Goal Predictor

■ Analyse Baskets to identify personalised goals

5. Now & Next Dietary Insights through a. Retailer Insights

■ Health Hub Performance

• Track Basket Health and Sustainability Scores for all customers (are customers sustaining their change in behaviour) • Understand programme engagement, e.g., number of active goals and associated customers

• Health and Sustainability Customer Segmentations b. Personalised Shopper Insights

■ Goal tracking (after every order)

■ Basket insights (anytime)

Capabilities (how the solutions manifest for customers to drive healthier online grocery food baskets)

1. Customer Opt in / opt out to the programme

2. Customer Profde setup (allergies, lifestyle diet, macronutrient and sustainability)

3. Health Hub (dedicated area to manage goals and view insights) a. Goal Setup i. Personalised Goal Suggestions based on your historic purchases (opt in) ii. Customer defines their own goals from standard list (e.g. more fruit, more veg, more plant based, less salt, less sugar, less fat etc) b. Manage & Maintain Goals i. View Active Goals and time remaining c. Insights (all baskets since programme opt in and post order) i. Consumer facing insights on your specific basket (post order) ii. Historic trends across basket health and sustainability scores and benchmarking iii. Top Line Insights on your purchases e.g., YOY changes to basket health and sustainability scores d. Incentivize/Reward i. Share goals, progress towards goals, completed goals, rewards ii. Invite friends and family to take part iii. Post shop incentives to support hitting your goal (push notifications, emails) to utilise in your next shopping trip

4. Personalization (when Shopping) a. Curated Personalised Product recommendations (to meet your goals) b. Flagging healthier and sustainable substitutes i. Products you normally buy ii. Browsing pages iii. Search Results c. Personalised Basket Recommendations i. Missing products to hit your goal ii. Healthier and more sustainable alternatives d. Flagging products that meet your broader profile (not goals)

5. Retailer Insights (Profiling solution performance) a. Track Basket Health and Sustainability Scores for all customers b. Understand number of active goals and associated customers c. Health and Sustainability Customer Segmentations

Core Technology Modules

The core technology modules underpinning the solution are outlined here. The modules are detailed later in the document.

Personalized Product Discovery

• Product TAGs (allergy, diet, nutrition, sustainability)

• Health Score

Personalized Recipe Discovery

• Recipe TAGs

• Recipe Nutritional Data

Intelligent Curation

• Smart Swaps (substitution)

• Product Recommendations

• Contextual Basket Recommendations • Goal Predictor

Now & Next Dietary Insights

• Basket Insights and Analyser

Technical Details for Health Hub Modules

Product TAGs

The TAGs system refers to a set of technologies developed to assist users in finding and identifying products that are compatible with their dietary needs. The related app can then make suitable product recommendations: this is in practice can be very helpful to ordinary consumers when the app is part of a food retailer’s app: e.g. the functionality is an integrated part of an online grocery purchasing/delivery app.

The system accurately applies dietary attributes (TAGs) such potential allergies, diet types, nutrition/goals, and sustainability requirements, to a consumer’s chosen food products (e.g. those available from the online grocery store) by interpreting the data listed on the food packaging.

TAGs are binary and indicate suitability or lack of suitability for a particular dietary attribute - for example, “Vegan”, “Organic” or “Sesame Seeds”.

Dietary attributes include the following parameters:

Allergy:

Peanuts

Eggs

Tree nuts

Shellfish

Wheat

Soy Gluten

Lupin

Mustard

Sesame

Sulphites

Celery

Milk

Diet:

Vegetarian

Vegan

Lactose free

Gluten free

Paleo

Keto

Low carb

Low cholesterol

Low fat

Fruitarian

Dukan

Atkins

Diabetic

Kosher

Halal

Nutrition: sodium sugar added sugar saturated fat total fat lower cholesterol protein fibre vitamin D calcium

Sustainability: organic sustainably sourced fish red meat plant rich protein (includes legumes) recyclable packaged products higher welfare animal products GMO free

TAGs are necessary when suggesting goals to a customer and in evaluating to what extent the customer is achieving those goals. Example goals include the following:

Nutrition/nutritional goals:

Buy less sodium

Buy less sugar

Buy less added sugar

Buy less saturated fat

Buy less total fat

Buy more lower cholesterol

Buy more protein

Buy more fibre (must also not be above x% total sugar)

Buy more Vitamin D (must also not be above x% total fat)

Buy more calcium (must also not be above x% total sugar)

Sustainability goals:

Buy more organic

Buy more sustainably sourced fish Buy less red meat

Buy more plant rich protein (includes legumes)

Buy more locally sourced Buy more recyclable packaged products Buy more higher welfare animal products Buy more GMO free

The food product packaging data includes but is not limited to the product title/name, the product description, the ingredients, the nutritional information and packaging statements including allergen warnings and marketing material. It also includes data that is not literally present on the packaging for an item of food, but can otherwise be linked to that food product (e.g. the product supplier’s website or social media feed; trusted social media feeds from reviewers and others).

The system receives product packaging data in all standard data formats (e.g. XML, JSON, CSV) and delivery mechanisms (e.g. API, SFTP) from external data sources, including retailer data feeds and third-party product data aggregators. The data is subsequently parsed, extracted, cleaned and normalised: product packaging data has no specified format and therefore is normalised to adhere to the generic Spoon Guru product model, including (but not limited to) a product title, description, ingredients, packaging statements, and nutrients.

We can generalise to:

A computer implemented system configured to automatically ingest food product packaging data for different food products from multiple different sources and normalise that ingested data to a standardised food product model.

This process involves logic-based rulesets, natural language processing (NLP) and machine learning for intelligent parsing and adding metadata (for example, for reliably extracting ingredients from statements using named entity recognition models).

We can generalise to: A computer implemented process for analysing food products, including the steps:

(a) automatically ingesting and analysing food product packaging data for different food products; and

(b) using the analysed data as an input to a process that automatically assesses whether the food product matches or conforms to food attributes for a specific product, where the food attributes define a consumer’s food preferences, or requirements.

Further machine learning and rule-based processes are applied to generate dietary “assertions” for each product. These processes use Spoon Guru's dietary configuration data as input. Spoon Guru configuration data is multifaceted and has been collected over years of nutritional review and data quality assurance. It includes ingredients linked to attributes, tokenization rules and data cleaning logic. Each product will have many dietary assertions. They are created by 3 key processes:

1. Titles, descriptions and ingredients are processed by filter processing, which checks each entity for the presence of Spoon Guru ingredients from its internal dictionary and applies assertions accordingly. An example of an ingredients assertion is:

"Filters: Ingredient 'chicken breast' matches Spoon Guru Ingredient 'chicken'"

2. Product packaging statements are processed by deep learning models, the prediction output of which is formed into a statement assertion, e.g.:

"Machine Learning: Statement Contains Wheat was mapped to “contains - wheat"

3. The products nutrient values are normalised to per serving & per lOOg/ml values and nutrient assertions are formed indicating suitability for nutrient-based diets.

Once all assertions are created for a product, these are evaluated holistically to conclude with a positive or negative attribution for each TAG.

We can generalise to:

A computer implemented system configured to implement machine learning and rule-based processes to generate multiple assertions for a food product, each assertion defining a statement that applies to the food product, the statement conforming to predefined rules or syntax, and the complete set of assertions for a product defining, in a standardised way, the product to enable a computer implemented process to determine if the product is compatible with one or more food attributes, where the food attributes define possible consumer preferences or requirements (e.g. in respect of personal allergy, diet, nutrition, and sustainability requirements).

For example, if we take a product (e.g. a loaf of wholemeal bread made with wheat, water, flour and yeast), then the allergy TAGS, the assertions + attributes for each TAG might look like this for this wholemeal bread product:

The Diet tags for this wholemeal bread product could look like:

A holistic approach is required here to deal with data ambiguities. For example, the ingredient ‘lecithin’ can be produced from milk or soy - combined with a certified statement saying ‘Free from Soy’, we can infer that it is produced from milk for this product. The process of combining assertions according to complex, extensive and sophisticated rules is enabled by a core piece of in-house technology termed the ‘Dietary Rules Engine’. This is an application with a front end interface that allows configuration of extremely complex logic and exclusion criteria by creating and associating rules with the assertion metadata detailed in the previous step.

This process is repeated for all products and all TAGs until each product has a full set of TAGs.

Once core system processing is finalised, products are passed through Quality Assurance (QA) processes to identify potential issues - each of the detected issues is termed a ‘Quality Assurance object’, and requires manual processing to resolve. This consists of a series of automated checks to identify areas where the Spoon Guru decision on a particular tag has a higher than average likelihood of being incorrect. Any potential issues are surfaced in the Quality Assurance console, an interactive web application that enables nutritionists to analyse and review issues. Any areas where the core Spoon Guru system is incorrect are fixed by editing the Spoon Guru configuration. Once the system output matches the expected output, the issue is marked as resolved and is no longer surfaced to the nutritionists. As such, the system improves daily in small increments and allows the building of complex dietary relationships and very high accuracy across attributes. The core Spoon Guru system itself is reliant on an iterative and cyclical quality assurance process to improve the quality and accuracy of its TAGs.

For the efficiency of the overall system, products are monitored from one system run to the next. A product can be in one or more of the following states in this context:

1. New (a new product that previously did not exist in the system)

2. Changed (a product that existed previously, but the data sent by the client has changed in some meaningful way- for example, the ingredients have altered)

3. Differential (the products TAGs have changed in some way from the previous processing run)

4. Unchanged (the product existed before and is completely unchanged)

All products in any of the first 3 states are passed through the automated checks:

1. For each TAG (e.g. ‘vegan’, or ‘organic’), Spoon Guru trains and maintains a machine learning model that predicts that TAG attribute for each product. These predictions are compared to the Spoon Guru attribute value post-processing and any inconsistencies are flagged for review.

2. Products are passed through a series of logic-based checks that have been built up over time and in response to specific areas that may require human attention.

Spoon Guru also processes data in multiple languages. To achieve this, product data/metadata is sent to the Google Cloud Translation API in the original language and loaded into the system in English for processing. Data in both the original language and translated English are displayed side by side in the Quality Assurance console, and quality assurance is prioritised towards native speakers to detect errors in translation - incorrect translations are overridden manually via the console.lslpj

Health Score

The Spoon Guru Health Score uses a product’s or recipe’s data (nutritional information and ingredients) in order to calculate a nutritional score for a given product or recipe. The score is based upon the Nutrient Profiling model used created by the Food Standards Agency (FSA) in 2004-2005 as a tool to help Ofcom differentiate foods and improve the balance of television advertising to children. The Spoon Guru Health Score provides a tool that can be displayed to consumers to help understand or compare products or recipes and help decision making when finding suitable products online. Additionally it is used in the Smart Swaps engine (see above) or can be used by retailers to understand and segment customer behaviour to improve personalisation.

The Health Score is calculated by scoring each of the negative nutrients (saturated fat, sugar, sodium, total energy (kJ)) and positive nutrients (protein, fibre) and assessing the total fruit, vegetable or nut content of the product or recipe. Fruit, vegetable or nut content of the product or recipe is estimated using machine learning. These individual scores are then combined and normalised to provide the final score. This is then provided to the retailer via a web API.

We can generalise to:

A computer implemented system configured to generate a single score or rating for a food product, based on: (i) scoring each of the negative nutrients (saturated fat, sugar, sodium, total energy (kJ)) and positive nutrients (protein, fibre) and assessing the total fruit, vegetable or nut content of the product or recipe; and (ii) combining and normalising these scores to provide the final single score.

Recipe TAGs

The Spoon Guru Recipe Engine uses aggregated product data from a list of shoppable recipe ingredients to calculate Product TAGs (see above) and a full Nutrient panel for a recipe.

Each recipe can have many dietary assertions. These are created by two key processes:

1. The recipe nutrient values are used to form nutrient assertions indicating suitability for nutrient-based diets.

2. Product dietary assertions are aggregated across each recipe ingredient to build up a set of assertions indicating the suitability for additional diets such as “vegan” or “gluten free”. Recipe Nutritional Data

The Recipe nutrient calculation uses recipe data (ingredient quantities and the recipe serving size) combined with product nutrient data for each ingredient, as well as other ingredient metadata such as volume to weight conversions sourced from the USDA database of foods.

The recipe nutrient panel is calculated in the following way: The amount (in grams) of an ingredient is calculated by combining the quantity provided in the recipe with additional metadata, such as volume to weight conversions (sourced from the USDA database of foods). The nutrient profile of an ingredient is either provided by product data, or sourced from the USDA database of food if unavailable in product data. The ingredient quantity is combined with its nutrient profile to find the total nutrient contribution for the ingredient. These values are then aggregated across the entire recipe to calculate the overall nutrient breakdown.

Once we have the total nutrient panel for the entire recipe, we then normalise the panel according to the following:

1. Recipe panel per serving, using the recipe serving size

2. Recipe panel as a percentage of recommended daily intake, suing the recipe panel per serving and the recommended daily allowance for each nutrient (based on government guidelines)

3. Recipe panel per lOOg, using the total raw weight of the recipe which we find by aggregating the ingredient quantities across the whole recipe.

Smart Swaps

The Spoon Guru Smart Swaps engine was developed to assist people with the task of finding altemative/substitute products, given an original source product, which are compatible with their needs (including dietary preferences, allergies, health preferences and requirements). The goal is to surface personalised recommendations to consumers at various touch points throughout the customer’s journey on a retailer’s website.

The engine initially computes a similarity index between the source product and all other products within the catalogue using a multiplicity of product level features (ingredients, packaging statements, description, weight, pack size, price, brand, commercial taxonomy and classification), additional derived features (flavours, similar ingredients) and customer interaction data (clicks, adds to basket). The similarity index and derived features are computed using a combination of rule based, natural language processing and machine learning techniques (including named entity recognition, vector modelling and collaborative filtering).

Once similar products have been generated, they are passed through a set of filtering based upon pre-configured rules which may be different for each installation. For example, if smart swaps are surfacing healthier products as recommendations, the similar products are assessed using the Spoon Guru Nutrition Score (see below) to ensure they meet a minimum Nutrition Score threshold in order to be considered as a suitable swap. Additional commercial rules may be applied here including price, pack size, brand or category constraints.

Eligible swaps are finally reviewed via semi-automated quality assurance processes by qualified nutritionists. If all checks are passed they are activated and will be served back to the client via a web API.

Product Recommendations

This module provides tailored recommendations of suggested products for a given user via a web accessible API. These can then be displayed as part of curated, personalised lists, or as recommendations displayed in containers throughout the search and browse journey on a retailer website or application.

The product recommendations are based upon the users previous purchases and shopping behaviour, which allows Spoon Guru to harness machine learning driven techniques (for example, collaborative filtering) to identify products that they are likely to engage with and/or purchase. This includes products that are purchased by other, highly similar users and products that have the highest predicted probability of purchase based upon the available data upon the users shopping behaviour and purchase history. One use case is that the product recommendations are enhanced by incorporating information regarding the user’s explicitly stated or predicted goals (see goal predictor module), using the TAGs and Health Score modules previously described, in order to output recommendations to the user that they are not only likely to purchase, but are also aligned with their personal health and dietary goals. We can generalise to:

A computer implemented online grocery system configured to automatically identify specific food products that meet a consumer’s self-defined food attributes, being their food preferences or requirements (e.g. in respect of personal allergy, diet, nutrition, goals, and sustainability requirements) and make recommendations based on a ML-based analysis of these identified food products, and previous purchases.

Contextual Basket Recommendations

This module is similar to the product recommendations module described above, however it additionally takes into account the context provided by the state and contents of the current, in session shopping basket for the user. As such, it provides contextual basket recommendations that are aligned with the users health and dietary goals and which they are predicted to be likely to engage with and purchase based upon their current basket.

Basket Insights and Analyser

This dietary -based retail basket analysis module evaluates a user’s current basket and previous shopping history across a variety of dietary, health and sustainability attributes, including nutrients (e.g. sodium, fibre), proportions of dietary categories (e.g. fruit, vegetable or nut health, plant based, meat) and scores (e.g. health sore, see above). This relies on the TAGs and Health Score modules described previously to provide a dietary lens upon a user's shopping basket.

The analysis is subsequently used to segment customers into different, homogenous cohorts by dietary preferences and to serve customers tailored insights as to how they compare against their stated or predicted dietary goals as compared to other customers or against themselves across different timespans (e.g. year on year). For example, a user with a stated goal of reducing sodium can be shown trends as to how their basket level sodium compares to the previous year and to other users in the same segment for sodium (e.g. a high sodium segment). This then forms the basis of displaying to the user tailored nutritional guidance and product recommendations . Goal Predictor

The goal predictor module builds upon dietary segmentations and cohort analysis from the basket analyser module previously described to profile a given user against nutritional best practice and suggest health or sustainability goals that they are predicted using machine learning to be likely to engage with and to offer them the greatest nutritional benefit. Examples of these goals include nutritional or dietary composition (eat more plants, eat less meat, eat less sodium).

Worked Examples

Example of a product and the steps for processing in the system: Product or recipe is ingested from an external data source (e.g. API) (see above for format) - Burtons Biscuits Jammie Dodgers The data is validated and parsed algorithmically, running hundreds of logic based and machine learning checks to ensure the data conforms to the Spoon Guru internal data specification. For example, in the example above, the text string “For allergens, including Cereals containing Gluten, see ingredients in bold” is found in the ingredients list. This is not an ingredient but a statement in the Spoon Guru spec so it is identified and parsed using machine learning and moved to the ‘statements’ field. The ingredients are parsed and cleaned in a similar manner outputting a clean list of ingredients like so: [‘wheat’, ‘sugar’, ... etc] The formatted and cleaned data is matched against hundreds of rules and machine learning models to create dietary assertions (intermediary statements of facts about the product). For example, the statements “suitable for vegans” and “Vegan friendly? ...yeah!” are marked using machine learning and natural language processing models as positive for the Vegan TAG. Two ingredients that are potentially made from animal sources (glycerol and calcium) are marked with assertions indicating this. The assertions (the product will have hundreds of them spanning all our TAG attributes and all the data on the product) are evaluated holistically for each TAG. In this example, the ambiguous ingredients are overridden by the “suitable for vegans” statement which indicates that those ingredients are not in this case derived from animal sources. Without the statement, the product would be excluded for the Vegan TAG but as it is there, this product is marked as Vegan.

5. This concludes core processing with the product having been evaluated for each attribute. Hundreds of additional output checks are now run, checking assumptions built up from years of experience handling diverse data and also including random sampling of outputs. As such areas where attributes may be incorrect and a random sample of outputs are flagged for human review.

6. A team of nutritionists work in a bespoke web application, the QA console, continuously reviewing potential issues and configuring or updating logic, rules and metadata. All changes are reprocessed within 24 hours, including retraining machine learning models. As such it improves daily in small increments.

Worked Example from User Perspective

1. Retailer shopper ‘Jane’ is vegan and tries to eat healthily - and it is a challenge finding foods that she can eat. She begins to shop online with a retailer using Spoon Guru technology.

2. Spoon Guru TAGs are integrated by the retailer into search and browse user experiences. When Jane searches for ‘vegan sausages’, she immediately sees that the results are much more accurate than in other retailers. Not only are the results actually vegan, whereas typically on close inspection she finds that search results are highly inaccurate, but there is a much broader coverage of products than usual. She finds products that are vegan that she had no idea existed. She is also able to filter search results by dietary attributes and as a result has a much smoother shopping experience.

3. Jane begins to see that she is being recommended products that are highly relevant to her - on product pages, at checkout and in marketing communications. She is finding healthier options and versions of products that she usually buys, that are also vegan, as the retailer is leveraging Spoon Guru TAGs, Smart Swaps and recommendations. As a result she adds more to her basket.