CROSS-REFERENCE

[0001] This application claims priority to U.S. Provisional Patent Application No. 63/347,307, filed May 31, 2022, which is entirely herein incorporated by reference.

BACKGROUND

[0002] Quantitative measures of foods are often valuable to consumers. Examples of quantitative measures of foods are nutrient quantities, such as calories, carbohydrates, fats, and proteins, in addition to diets that the food is consistent with, such as the vegetarian diet, the vegan diet, the Atkins diet, and the paleo diet. Additionally, quantitative measures could be non-nutrient, non-dietary in nature, such as whether the food was fermented or whether it contains red meat.

[0003] It may be difficult for consumers to extract these characteristics (e.g., diet consistency or nutrient quantities) from text describing food items.

SUMMARY

[0004] In one aspect, the present disclosure provides a method for predicting quantitative measures for food items from text or other descriptions of the food items. A user device can generate and apply food text data to a feature prediction machine learning (ML) system that can predict one or more quantitative food features. The feature prediction ML system can include one or more artificial neural network trained with the training data comprising training food text data and corresponding training feature data; and provide the at least one quantitative food feature on the user device.

[0005] In an aspect, a method is disclosed, comprising: by operation of a user device, generate food text data; apply the generated food text data to a feature prediction machine learning (ML) system to predict at least one quantitative food feature, the feature prediction ML system comprising at least one artificial neural network trained with the training data comprising training food text data and corresponding training feature data; and provide the at least one quantitative food feature on the user device.

[0006] In some embodiments, the ML system comprises an artificial neural network.

[0007] In some embodiments, the artificial neural network comprises a recurrent network. [0008] In some embodiments, the recurrent network is selected from the group of a long short-term memory and a gated recurrent unit.

[0009] In some embodiments, the artificial neural network comprises an attention component.

[0010] In some embodiments, the attention components comprise a transformer network. [0011] In some embodiments, generating food text data comprises capturing text data with the user device.

[0012] In some embodiments, capturing the text data includes capturing non-text food data with the user device and converting the non-text food data to the generated food text data. [0013] In some embodiments, capturing text data is selected from the group of: menu text data, recipe text data and food log data.

[0014] In some embodiments, the quantitative feature comprises a diet compliance value. [0015] In some embodiments, the quantitative feature comprises nutrient information.

[0016] In some embodiments, the method further includes by operation of a trained expert machine learning system, generating the training feature data corresponding to the training food text data.

[0017] In an aspect, a system is disclosed, comprising: a user device configured to generate food text data and provide predicted quantitative food feature data for the generated food text data; at least one machine learning (ML) system in communication with the user device comprising at least one artificial neural network trained with the training data comprising training food text data and corresponding training quantitative food feature data, the ML system configured to receive the generated food text data, generate the predicted quantitative food feature data therefrom, and communicate the quantitative food feature data to the user device.

[0018] In some embodiments, the ML system comprises an artificial neural network.

[0019] In some embodiments, the artificial neural network comprises a recurrent network.

[0020] In some embodiments, the recurrent network is selected from the group of a long short-term memory and a gated recurrent unit.

[0021] In some embodiments, the artificial neural network comprises an attention component.

[0022] In some embodiments, the attention component comprises a transformer network.

[0023] In some embodiments, the user device is configured to capture the food text data from a source external to the user device.

[0024] In some embodiments, the user device is configured to capture non-text food data with the user device, and convert the non-text food data to the generated food text data. [0025] In some embodiments, the quantitative feature comprises a diet compliance value. [0026] In some embodiments, the quantitative feature comprises nutrient information.

[0027] In some embodiments, the system further includes at least one expert system configured to generate the training feature data from the training food text data.

[0028] In some embodiments, the at least one expert system comprises an expert ML system.

INCORPORATION BY REFERENCE

[0029] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent publications and patents or patent applications incorporated by reference contradict the disclosure contained in the specification, the specification is intended to supersede and/or take precedence over any such contradictory material.

BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings (also “Figure” and “FIG.” herein), of which:

[0031] FIGS. 1A and IB are diagrams showing method and systems for predicting quantitative food information from food text according to embodiments.

[0032] FIGS. 2A and 2B are diagrams showing method and systems for predicting nutrition and diet information from food text according to embodiments.

[0033] FIG. 3 is a diagram of a system according to an embodiment.

[0034] FIGS. 4A to 4C are diagrams of a user device and operations according to embodiments.

[0035] FIG. 5 shows a computer system that is programmed or otherwise configured to implement methods provided herein. DETAILED DESCRIPTION

[0036] While various embodiments of the invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions may occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed.

[0037] Embodiments describe a method for creating a system that takes as input a text description of food and outputs quantitative measures of that food.

[0038] In one embodiment, an example dataset is collected containing text descriptions of food items and corresponding quantitative measures of the food items described. For example, the text descriptions could be from food items listed on menus. For each described food item, a nutritionist or other specialist may estimate nutrients and dietary consistency. [0039] Weights of an artificial neural network could then be adjusted, for example through the use of a backpropagation algorithm, such that the artificial neural network predicts quantitative measures for the text description that closely agree with the example quantitative measures from the dataset according to an appropriate error measure.

[0040] FIG. 1A is a diagram showing methods and systems 100 according to embodiments. A system 100 can include sets of food text data 102, an expert 104 and a machine learning (ML) system 105.

[0041] In some embodiments, sets of food text data 102 can be acquired. Such sets 102 can be acquired in any suitable fashion, including but not limited to: food menu descriptions, labels, advertisements, or generated with other systems from food images.

[0042] An expert 104 can generate one or more features 106 corresponding to each food text data set. Examples of experts 104 can include, but are not limited to: one or more individuals having expertise in identifying food features, a machine system (including an artificial intelligence system), or a food manufacturer/producer. Features 106 can include any suitable feature corresponding to a food, including but not limited to: nutrient data, dietary consistency (e.g., how much the food may comply with a particular diet), availability (e.g., geographical/geolocation data), cost or demographic match (e.g., does a particular demographic population prefer this type of food). Features 106 can include one feature, the probability of one feature (e.g., of the one feature), multiple features and/or the probabilities of multiple features (e.g., of the multiple features).

[0043] Food text and corresponding features 102/106 can serve as training data for ML system 105. Food text 102 can serve as input values while feature data 106 can serve to evaluate a learning system predicted value. An ML system 105 can take any suitable form. In some embodiments, an ML system 105 can include an artificial neural network. Non-limiting examples of neural networks may comprisea recurrent network and/or a transformer network. In the embodiment shown, a ML system 105 can include function/model 108, error function 110, and parameter adjust operation 112. In response to one or more text data sets 102, a function/model 108 can predict one or more features 107 based on parameters. In some embodiments, processing of text data sets 102 by ML system 105 can include encoding positions of words with respect to one another in the text data. An error function 110 can determine an error 109 between features 107 predicted by function/model 108 and the expert generated features 106 corresponding to the input food text data set. Parameter adjust operation 112 can adjust parameters of function/model 108 based on error values 109 from error function 110 to attempt to converge on predictions with less error.

[0044] FIG. IB shows a prediction system and operations 100B according to embodiments. A system 100B can include a trained function/model 108B. A trained function/model 108B can be a function/model 108 after being trained with multiple food textfeature data sets 102/106, as shown in FIG. 1A.

[0045] A trained function/model 108B can receive new food text data 102B, and in response thereto, generate one or more predicted features 106B.

[0046] FIG. 2A is a diagram showing methods and systems 200 according to additional embodiments. FIG. 2A can include features like those of FIG. 1A, including sets of food text data 202, an expert 204, and a ML system 205.

[0047] In the embodiment of FIG. 2A, an expert 204 can result in features 206 for text data sets 202 that can include nutrition information (e.g., Nutrition Facts) and/or dietary consistency. In the example shown, dietary consistency with particular diets is shown as “Diet x”, where variations in x indicate a different diet. Dietary consistency can include, but is not limited to including, whether the food text indicates compliance with a diet, or a probability of compliance with a diet of the food item corresponding to the text. Nutrition can include any suitable nutrition data, including but not limited to: content (e.g., calories, fats, cholesterol, sodium, carbohydrates, fiber, sugars, proteins, minerals, vitamins), rating (e.g., blood glucose impact, blood pressure impact, or other health related scoring).

[0048] FIG. 2B shows methods and systems 200B according to another embodiment. A system 200B can include a trained function/model 208B, such as function/model 208 trained as shown in FIG. 2A. In response to new food text data 202B, a trained function/model 208B can generate predicted nutrition and/or dietary consistency information 206B. [0049] FIG. 3 shows a system 300 according to another embodiment. A system 300 can include one or more ML servers 330, application servers 328, data store 326, a food data source 350, subject devices 332 and one or more experts 304. Food data source 350, servers (330/328), subject devices 332 and expert 304 can be in communication with one another, such as through a network 334, which can include various interconnected networks, including the internet.

[0050] ML server 330 can include systems and execute methods like those described herein and equivalents.

[0051] An application server 328 can interact with one or more applications 338 running on a subject device 332. In some embodiments, data from food data sources 350 can be acquired via one or more applications 338 on a subject device (e.g., smart phone) 332 and provided to application server 328. Application server 328 can communicate with subject device 332 according to any suitable secure network protocol.

[0052] An expert 304 can be an expert as described herein and equivalents. An expert 304 can receive food text descriptions (e.g., food text data sets). In some embodiments, food text descriptions can include food text data sets 302B from a data store 326 or from sources external to system 300 302A. From food text descriptions 302A/B, an expert 304 can generate feature data 306 for each food text data set, as described herein and equivalents. Such features can include nutrition information and/or dietary consistency data. This can create training data sets 302/306. Training data sets 302/306 can be used to train function/models 308 within ML servers 330.

[0053] A data store 328 can store data for a system 300. In some embodiments, data store 328 can store data received from food data source 330, food text data sets 302B, as well as generated training sets 302/306. A data store 328 can take any suitable form, including one or more network attached storage systems. In some embodiments, all or a portion of data store can be integrated with any of the servers (330, 328).

[0054] In some embodiments, food text data 302 can be generated by a user. In some embodiments, text data 352-0 can be entered by a user (e.g.., typed) or captured by a user (e.g., voice recognition, image capture). In addition or alternatively, image data can be images of actual food (e.g., 352-1), or food packaging, and such image data can be applied to an application on a server to generate a food text set for the image.

[0055] A subject device 332 can be any suitable device, including but not limited to, a smart phone, personal computer, wearable device, or tablet computing device. A subject device 332 can include one or more applications 338 that can communicate with application server 328 to provide data to, and receive data from, models/functions 308 residing on ML servers 330. Such applications 328 can take the form of, and/or include the functions of the various embodiments disclosed herein.

[0056] FIG. 4A to 4C are diagrams of an application 438 according to embodiments. Referring to FIG. 4A, an application 438 can reside on a subject device 432 as instructions executable by a processor or the like, in an embodiment.

[0057] Referring to FIG. 4B, once initiated, an application 438 can request/receive food text data 402B. Such food text data 402B can be acquired in any suitable fashion including as disclosed herein and equivalents. Received food text data 402B can be subject to a trained model/function to generate feature data according to any of the embodiments described herein and equivalents. Such generation of feature data can be performed by the device 432 (e.g., model/function resides on the device) or can be performed by a remote ML server 442, or a combination thereof.

[0058] Referring FIG. 4C, predicted feature data 406A/B from a trained model/function can be presented on subject device 432. In the embodiment shown, predicted feature data can include nutrition information 406A and/or dietary consistency 406B. However, alternate embodiments can present any other suitable features generated by training with food text data sets.

[0059] The present disclosure can provide methods for performing at least the following: predicting quantitative measures of a food item from text about the food item using an artificial neural network; where the artificial neural network is trained using examples of text and quantitative measures; where the artificial neural network contains a recurrent network such as an LSTM (long short-term memory), a GRU (gated recurrent unit) and/or an attention component, such as a transformer network. Quantitative measures can be nutrients. Such nutrients can be carbohydrates, fats, dietary fiber, minerals, proteins, vitamins, or water. Quantitative measures can be whether the food adheres to a diet. Such diets can include, but are not limited to: the paleo diet, the vegan diet, the low-carb diet, the vegetarian diet, the Atkins diet, the HCG (human chorionic gonadotropin) diet, the Zone diet. Food item text can be menu item text. Food item text and be recipe text. Food item text can be free form user text food logs

[0060] It is understood that various blocks shown in the figures described herein can include any of various circuits configured to execute the indicated functions, including but not limited to server systems that may or may not include customized hardware for accelerating operations, logic circuits, including custom logic circuits or programmable logic circuits. Such functions can also correspond to all or a portion of code executable by one or more processors that is stored on machine readable media. Data values as described herein can also be stored in machine readable media. Machine readable media can store code and/or data in a non-transitory form, in volatile and/or nonvolatile storage circuits.

[0061] It should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

[0062] It is also understood that the embodiments of the invention may be practiced in the absence of an element and/or step not specifically disclosed. That is, an inventive feature of the invention may be elimination of an element.

[0063] According to embodiments, blocks or actions that do not depend upon each other can be arranged or executed in parallel.

[0064] Accordingly, while the various aspects of the particular embodiments set forth herein have been described in detail, the present invention could be subject to various changes, substitutions, and alterations without departing from the spirit and scope of the invention.

Computer systems

[0065] The present disclosure provides computer systems that are programmed to implement methods of the disclosure. FIG. 5 shows a computer system 501 that is programmed or otherwise configured to predict food quantities by analyzing text data. The computer system 501 can regulate various aspects of analyzing text of the present disclosure, such as, for example, implementing machine learning algorithms. The computer system 501 can be an electronic device of a user or a computer system that is remotely located with respect to the electronic device. The electronic device can be a mobile electronic device or a desktop computing device.

[0066] The computer system 501 includes a central processing unit (CPU, also “processor” and “computer processor” herein) 505, which can be a single core or multi core processor, or a plurality of processors for parallel processing. The computer system 501 also includes memory or memory location 510 (e.g., random-access memory, read-only memory, flash memory), electronic storage unit 515 (e.g., hard disk), communication interface 520 (e.g., network adapter) for communicating with one or more other systems, and peripheral devices 525, such as cache, other memory, data storage and/or electronic display adapters. The memory 510, storage unit 515, interface 520 and peripheral devices 525 are in communication with the CPU 505 through a communication bus (solid lines), such as a motherboard. The storage unit 515 can be a data storage unit (or data repository) for storing data. The computer system 501 can be operatively coupled to a computer network (“network”) 530 with the aid of the communication interface 520. The network 530 can be the Internet, an internet and/or extranet, or an intranet and/or extranet that is in communication with the Internet. The network 530 in some cases is a telecommunication and/or data network. The network 530 can include one or more computer servers, which can enable distributed computing, such as cloud computing. The network 530, in some cases with the aid of the computer system 501, can implement a peer-to-peer network, which may enable devices coupled to the computer system 501 to behave as a client or a server.

[0067] The CPU 505 can execute a sequence of machine-readable instructions, which can be embodied in a program or software. The instructions may be stored in a memory location, such as the memory 510. The instructions can be directed to the CPU 505, which can subsequently program or otherwise configure the CPU 505 to implement methods of the present disclosure. Examples of operations performed by the CPU 505 can include fetch, decode, execute, and writeback.

[0068] The CPU 505 can be part of a circuit, such as an integrated circuit. One or more other components of the system 501 can be included in the circuit. In some cases, the circuit is an application specific integrated circuit (ASIC).

[0069] The storage unit 515 can store files, such as drivers, libraries and saved programs. The storage unit 515 can store user data, e.g., user preferences and user programs. The computer system 501 in some cases can include one or more additional data storage units that are external to the computer system 501, such as located on a remote server that is in communication with the computer system 501 through an intranet or the Internet.

[0070] The computer system 501 can communicate with one or more remote computer systems through the network 530. For instance, the computer system 501 can communicate with a remote computer system of a user (e.g., a mobile device). Examples of remote computer systems include personal computers (e.g., portable PC), slate or tablet PC’s (e.g., Apple® iPad, Samsung® Galaxy Tab), telephones, Smart phones (e.g., Apple® iPhone, Android-enabled device, Blackberry®), or personal digital assistants. The user can access the computer system 501 via the network 530.

[0071] Methods as described herein can be implemented by way of machine (e.g., computer processor) executable code stored on an electronic storage location of the computer system 501, such as, for example, on the memory 510 or electronic storage unit 515. The machine executable or machine readable code can be provided in the form of software. During use, the code can be executed by the processor 505. In some cases, the code can be retrieved from the storage unit 515 and stored on the memory 510 for ready access by the processor 505. In some situations, the electronic storage unit 515 can be precluded, and machine-executable instructions are stored on memory 510.

[0072] The code can be pre-compiled and configured for use with a machine having a processer adapted to execute the code, or can be compiled during runtime. The code can be supplied in a programming language that can be selected to enable the code to execute in a pre-compiled or as-compiled fashion.

[0073] Aspects of the systems and methods provided herein, such as the computer system 501, can be embodied in programming. Various aspects of the technology may be thought of as “products” or “articles of manufacture” typically in the form of machine (or processor) executable code and/or associated data that is carried on or embodied in a type of machine readable medium. Machine-executable code can be stored on an electronic storage unit, such as memory (e.g., read-only memory, random-access memory, flash memory) or a hard disk. “Storage” type media can include any or all of the tangible memory of the computers, processors or the like, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives and the like, which may provide non-transitory storage at any time for the software programming. All or portions of the software may at times be communicated through the Internet or various other telecommunication networks. Such communications, for example, may enable loading of the software from one computer or processor into another, for example, from a management server or host computer into the computer platform of an application server. Thus, another type of media that may bear the software elements includes optical, electrical and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links. The physical elements that carry such waves, such as wired or wireless links, optical links or the like, also may be considered as media bearing the software. As used herein, unless restricted to non-transitory, tangible “storage” media, terms such as computer or machine “readable medium” refer to any medium that participates in providing instructions to a processor for execution.

[0074] Hence, a machine readable medium, such as computer-executable code, may take many forms, including but not limited to, a tangible storage medium, a carrier wave medium or physical transmission medium. Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computer(s) or the like, such as may be used to implement the databases, etc. shown in the drawings. Volatile storage media include dynamic memory, such as main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wire and fiber optics, including the wires that comprise a bus within a computer system. Carrier-wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media therefore include for example: a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any other optical medium, punch cards paper tape, any other physical storage medium with patterns of holes, a RAM, a ROM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave transporting data or instructions, cables or links transporting such a carrier wave, or any other medium from which a computer may read programming code and/or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0075] The computer system 501 can include or be in communication with an electronic display 535 that comprises a user interface (UI) 540 for providing, for example, a dashboard. Examples of UFs include, without limitation, a graphical user interface (GUI) and web-based user interface.

[0076] Methods and systems of the present disclosure can be implemented by way of one or more algorithms. An algorithm can be implemented by way of software upon execution by the central processing unit 505. The algorithm can, for example, analyze food text data. [0077] Whenever the term “at least,” “greater than,” or “greater than or equal to” precedes the first numerical value in a series of two or more numerical values, the term “at least,” “greater than” or “greater than or equal to” applies to each of the numerical values in that series of numerical values. For example, greater than or equal to 1, 2, or 3 is equivalent to greater than or equal to 1, greater than or equal to 2, or greater than or equal to 3. [0078] Whenever the term “no more than,” “less than,” or “less than or equal to” precedes the first numerical value in a series of two or more numerical values, the term “no more than,” “less than,” or “less than or equal to” applies to each of the numerical values in that series of numerical values. For example, less than or equal to 3, 2, or 1 is equivalent to less than or equal to 3, less than or equal to 2, or less than or equal to 1.

[0079] While preferred embodiments of the present invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. It is not intended that the invention be limited by the specific examples provided within the specification. While the invention has been described with reference to the aforementioned specification, the descriptions and illustrations of the embodiments herein are not meant to be construed in a limiting sense. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the invention. Furthermore, it shall be understood that all aspects of the invention are not limited to the specific depictions, configurations or relative proportions set forth herein which depend upon a variety of conditions and variables. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention. It is therefore contemplated that the invention shall also cover any such alternatives, modifications, variations or equivalents. It is intended that the following claims define the scope of the invention and that methods and structures within the scope of these claims and their equivalents be covered thereby.