Title:
NEURAL NETWORK-BASED SYSTEMS AND COMPUTER-IMPLEMENTED METHODS FOR IDENTIFYING AND/OR EVALUATING ONE OR MORE FOOD ITEMS PRESENT IN A VISUAL INPUT
Document Type and Number:
WIPO Patent Application WO/2020/050973
Kind Code:
A3
Abstract:
In some embodiments, the present invention provides for an exemplary inventive convolutional neural network-based and computer-implemented method for identifying and evaluating pizza, including: collecting video input representative of the food and received from at least one camera, applying a first CNN to select, from the video input, a set of best pizza containing video frames of a particular pizza from the plurality of pizza containing video frames; applying the first CNN to identify a best pizza containing image from the set, to localize at least one pizza portion of the particular pizza in the identified best pizza containing image, and to determine a type of the pizza of the particular pizza from the identified best pizza containing image; applying a second CNN to determine a map of pizza components of the particular pizza by automatically performing pizza image segmentation and to automatically score the particular pizza based on the determined map of pizza components.
Inventors:
SHANGIN LEONID (RU)
IVANOV ILYA (RU)
WYNNE WILLIAM (US)
IVANOV ILYA (RU)
WYNNE WILLIAM (US)
Application Number:
PCT/US2019/047171
Publication Date:
June 25, 2020
Filing Date:
August 20, 2019
Export Citation:
Assignee:
FLORENT TECH LLC (US)
International Classes:
G06Q10/06; G06N3/04; G06Q50/12
Other References:
DU C-J ET AL: "Recent developments in the applications of image processing techniques for food quality evaluation", TRENDS IN FOOD SCIENCE AND TECHNOLOGY, ELSEVIER SCIENCE PUBLISHERS, GB, vol. 15, no. 5, 1 May 2004 (2004-05-01), pages 230 - 249, XP004499338, ISSN: 0924-2244, DOI: 10.1016/J.TIFS.2003.10.006
DU C J ET AL: "Learning techniques used in computer vision for food quality evaluation: a review", JOURNAL OF FOOD ENGINEERING, BARKING, ESSEX, GB, vol. 72, no. 1, 1 January 2006 (2006-01-01), pages 39 - 55, XP027890960, ISSN: 0260-8774, [retrieved on 20060101]
DA-WEN SUN ET AL: "Pizza quality evaluation using computer vision--part 1", JOURNAL OF FOOD ENGINEERING, vol. 57, no. 1, 1 March 2003 (2003-03-01), GB, pages 81 - 89, XP055694158, ISSN: 0260-8774, DOI: 10.1016/S0260-8774(02)00275-3
DA-WEN SUN ET AL: "Pizza quality evaluation using computer vision--Part 2", JOURNAL OF FOOD ENGINEERING, vol. 57, no. 1, 1 March 2003 (2003-03-01), GB, pages 91 - 95, XP055694160, ISSN: 0260-8774, DOI: 10.1016/S0260-8774(02)00276-5
NOAH PICARD: "Smooth Segmentation in Videos: Blind Consistency Over Semantic Segmentations Produced by Fully Convolutional Networks", THESIS - DEPARTMENT OF COMPUTER SCIENCE AT BROWN UNIVERSITY, 1 April 2018 (2018-04-01), XP055597126, Retrieved from the Internet [retrieved on 20190617]
WU QING ET AL: "The application of deep learning in computer vision", 2017 CHINESE AUTOMATION CONGRESS (CAC), IEEE, 20 October 2017 (2017-10-20), pages 6522 - 6527, XP033290540, DOI: 10.1109/CAC.2017.8243952
MISIMI EKREM ET AL: "Robust classification approach for segmentation of blood defects in cod fillets based on deep convolutional neural networks and support vector machines and calculation of gripper vectors for robotic processing", COMPUTERS AND ELECTRONICS IN AGRICULTURE, ELSEVIER, AMSTERDAM, NL, vol. 139, 24 May 2017 (2017-05-24), pages 138 - 152, XP085061233, ISSN: 0168-1699, DOI: 10.1016/J.COMPAG.2017.05.021
QI XUAN ET AL: "CNN based key frame extraction for face in video recognition", 2018 IEEE 4TH INTERNATIONAL CONFERENCE ON IDENTITY, SECURITY, AND BEHAVIOR ANALYSIS (ISBA), IEEE, 11 January 2018 (2018-01-11), pages 1 - 8, XP033329301, DOI: 10.1109/ISBA.2018.8311477
YANGYU CHEN ET AL: "Less Is More: Picking Informative Frames for Video Captioning", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 5 March 2018 (2018-03-05), XP080857708
ANONYMOUS: "Convolutional neural network - Wikipedia", 29 September 2017 (2017-09-29), XP055526706, Retrieved from the Internet [retrieved on 20181126]
MOHAMMAD MASHI ET AL: "A knowledge-based system for measuring the quality of the distribution of salami slices on pizza crusts", APPLIED ELECTRICAL ENGINEERING AND COMPUTING TECHNOLOGIES (AEECT), 2011 IEEE JORDAN CONFERENCE ON, IEEE, 6 December 2011 (2011-12-06), pages 1 - 8, XP032097260, ISBN: 978-1-4577-1083-4, DOI: 10.1109/AEECT.2011.6132533
DU C J ET AL: "Learning techniques used in computer vision for food quality evaluation: a review", JOURNAL OF FOOD ENGINEERING, BARKING, ESSEX, GB, vol. 72, no. 1, 1 January 2006 (2006-01-01), pages 39 - 55, XP027890960, ISSN: 0260-8774, [retrieved on 20060101]
DA-WEN SUN ET AL: "Pizza quality evaluation using computer vision--part 1", JOURNAL OF FOOD ENGINEERING, vol. 57, no. 1, 1 March 2003 (2003-03-01), GB, pages 81 - 89, XP055694158, ISSN: 0260-8774, DOI: 10.1016/S0260-8774(02)00275-3
DA-WEN SUN ET AL: "Pizza quality evaluation using computer vision--Part 2", JOURNAL OF FOOD ENGINEERING, vol. 57, no. 1, 1 March 2003 (2003-03-01), GB, pages 91 - 95, XP055694160, ISSN: 0260-8774, DOI: 10.1016/S0260-8774(02)00276-5
NOAH PICARD: "Smooth Segmentation in Videos: Blind Consistency Over Semantic Segmentations Produced by Fully Convolutional Networks", THESIS - DEPARTMENT OF COMPUTER SCIENCE AT BROWN UNIVERSITY, 1 April 2018 (2018-04-01), XP055597126, Retrieved from the Internet
WU QING ET AL: "The application of deep learning in computer vision", 2017 CHINESE AUTOMATION CONGRESS (CAC), IEEE, 20 October 2017 (2017-10-20), pages 6522 - 6527, XP033290540, DOI: 10.1109/CAC.2017.8243952
MISIMI EKREM ET AL: "Robust classification approach for segmentation of blood defects in cod fillets based on deep convolutional neural networks and support vector machines and calculation of gripper vectors for robotic processing", COMPUTERS AND ELECTRONICS IN AGRICULTURE, ELSEVIER, AMSTERDAM, NL, vol. 139, 24 May 2017 (2017-05-24), pages 138 - 152, XP085061233, ISSN: 0168-1699, DOI: 10.1016/J.COMPAG.2017.05.021
QI XUAN ET AL: "CNN based key frame extraction for face in video recognition", 2018 IEEE 4TH INTERNATIONAL CONFERENCE ON IDENTITY, SECURITY, AND BEHAVIOR ANALYSIS (ISBA), IEEE, 11 January 2018 (2018-01-11), pages 1 - 8, XP033329301, DOI: 10.1109/ISBA.2018.8311477
YANGYU CHEN ET AL: "Less Is More: Picking Informative Frames for Video Captioning", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 5 March 2018 (2018-03-05), XP080857708
ANONYMOUS: "Convolutional neural network - Wikipedia", 29 September 2017 (2017-09-29), XP055526706, Retrieved from the Internet
MOHAMMAD MASHI ET AL: "A knowledge-based system for measuring the quality of the distribution of salami slices on pizza crusts", APPLIED ELECTRICAL ENGINEERING AND COMPUTING TECHNOLOGIES (AEECT), 2011 IEEE JORDAN CONFERENCE ON, IEEE, 6 December 2011 (2011-12-06), pages 1 - 8, XP032097260, ISBN: 978-1-4577-1083-4, DOI: 10.1109/AEECT.2011.6132533
Attorney, Agent or Firm:
BERSH, Lennie, A. (US)
Download PDF: