Title:
METHOD AND APPARATUS FOR THREE-DIMENSIONAL RECONSTRUCTION OF A HUMAN HEAD FOR RENDERING A HUMAN IMAGE
Document Type and Number:
WIPO Patent Application WO/2023/085624
Kind Code:
A1
Abstract:
A method of three-dimensional reconstruction of human heads using a single photo in the form of polygonal mesh, with animation and realistic rendering capabilities for novel head poses is provided. The method includes encoding, by using a first convolutional neural network, a single source image into a neural texture; estimating, by a pre-trained detailed expression capture and animation (DECA) system, a face shape, a facial expression, and a head pose by using the single source image and a target image, and providing an initial mesh; providing a predicted mesh of a head mesh based on the initial mesh and the neural texture; rendering a human image by using the predicted mesh.
More Like This:
Inventors:
KHAKHULIN TARAS ANDREEVICH (RU)
SKLYAROVA VANESSA VALERIEVNA (RU)
LEMPITSKY VICTOR SERGEEVICH (RU)
ZAKHAROV EGOR OLEGOVICH (RU)
SKLYAROVA VANESSA VALERIEVNA (RU)
LEMPITSKY VICTOR SERGEEVICH (RU)
ZAKHAROV EGOR OLEGOVICH (RU)
Application Number:
PCT/KR2022/015750
Publication Date:
May 19, 2023
Filing Date:
October 17, 2022
Export Citation:
Assignee:
SAMSUNG ELECTRONICS CO LTD (KR)
International Classes:
G06T17/20; G06N3/04; G06T13/20; G06T15/04; G06T15/20; G06V40/16
Domestic Patent References:
| WO2017223530A1 | 2017-12-28 |
Foreign References:
| US20180158240A1 | 2018-06-07 |
Other References:
FENG YAO YFENG@TUEBINGEN.MPG.DE; FENG HAIWEN HFENG@TUEBINGEN.MPG.DE; BLACK MICHAEL J. BLACK@TUEBINGEN.MPG.DE; BOLKART TIMO TBOLKAR: "Learning an animatable detailed 3D face model from in-the-wild images", ACM TRANSACTIONS ON GRAPHICS, ACM, NY, US, vol. 40, no. 4, 19 July 2021 (2021-07-19), US , pages 1 - 13, XP058628574, ISSN: 0730-0301, DOI: 10.1145/3450626.3459936
XIANG FANBO; XU ZEXIANG; HASAN MILOS; HOLD-GEOFFROY YANNICK; SUNKAVALLI KALYAN; SU HAO: "NeuTex: Neural Texture Mapping for Volumetric Neural Rendering", 2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, 20 June 2021 (2021-06-20), pages 7115 - 7124, XP034007854, DOI: 10.1109/CVPR46437.2021.00704
DOU PENGFEI; SHAH SHISHIR K.; KAKADIARIS IOANNIS A.: "End-to-End 3D Face Reconstruction with Deep Neural Networks", 2017 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE COMPUTER SOCIETY, US, 21 July 2017 (2017-07-21), US , pages 1503 - 1512, XP033249490, ISSN: 1063-6919, DOI: 10.1109/CVPR.2017.164
TARAS KHAKHULIN et al., `Realistic One-shot Mesh-based Head Avatars`, arXiv:2206.08343, pp. 1-29, June 2022 [retrieved on 2023-01-16]. Retrieved from:
XIANG FANBO; XU ZEXIANG; HASAN MILOS; HOLD-GEOFFROY YANNICK; SUNKAVALLI KALYAN; SU HAO: "NeuTex: Neural Texture Mapping for Volumetric Neural Rendering", 2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, 20 June 2021 (2021-06-20), pages 7115 - 7124, XP034007854, DOI: 10.1109/CVPR46437.2021.00704
DOU PENGFEI; SHAH SHISHIR K.; KAKADIARIS IOANNIS A.: "End-to-End 3D Face Reconstruction with Deep Neural Networks", 2017 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE COMPUTER SOCIETY, US, 21 July 2017 (2017-07-21), US , pages 1503 - 1512, XP033249490, ISSN: 1063-6919, DOI: 10.1109/CVPR.2017.164
TARAS KHAKHULIN et al., `Realistic One-shot Mesh-based Head Avatars`, arXiv:2206.08343, pp. 1-29, June 2022 [retrieved on 2023-01-16]. Retrieved from:
Attorney, Agent or Firm:
Y.P.LEE, MOCK & PARTNERS (KR)
Download PDF: