Title:
EDITABLE FREE-VIEWPOINT VIDEO USING A LAYERED NEURAL REPRESENTATION
Document Type and Number:
WIPO Patent Application WO/2023/004559
Kind Code:
A1
Abstract:
Described herein is a computer-implemented method of generating editable free-viewport videos. A plurality of video of a scene from a plurality of views is obtained. The scene comprises an environment and one or more dynamic entities. A 3D bounding-box is generated for each dynamic entity in the scene. A computer device encodes a machine learning model comprising an environment layer and a dynamic entity layer for each dynamic entity in the scene. The environment layer represents a continuous function of space and time of the environment. The dynamic entity layer represents a continuous function of space and time of the dynamic entity. The dynamic entity layer comprises a deformation module and a neural radiance module. The deformation module is configured to deform a spatial coordinate in accordance with a timestamp and a trained deformation weight. The neural radiance module is configured to derive a density value and a color in accordance with the deformed spatial coordinate, the timestamp, a direction, and a trained radiance weight. The machine learning model is trained using the plurality of videos. The scene is rendered in accordance with the trained machine learning model.
More Like This:
| JP4563513 | Image processing device and pseudo stereoscopic image generator |
| WO/2018/163898 | FREE VIEWPOINT MOVEMENT DISPLAY DEVICE |
| WO/2010/003844 | METHOD FOR THE REAL-TIME COMPOSITION OF A VIDEO |
Inventors:
ZHANG JIAKAI (CN)
YU JINGYI (CN)
XU LAN (CN)
YU JINGYI (CN)
XU LAN (CN)
Application Number:
PCT/CN2021/108513
Publication Date:
February 02, 2023
Filing Date:
July 26, 2021
Export Citation:
Assignee:
UNIV SHANGHAI TECH (CN)
International Classes:
G06T15/20; G06T17/05
Foreign References:
| US20170347055A1 | 2017-11-30 | |||
| CN113099208A | 2021-07-09 | |||
| CN112862901A | 2021-05-28 |
Other References:
PUMAROLA ALBERT; CORONA ENRIC; PONS-MOLL GERARD; MORENO-NOGUER FRANCESC: "D-NeRF: Neural Radiance Fields for Dynamic Scenes", 2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, 20 June 2021 (2021-06-20), pages 10313 - 10322, XP034007357, DOI: 10.1109/CVPR46437.2021.01018
Attorney, Agent or Firm:
SHANGHAI SAVVY INTELLECTUAL PROPERTY AGENCY (CN)
Download PDF: