Title:
TRAINING MASKED AUTOENCODERS FOR IMAGE INPAINTING
Document Type and Number:
WIPO Patent Application WO/2023/221043
Kind Code:
A1
Abstract:
The disclosure herein describes training an encoder network to inpaint images with masked portions. A primary encoding process is used to encode a visible portion of a masked input image into encoded token data. The encoded token data is then decoded into both pixel regression output and feature prediction output, wherein both outputs include inpainted image data associated with the masked portion of the masked input image. A pixel regression loss is determined using the pixel regression output and pixel data of an unmasked version of the masked input image. A feature prediction loss is determined using the feature prediction output and ground truth encoding output of the unmasked version of the masked input image. The primary encoding process is then trained using the pixel regression loss and the feature prediction loss, whereby the primary encoding process is trained to encode structural features of input images into encoded token data.
Inventors:
CHEN DONGDONG (US)
BAO JIANMIN (US)
ZHANG TING (US)
YUAN LU (US)
CHEN DONG (US)
WEN FANG (US)
DONG XIAOYI (US)
BAO JIANMIN (US)
ZHANG TING (US)
YUAN LU (US)
CHEN DONG (US)
WEN FANG (US)
DONG XIAOYI (US)
Application Number:
PCT/CN2022/093897
Publication Date:
November 23, 2023
Filing Date:
May 19, 2022
Export Citation:
Assignee:
MICROSOFT TECHNOLOGY LICENSING LLC (US)
CHEN DONGDONG (US)
BAO JIANMIN (CN)
ZHANG TING (CN)
YUAN LU (US)
CHEN DONG (US)
WEN FANG (CN)
DONG XIAOYI (CN)
CHEN DONGDONG (US)
BAO JIANMIN (CN)
ZHANG TING (CN)
YUAN LU (US)
CHEN DONG (US)
WEN FANG (CN)
DONG XIAOYI (CN)
International Classes:
G06N3/04; G06N3/08; G06T5/00
Other References:
LIU QIANKUN ET AL: "Reduce Information Loss in Transformers for Pluralistic Image Inpainting", 15 May 2022 (2022-05-15), arXirv.org, pages 1 - 18, XP055982194, Retrieved from the Internet [retrieved on 20221116]
XIAOYI DONG ET AL: "PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 24 November 2021 (2021-11-24), XP091102681
XIAOYI DONG ET AL: "PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 24 November 2021 (2021-11-24), XP091102681
Attorney, Agent or Firm:
SHANGHAI PATENT & TRADEMARK LAW OFFICE, LLC (CN)
Download PDF: