YU YUE (US)
YU HAOPING (US)
US20210090301A1 | 2021-03-25 | |||
US20200286261A1 | 2020-09-10 | |||
US20130114910A1 | 2013-05-09 | |||
US20120262444A1 | 2012-10-18 | |||
US20070194975A1 | 2007-08-23 |
Claims 1. A computer-implemented method for decoding a coded mesh bitstream of a dynamic mesh representing three-dimensional (3D) content, the method comprising: reconstructing geometry information of the dynamic mesh from a geometry component bitstream in the coded mesh bitstream, the reconstructed geometry information comprising data specifying a plurality of vertices of the dynamic mesh; reconstructing connectivity information of the dynamic mesh from a connectivity component bitstream in the coded mesh bitstream, the reconstructed connectivity information comprising data specifying a plurality of faces of the dynamic mesh; refining the reconstructed connectivity information based on the reconstructed geometry information to generate refined connectivity information by at least dividing a face out of the plurality of faces specified by the reconstructed connectivity information into two faces based on a vertex of the plurality of vertices specified in the reconstructed geometry information; reconstructing the dynamic mesh based on the reconstructed geometry information and the refined connectivity information; and causing the reconstructed dynamic mesh to be rendered for display. 2. The computer-implemented method of claim 1, wherein refining the reconstructed connectivity information based on the reconstructed geometry information comprises: identifying a vertex of the plurality of vertices that is located inside a face of the plurality of faces; determining a projected vertex of the vertex on an edge of the face; and dividing the face into two refined faces, each refined face has the projected vertex as one vertex. 3. The computer-implemented method of claim 2, wherein each of the two refined faces have a same orientation as the face. 4. The computer-implemented method of claim 2, wherein determining the projected vertex of the vertex comprises: determining a nearest edge for the vertex among edges of the face, wherein a distance between the vertex to the nearest edge is smaller than another distance between the vertex to another edge of the edges of the face; and determining the projected vertex as an intersection of a line with the nearest edge, the line passing through the vertex and an opposite vertex of the face that does not belong to the nearest edge. 5. The computer-implemented method of claim 1, wherein refining the reconstructed connectivity information based on the reconstructed geometry information comprises: identifying two or more vertices of the plurality of vertices that are located inside a face of the plurality of faces; selecting, among the two or more vertices, a vertex for processing based on a distance between each of the two or more vertices and a corresponding nearest edge; determining a projected vertex of the selected vertex on an edge of the face; and dividing the face into two refined faces, each refined face has the projected vertex as one vertex. 6. The computer-implemented method of claim 5, wherein selecting the vertex for processing among the two or more vertices comprises: determining, for each vertex of the two or more vertices, a nearest edge among edges of the face, wherein a distance between the vertex to the nearest edge is smaller than another distance between the vertex to another edge of the edges of the face; and selecting the vertex for processing as a vertex among the two or more vertices that has a largest distance to the corresponding nearest edge. 7. The computer-implemented method of claim 1, wherein reconstructing the geometry information of the dynamic mesh from the geometry component bitstream comprises decoding the geometry component bitstream by a video decoder. 8. A non-transitory computer-readable medium having program code that is stored thereon, the program code executable by one or more processing devices for performing operations comprising: reconstructing geometry information of a dynamic mesh from a geometry component bitstream in a coded mesh bitstream of the dynamic mesh, the reconstructed geometry information comprising data specifying a plurality of vertices of the dynamic mesh; reconstructing connectivity information of the dynamic mesh from a connectivity component bitstream in the coded mesh bitstream, the reconstructed connectivity information comprising data specifying a plurality of faces of the dynamic mesh; refining the reconstructed connectivity information based on the reconstructed geometry information to generate refined connectivity information by at least dividing a face out of the plurality of faces specified by the reconstructed connectivity information into two faces based on a vertex of the plurality of vertices specified in the reconstructed geometry information; reconstructing the dynamic mesh based on the reconstructed geometry information and the refined connectivity information; and causing the reconstructed dynamic mesh to be rendered for display. 9. The non-transitory computer-readable medium of claim 8, wherein refining the reconstructed connectivity information based on the reconstructed geometry information comprises: identifying a vertex of the plurality of vertices that is located inside a face of the plurality of faces; determining a projected vertex of the vertex on an edge of the face; and dividing the face into two refined faces, each refined face has the projected vertex as one vertex. 10. The non-transitory computer-readable medium of claim 9, wherein each of the two refined faces have a same orientation as the face. 11. The non-transitory computer-readable medium of claim 9, wherein determining the projected vertex of the vertex comprises: determining a nearest edge for the vertex among edges of the face, wherein a distance between the vertex to the nearest edge is smaller than another distance between the vertex to another edge of the edges of the face; and determining the projected vertex as an intersection of a line with the nearest edge, the line passing through the vertex and an opposite vertex of the face that does not belong to the nearest edge. 12. The non-transitory computer-readable medium of claim 8, wherein refining the reconstructed connectivity information based on the reconstructed geometry information comprises: identifying two or more vertices of the plurality of vertices that are located inside a face of the plurality of faces; selecting, among the two or more vertices, a vertex for processing based on a distance between each of the two or more vertices and a corresponding nearest edge; determining a projected vertex of the selected vertex on an edge of the face; and dividing the face into two refined faces, each refined face has the projected vertex as one vertex. 13. The non-transitory computer-readable medium of claim 12, wherein selecting the vertex for processing among the two or more vertices comprises: determining, for each vertex of the two or more vertices, a nearest edge among edges of the face, wherein a distance between the vertex to the nearest edge is smaller than another distance between the vertex to another edge of the edges of the face; and selecting the vertex for processing as a vertex among the two or more vertices that has a largest distance to the corresponding nearest edge. 14. The non-transitory computer-readable medium of claim 8, wherein reconstructing the geometry information of the dynamic mesh from the geometry component bitstream comprises decoding the geometry component bitstream by a video decoder. 15. A system comprising: a processing device; and a non-transitory computer-readable medium communicatively coupled to the processing device, wherein the processing device is configured to execute program code stored in the non-transitory computer-readable medium and thereby perform operations comprising: reconstructing geometry information of a dynamic mesh from a geometry component bitstream in a coded mesh bitstream of the dynamic mesh, the reconstructed geometry information comprising data specifying a plurality of vertices of the dynamic mesh; reconstructing connectivity information of the dynamic mesh from a connectivity component bitstream in the coded mesh bitstream, the reconstructed connectivity information comprising data specifying a plurality of faces of the dynamic mesh; refining the reconstructed connectivity information based on the reconstructed geometry information to generate refined connectivity information by at least dividing a face out of the plurality of faces specified by the reconstructed connectivity information into two faces based on a vertex of the plurality of vertices specified in the reconstructed geometry information; reconstructing the dynamic mesh based on the reconstructed geometry information and the refined connectivity information; and causing the reconstructed dynamic mesh to be rendered for display. 16. The system of claim 15, wherein refining the reconstructed connectivity information based on the reconstructed geometry information comprises: identifying a vertex of the plurality of vertices that is located inside a face of the plurality of faces; determining a projected vertex of the vertex on an edge of the face; and dividing the face into two refined faces, each refined face has the projected vertex as one vertex. 17. The system of claim 16, wherein each of the two refined faces have a same orientation as the face. 18. The system of claim 16, wherein determining the projected vertex of the vertex comprises: determining a nearest edge for the vertex among edges of the face, wherein a distance between the vertex to the nearest edge is smaller than another distance between the vertex to another edge of the edges of the face; and determining the projected vertex as an intersection of a line with the nearest edge, the line passing through the vertex and an opposite vertex of the face that does not belong to the nearest edge. 19. The system of claim 15, wherein refining the reconstructed connectivity information based on the reconstructed geometry information comprises: identifying two or more vertices of the plurality of vertices that are located inside a face of the plurality of faces; selecting, among the two or more vertices, a vertex for processing based on a distance between each of the two or more vertices and a corresponding nearest edge; determining a projected vertex of the selected vertex on an edge of the face; and dividing the face into two refined faces, each refined face has the projected vertex as one vertex. 20. The system of claim 19, wherein selecting the vertex for processing among the two or more vertices comprises: determining, for each vertex of the two or more vertices, a nearest edge among edges of the face, wherein a distance between the vertex to the nearest edge is smaller than another distance between the vertex to another edge of the edges of the face; and selecting the vertex for processing as a vertex among the two or more vertices that has a largest distance to the corresponding nearest edge. |
[0065] In some examples, the decoding process include the following steps: obtaining a coded mesh bitstream from network or a storage source, demultiplexing components of the coded bitstream and decoding in parallel or otherwise independently, the geometry component bitstream, the attribute component bitstream, the connectivity component bitstream, and the mapping component bitstream. The decoding process further includes refining the decoded connectivity information to reconstruct or recover the original connectivity information that has been simplified by subdividing edges of faces or triangles that contain vertex inside thereof. The refined and decoded components can be used to reconstruct the mesh frames. By refining the decoded connectivity information, the distortions and visual artifacts in the decoded mesh frames that were introduced by the connectivity simplification can be reduced.
[0066] FIG. 11 illustrates an example decoder system 1100 for mesh decoding with connectivity refinement, according to various embodiments of the present disclosure. The decoder system 1100 shown in FIG. 11 includes various components or modules that are similar to those in FIG. 2. In addition, the decoder system 1100 includes a connectivity refinement module 1102 to recover the connectivity information that was simplified at the encoder before the connectivity information is sent to the mesh reconstruction module 226 for reconstructing the mesh frame sequence 1104.
[0067] In some examples, the connectivity refinement module 1102 obtains the geometry information decoded by the geometry reconstruction module 232 from the geometry component bitstream. Vertex coordinates (X, Y, Z) are reconstructed using the decoded geometry and atlas components. The connectivity refinement module 1102 further obtains the connectivity information decoded from the connectivity component bitstream by the connectivity reconstruction module 234. The connectivity refinement module 1102 can mark the vertices from the geometry component that belong to the face list in the decoded connectivity information as visited, and mark the vertices that do not belong to the face list as non-visited.
[0068] For each non-visited vertex P', the connectivity refinement module 1102 finds a face or triangle inside which P' is located, e.g., The connectivity refinement module 1102 projects the vertex P' to the nearest edge (e.g., ) of the triangle to find a projection vertex P” In some examples, the nearest edge is identified through the minimum Euclidean distance between the vertex and the edge. The projection vertex P" is defined as an intersection of the ray casted from the opposite vertex V 2 of the triangle 3 to the nearest edge of the triangle The connectivity refinement module 1102 subdivides the triangle into two triangles both with the same orientation as the triangle 2 3 . If there are more than one point inside the triangle 3 , the connectivity refinement module 1102 can iterate this process recursively starting from the point that has the maximum Euclidian distance to the edge. Then the updated triangle is used for further subdivision. The refined connectivity information can be used to reconstruct the mesh frames along with other decoded components.
[0069] FIG. 12 shows an example of the decoder side connectivity refinement process, according to some embodiments of the present disclosure. In FIG. 12, the decoded geometry 1202 includes four points: P’, and as shown in sub-figure (A). Without connectivity simplification, the four points should form two faces: and . Due to the simplification during encoding, these two faces have been merged as indicated by sub-figure (B) which shows the decoded geometry and connectivity information. In sub-figure (B), V 2 , and V 3 form a face/triangle with the point P’ inside this face. Because the connectivity information has been simplified at the encoder, the isolated point P’ inside the face indicates that the decoded triangle is a merged triangle from at least two triangles. To restore the triangles, the nearest edge for P’ in triangle is identified as in this example. P' is projected to this nearest edge to obtain the projection vertex P”. The projection can be performed by extending the ray casted from the opposite vertex of the edge, V , to find the intersection with the edge as shown in sub-figure (C) of FIG. 12. The triangle is then divided into two triangles V P"V 2 and P"^^ each having the same orientation as triangle AS such, the decoded geometry and refined connectivity information shown in sub figure (D) includes the two triangles instead of the merged triangle
[0070] FIG. 13 shows another example of the connectivity refinement process, according to some embodiments of the present disclosure. In FIG. 13, there are more than one point inside the decoded triangle. As shown in sub-figure (A), the decoded geometry information includes five points: V As shown in sub-figure (B), the decoded connectivity information indicates that one face is encoded as with and inside this face. The nearest edge for P^ is identified as and the nearest edge for 2 is identified as as shown in sub-figure (C) of FIG. 13. Between P ^ and has a larger distance to the nearest edge and thus the refinement is performed for ^ first. As shown in sub- figure (D), the projection vertex P is found for as the intersection of the ray with the edge The triangle is thus divided into two triangles and t each having the same counter-clockwise orientation as the triangle V
[0071] If any of the triangles contains a point inside it, such a triangle will be further divided. In this example, triangle has P inside it and thus will be further analyzed for division. As shown in sub-figure (E), the projection vertex P is found for 2 as the intersection of the ray with P ’s nearest edge Triangle V thus can be further divided into triangle and triangle both have the same orientation as the triangle . Sub-figure (F) shows the reconstructed topology formed by the decoded geometry information with the refined connectivity information, where there are three reconstructed triangles, instead of the simplified one triangle before the refinement.
[0072] Referring now to FIG. 14, FIG. 14 depicts an example of a process 1400 for mesh coding with connectivity simplification, according to some embodiments of the present disclosure. One or more computing devices implement operations depicted in FIG. 1400 by executing suitable program code. For example, the encoder system 800 in FIG. 8 may implement the operations depicted in FIG. 1400 by executing the corresponding program code. For illustrative purposes, the process 1400 is described with reference to some examples depicted in the figures. Other implementations, however, are possible.
[0073] At block 1402, the process 1400 involves accessing a dynamic mesh to be encoded. As discussed above, the dynamic mesh may be represented as uncompressed mesh frame sequence that include mesh frames. A mesh frame is a data format that describes 3D content (e.g., 3D objects) in a digital representation as a collection of geometry, connectivity, attribute, and attribute mapping information. Each mesh frame is characterized by a presentation time and duration. A mesh frame sequence (e.g., sequence of mesh frames) forms a dynamic mesh video. The uncompressed mesh frame sequence can be segmented into segmented mesh data. Based on the segmented mesh data, the encoder system 800 can generate attribute component images, geometry component images, connectivity component images, and mapping component images. [0074] At block 1404, the process 1400 involves encoding the geometry component images using a video encoder to generate a geometry component bitstream as described in detail with respect to FIGS. 1 and 8. At block 1406, the process 1400 involves decoding the encoded geometry component images to generate reconstructed geometry component images. The decoding is performed using a video decoder that corresponds to the video encoder used to the encode the geometry component images. As discussed above, in examples where the video encoder (e.g., video encoder 120b) is a lossy encoder, due to prediction and quantization, the reconstructed geometry component images will be different from the geometry component images. As such, the reconstructed position of the decoded vertexes of the mesh may differ from the original position of the vertexes of the mesh. As a result, neighboring faces may degenerate into simplified geometry. [0075] At block 1408, the process 1400 involves determining a vertex in the reconstructed geometry component images that belongs to the face(s) to be removed from connectivity component images of the dynamic mesh. To determine the face to be removed, four vertices of a pair of neighboring faces or triangles of the mesh can be analyzed. In some examples, the two neighboring triangles have the same orientation. A triangle can be formed using three of the vertices with the fourth vertex inside the triangle. The distance from the fourth vertex to each of the three edges of the triangle can be determined. The smallest distance is determined among the three distances. If the smallest distance is below a pre-determined threshold, these two neighboring faces can be selected as faces to be removed and replaced with a single face. The generated triangle can be used as the triangle to be encoded in place of the two neighboring triangles (i.e., the two neighboring triangles are merged into the generated triangle). This step can be repeated to identify multiple vertices to be removed and multiple pairs of neighboring triangles to be merged. [0076] At block 1410, the process 1400 involves updating the connectivity component images of the dynamic mesh to simplify the connectivity information. For example, the simplification can include removing the faces identified in block 1408 from the connectivity component images and updating the connectivity component images to replace the data for the pairs of neighboring triangles with the corresponding merged triangles. At block 1412, the process 1400 involves encoding the updated connectivity component images to generate a connectivity component bitstream. As discussed above in detail with respect to FIGS.1 and 8, the encoding may involve using a video encoder to generate the connectivity component bitstream. At block 1414, the process 1400 involves generating a coded mesh bitstream by including at least the geometry component bitstream and the connectivity component bitstream. For example, the coded mesh bitstream can be generated by multiplexing the mesh bitstream payload, that includes the geometry component bitstream and the connectivity component bitstream as well as other bitstream such as attribute component bitstream, mapping component bitstream, and so on, with a mesh bitstream header. [0077] Referring now to FIG. 15, FIG. 15 depicts an example of a process 1500 for decoding a coded mesh bitstream with connectivity refinement, according to some embodiments of the present disclosure. One or more computing devices implement operations depicted in FIG. 1500 by executing suitable program code. For example, the decoder system 1100 in FIG. 11 may implement the operations depicted in FIG. 1100 by executing the corresponding program code. For illustrative purposes, the process 1500 is described with reference to some examples depicted in the figures. Other implementations, however, are possible. [0078] At block 1502, the process 1500 involves accessing a coded mesh bitstream of a dynamic mesh for decoding. The coded mesh bitstream is encoded with connectivity simplification described above. The coded mesh bitstream can include a geometry component bitstream, an attribute component bitstream, a connectivity component bitstream, and a mapping component bitstream. [0079] At block 1504, the process 1500 involves reconstructing geometry information of the dynamic mesh from the geometry component bitstream in the coded mesh bitstream. The reconstructed geometry information includes data specifying vertices of the dynamic mesh. As discussed above in detail with respect to FIGS. 2 and 11, the geometry information of the dynamic mesh can be reconstructed from the geometry component bitstream by applying a video decoder to the geometry component bitstream to generate reconstructed geometry component images which is then used to reconstruct the ordered list of vertex coordinates in the geometry information. [0080] At block 1506, the process 1500 involves reconstructing connectivity information of the dynamic mesh from the connectivity component bitstream in the coded mesh bitstream. The reconstructed connectivity information includes data specifying the faces (e.g., triangles) of the dynamic mesh. Similar to the geometry information, the connectivity information of the dynamic mesh can be reconstructed from the connectivity component bitstream by applying a video decoder to the connectivity component bitstream to generate reconstructed connectivity component images. The generated reconstructed connectivity component images are then used to reconstruct the ordered list of face information with the corresponding vertex index and, in some cases, the texture index. Other components of the coded mesh bitstream, such as the attribute component bitstream, mapping component bitstream, can also be decoded and used to reconstruct the corresponding information. [0081] At block 1508, the process 1500 involves refining the reconstructed connectivity information based on the reconstructed geometry information to generate refined connectivity information. As discussed above in detail, the refinement process divides a face specified by the reconstructed connectivity information into two faces based on a vertex specified in the reconstructed geometry information but not listed as a vertex of any faces specified by the reconstructed connectivity information. For example, a vertex that is located inside a face of the reconstructed connectivity information can be identified and its projected vertex on an edge of the face can be determined. The face can then be divided into two refined faces, each refined face having the projected vertex as one vertex and having the same orientation as the initially reconstructed face. [0082] At block 1510, the process 1500 involves reconstructing the dynamic mesh based on the reconstructed geometry information and the refined connectivity information as well as other information including the attribute information, the mapping information, and so on. At block 1512, the process 1500 involves causing the reconstructed dynamic mesh to be rendered for display. For example, the reconstructed dynamic mesh can be transmitted to a device or a module configured to render the 3D object represented by the reconstructed dynamic mesh to generate rendered images or video for display. [0083] FIG. 16 illustrates an example of an encoder system 1600 for mesh coding with connectivity simplification and attribute and mapping adjustment, according to various embodiments of the present disclosure. The encoder system 1600 shown in FIG. 16 includes various components or modules that are similar to those in FIG. 8. In addition, the encoder system 1600 includes a mapping topology update module 1602 and an attribute image update module 1604 to update the mapping topology and the attribute images, respectively, in addition to the connectivity simplification to reduce the distortion caused by the reconstruction errors of the geometry information. As shown in FIG. 10, because of the lossy compression of the geometry information, the reconstructed vertices may have different coordinates than the original coordinates. In this example, a vertex P becomes P’ after the encoding and reconstruction. To reduce the visual artifacts caused by this reconstruction error, the attribute information and the mapping information should be adjusted accordingly. For example, the attribute image update module 1604 can update the attribute component images to reflect the distortion caused by geometry encoding and reconstruction. To do so, the attribute image update module 1604 can employ an attribute reprojection process. [0084] The distortions associated with an attribute image that was introduced by geometry compression error can be described as a transformation such as an affine transformation. The transformation is applied to a point that belongs to the edge between the two neighboring triangles that are degenerated and merged during the connectivity simplification, such as point P in FIG.10. As such, the reprojection process can be implemented as the same transformation applied to corresponding attribute samples of each degenerated triangular face in the attribute image. The boundaries of the triangular face are defined by the vertex mapping coordinates (U, V) and the connectivity information (f_idx_1, f_idx_2, f_idx_3) as shown in FIG. 5. The transformation T() can be represented using a 4x4 matrix M, where ^ ^ ^ = ^ ^ ^ The transformation T() can be an affine transformation, The affine transformation T() transforms point (or vector) p to point (or vector) p’. The transformation T() of point p to point p’ is obtained by performing the matrix-vector multiplication Mp: One example of matrix M to implement shear transformation T() is ( 4) For example, the shear transform parameters Sh can be derived from the vectors representing the points P’ and P shown in FIG.10. [0085] Another example of matrix M to implement rotation transformation T() is ^ . (5) [0086] FIG. 17 shows examples of the affine transformations. Sub-figure (A) shows an example of the shear transformation for the 2-dimensional case and sub-figure (B) shows an example of the rotation transformation for the 2-dimensional case. The attribute image update module 1604 can select the format of the affine transformation (e.g., the shear transformation, the rotation transformation, or another transformation) and determine the parameters of the transformation based on the distortions associated with the lossy compression applied to geometry component and connect connectivity simplification. [0087] Alternatively, or additionally, an interpolation transformation filter, such as a bilinear interpolation transformation filter, a Lanczos interpolation transformation filter, can be used to reproject the face of the attribute image corresponding to the degenerated face on the simplified topology. The parameters of the filter, such as the coefficients, positions, and length of the filter can be signaled in the coded mesh bitstream or be an integral part of the encoder, such as look-up table. For examples, the parameters can be signaled using supplemental enhancement information message or by indicating a filter index in a look-up table. The updated attribute component image can be provided to the video coder 120a for encoding as discussed above with respect to FIG.1. [0088] Similarly, the mapping topology update module 1602 can update the mapping component images in accordance with the connectivity simplification. The mapping topology update module 1602 can use the analysis results generated by the reconstructed geometry analysis module 804 which can include the vertices that are selected to be removed and, in some examples, the triangles to be merged. The mapping component coordinates (U,V) can be updated according to the changes of an attribute component image. The mapping coordinate associated with the vertex P in the original attribute image is updated with transformed mapping coordinates associated with vertex P’. [0089] The updated mapping component images may be provided to the video coder 120d for encoding as discussed above with respect to FIG. 1. The multiplexer 122 may then multiplex the encoded information to generate the coded mesh frame sequence 1610 in a way similar to that described with respect to FIG.1. Because of the attribute update and mapping update in addition to the connectivity simplification, the coded mesh frame sequence 1610 is different from the coded mesh frame sequence 124 generated using the encoder system 100 and the coded mesh frame sequence 810 generated using the encoder system 800. [0090] FIG. 18 illustrates an example decoder system 1800 for mesh decoding with connectivity refinement and mapping refinement, according to various embodiments of the present disclosure. The decoder system 1800 shown in FIG. 18 includes various components or modules that are similar to those in FIG.11. In addition, the decoder system 1800 includes a mapping refinement module 1802 to recover the mapping information that was updated at the encoder before the mapping information is sent to the mesh reconstruction module 226 for reconstructing the mesh frame sequence 1804. [0091] In some examples, the mapping refinement module 1802 can adjust the position of the vertex P’ in the attribute component image according to the equation (1). For example, the mapping refinement module 1802 can estimate the transformation defined in equation (1) with P being the reconstructed point position and P’ being the projected point position. Based on the estimated transformation, the transform defined in equation (6) is applied to U and V coordinates of the reconstructed mapping component images. The refined mapping information can be used to reconstruct the mesh frames along with other decoded components. ^ [0092] Referring now to FIG. 19, FIG. 19 depicts an example of a process 1900 for mesh coding with connectivity simplification and attribute and mapping adjustment, according to some embodiments of the present disclosure. One or more computing devices implement operations depicted in FIG. 1900 by executing suitable program code. For example, the encoder system 1600 in FIG. 16 may implement the operations depicted in FIG. 1900 by executing the corresponding program code. For illustrative purposes, the process 1900 is described with reference to some examples depicted in the figures. Other implementations, however, are possible. [0093] At block 1902, the process 1900 involves accessing a dynamic mesh to be encoded. As discussed above, the dynamic mesh may be represented as uncompressed mesh frame sequence that include mesh frames. A mesh frame is a data format that describes 3D content (e.g., 3D objects) in a digital representation as a collection of geometry, connectivity, attribute, and attribute mapping information. Each mesh frame is characterized by a presentation time and duration. A mesh frame sequence (e.g., sequence of mesh frames) forms a dynamic mesh video. The uncompressed mesh frame sequence can be segmented into segmented mesh data. Based on the segmented mesh data, the encoder system 1600 can generate attribute component images, geometry component images, connectivity component images, and mapping component images. [0094] At block 1904, the process 1900 involves encoding the geometry component images using a video encoder to generate a geometry component bitstream as described in detail with respect to FIG.1. At block 1906, the process 1900 involves decoding the encoded geometry component images to generate reconstructed geometry component images. The decoding is performed using a video decoder that corresponds to the video encoder used to the encode the geometry component images. As discussed above, in examples where the video encoder (e.g., video encoder 120b) is a lossy encoder, due to prediction and quantization, the reconstructed geometry component images will be different from the geometry component images. As such, the reconstructed position of the decoded vertexes of the mesh may differ from the original position of the vertexes of the mesh. As a result, neighboring faces may degenerate into simplified topology. [0095] At block 1908, the process 1900 involves determining a face containing a vertex in the reconstructed geometry component images to be removed from connectivity component images of the dynamic mesh. At block 1910, the process 1900 involves updating the connectivity component images of the dynamic mesh to simplify the connectivity information. Blocks 1908 and 1910 are similar to block 1408 and block 1410, respectively, described with regard to FIG.14. [0096] At block 1912, the process 1900 involves updating the attribute component images and the mapping component images of the dynamic mesh. As discussed above in detail with respect to FIG. 16, the encoder system 1600 can estimate the transformation applied to the vertices in the degenerated geometry information by the lossy compression and apply the same transformation to the corresponding faces in the attribute images. The mapping component coordinates (U,V) can be updated according to the changes of an attribute component image. The mapping coordinate associated with the vertex P in the original attribute image is updated with transformed mapping coordinates associated with vertex P’. [0097] At block 1914, the process 1900 involves encoding the updated connectivity component images to generate a connectivity component bitstream, encoding the updated attribute component images to generate an attribute component bitstream, and encoding the updated mapping component images to generate a mapping component bitstream. As discussed above in detail with respect to FIG.1, the encoding may involve using video encoders to generate the respective component bitstreams. At block 1916, the process 1900 involves generating a coded mesh bitstream by including at least the geometry component bitstream, the connectivity component bitstream, the attribute component bitstream, and the mapping component bitstream. For example, the coded mesh bitstream can be generated by multiplexing the mesh bitstream payload, that includes the encoded component bitstreams with a mesh bitstream header. [0098] Referring now to FIG. 20, FIG. 20 depicts an example of a process 2000 for decoding a coded mesh bitstream with connectivity refinement and mapping refinement, according to some embodiments of the present disclosure. One or more computing devices implement operations depicted in FIG.2000 by executing suitable program code. For example, the decoder system 1800 in FIG. 18 may implement the operations depicted in FIG. 2000 by executing the corresponding program code. For illustrative purposes, the process 2000 is described with reference to some examples depicted in the figures. Other implementations, however, are possible. [0099] At block 2002, the process 2000 involves accessing a coded mesh bitstream of a dynamic mesh for decoding. The coded mesh bitstream is encoded with connectivity simplification and attribute and mapping updates described above with respect to FIG.19. The coded mesh bitstream can include a geometry component bitstream, an attribute component bitstream, a connectivity component bitstream, and a mapping component bitstream. [0100] At block 2004, the process 2000 involves reconstructing geometry information of the dynamic mesh from the geometry component bitstream in the coded mesh bitstream. The reconstructed geometry information includes data specifying vertices of the dynamic mesh. As discussed above in detail with respect to FIG.2, the geometry information of the dynamic mesh can be reconstructed from the geometry component bitstream by applying a video decoder to the geometry component bitstream to generate reconstructed geometry component images which is then used to reconstruct the ordered list of vertex coordinates in the geometry information. [0101] At block 2006, the process 2000 involves reconstructing connectivity information of the dynamic mesh from the connectivity component bitstream in the coded mesh bitstream. At block 2008, the process 2000 involves refining the reconstructed connectivity information based on the reconstructed geometry information to generate refined connectivity information. Blocks 2006 and 2008 are similar to block 1506 and block 1508, respectively, described with regard to FIG.15. [0102] At block 2010, the process 2000 involves reconstructing the mapping information from the mapping component bitstream and refining the reconstructed mapping information. As discussed above in detail with respect to FIG. 2, the mapping information of the dynamic mesh can be reconstructed from the mapping component bitstream by applying a video decoder to the mapping component bitstream to generate reconstructed mapping component images which are then used to reconstruct the ordered list of projected vertex attribute coordinate information. The reconstructed mapping information can be refined by the mapping refinement module 1802 via adjusting the position of the vertex P’ in the attribute component images according to equation (1). For example, the mapping refinement module 1802 can update the mapping component images to reflect the distortions caused by the topology simplification. The mapping refinement module 1802 can estimate the transformation defined in equation (1) with P being the reconstructed point position and P’ being the projected point position. Based on the estimated transformation, the transform defined in equation (6) is applied to U and V coordinates of the reconstructed mapping image component. [0103] At block 2012, the process 2000 involves reconstructing the dynamic mesh based on the reconstructed geometry information, the refined connectivity information, and the refined mapping information as well as other information including the decoded attribute information and so on. At block 2014, the process 2000 involves causing the reconstructed dynamic mesh to be rendered for display. For example, the reconstructed dynamic mesh can be transmitted to a device or a module configured to render the 3D object represented by the reconstructed dynamic mesh to generate rendered images or video for display. [0104] FIG. 21 illustrates an example of an encoder system 2100 for mesh coding with connectivity simplification and attribute and mapping adjustment, according to various embodiments of the present disclosure. The encoder system 2100 shown in FIG. 21 includes various components or modules that are similar to those in FIG.16. Compared with FIG.16, the encoder system 2100 does not include the attribute image update module 1604 for updating the attribute images. In other words, the attribute component images are encoded by the video encoder 120a. The multiplexer 122 can multiplex the encoded information to generate the coded mesh frame sequence 2110 in a way similar to that described with respect to FIG. 1. Because of the attribute update is not performed at the encoder along with the connectivity simplification and mapping update, the coded mesh frame sequence 2110 is different from the coded mesh frame sequence 124 generated using the encoder system 100, the coded mesh frame sequence 810 generated using the encoder system 800, and the coded mesh frame sequence 1610 generated using the encoder system 1600. Not updating the attribute image at the encoder side allows the encoder system 2100 to generate the coded mesh frame sequence 2110 faster than the encoder system 1600 shown in FIG.16. [0105] FIG. 22 illustrates an example decoder system 2200 for mesh decoding with connectivity refinement, mapping refinement, and attribute refinement, according to various embodiments of the present disclosure. The decoder system 2200 shown in FIG. 22 includes various components or modules that are similar to those in FIG. 18. In addition, the decoder system 2200 includes an attribute refinement module 2202 to refine and transform the decoded attribute image according to the distortions occurred to the geometry information before the attribute image is sent to the mesh reconstruction module 226 for reconstructing the mesh frame sequence 2204. [0106] In some examples, the attribute refinement module 2202 estimates the transformation applied to each degenerated triangular face in a way similar to the estimation of the transformation performed by the attribute image update module 1604. For example, the attribute refinement module 2202 can estimate the affine transformation T() according Eqns. (1)-(5) with P being the original reconstructed point position and P’ being the projected point position. An inverse of the estimated affine transformation can be applied to the decoded attribute image. For example, if the estimated transformation is a shear transform with the shear transform parameters ^^ ^ , the inverse shear transform can be applied to the decoded attribute image according to the following: ൦ Here, represents reconstructed mapping component coordinates associated with reconstructed vertex ^^’ of the decoded attribute image and represents the refined mapping component coordinates associated with projected vertex ^ of the refined attribute image. [0107] Alternative, or in addition to, the attribute refinement filter in Eqn. (6), a 1- or 2- dimensional interpolation transformation filter, such as a bilinear interpolation transformation filter, a lanczos interpolation transformation filter, can be used to refine the face of the attribute component image corresponding to the degenerated face on the simplified topology. The parameters of the filter, such as the coefficients, positions, and size of the filter can be signaled in the coded mesh bitstream. For examples, the parameters can be signaled using supplemental enhancement information message or by indicating a filter index in a look-up table. The refined attribute component image can be used to reconstruct the mesh frames along with other decoded components as discussed above with respect to FIG.2. [0108] Referring now to FIG. 23, FIG. 23 depicts an example of a process 2300 for mesh coding with connectivity simplification and mapping adjustment without attribute adjustment, according to some embodiments of the present disclosure. One or more computing devices implement operations depicted in FIG.2300 by executing suitable program code. For example, the encoder system 2100 in FIG. 21 may implement the operations depicted in FIG. 2300 by executing the corresponding program code. For illustrative purposes, the process 2300 is described with reference to some examples depicted in the figures. Other implementations, however, are possible. [0109] At block 2302, the process 2300 involves accessing a dynamic mesh to be encoded. As discussed above, the dynamic mesh may be represented as uncompressed mesh frame sequence that include mesh frames. A mesh frame is a data format that describes 3D content (e.g., 3D objects) in a digital representation as a collection of geometry, connectivity, attribute, and attribute mapping information. Each mesh frame is characterized by a presentation time and duration. A mesh frame sequence (e.g., sequence of mesh frames) forms a dynamic mesh video. The uncompressed mesh frame sequence can be segmented into segmented mesh data. Based on the segmented mesh data, the encoder system 2100 can generate attribute component images, geometry component images, connectivity component images, and mapping component images. [0110] At block 2304, the process 2300 involves encoding the geometry component images using a video encoder to generate a geometry component bitstream as described in detail with respect to FIG.1. At block 2306, the process 2300 involves decoding the encoded geometry component images to generate reconstructed geometry component images. The decoding is performed using a video decoder that corresponds to the video encoder used to the encode the geometry component images. As discussed above, in examples where the video encoder (e.g., video encoder 120b) is a lossy encoder, due to prediction and quantization, the reconstructed geometry component images will be different from the geometry component images. As such, the reconstructed position of the decoded vertexes of the mesh may differ from the original position of the vertexes of the mesh. As a result, neighboring faces may degenerate into simplified geometry. [0111] At block 2308, the process 2300 involves determining a face containing a vertex in the reconstructed geometry component images to be removed from connectivity component images of the dynamic mesh. At block 2310, the process 2300 involves updating the connectivity component images of the dynamic mesh to simplify the connectivity information. Blocks 2308 and 2310 are similar to block 1408 and block 1410, respectively, described with regard to FIG.14. [0112] At block 2312, the process 2300 involves updating the mapping component images of the dynamic mesh. As discussed above in detail with respect to FIG.21, the encoder system 2100 can update the mapping component images by adjusting mapping coordinates associated with the vertices for the simplified connectivity. [0113] At block 2314, the process 2300 involves encoding the updated connectivity component images to generate a connectivity component bitstream and encoding the updated mapping component images to generate a mapping component bitstream. As discussed above in detail with respect to FIG.1, the encoding may involve using video encoders to generate the respective component bitstreams. At block 2316, the process 2300 involves generating a coded mesh bitstream by including at least the geometry component bitstream, the connectivity component bitstream, and the mapping component bitstream. For example, the coded mesh bitstream can be generated by multiplexing the mesh bitstream payload, that includes the encoded component bitstreams with a mesh bitstream header. [0114] Referring now to FIG. 24, FIG. 24 depicts an example of a process 2400 for decoding a coded mesh bitstream with connectivity refinement, mapping refinement, and attribute refinement, according to some embodiments of the present disclosure. One or more computing devices implement operations depicted in FIG.2400 by executing suitable program code. For example, the decoder system 2200 in FIG.22 may implement the operations depicted in FIG. 2400 by executing the corresponding program code. For illustrative purposes, the process 2400 is described with reference to some examples depicted in the figures. Other implementations, however, are possible. [0115] At block 2402, the process 2400 involves accessing a coded mesh bitstream of a dynamic mesh for decoding. The coded mesh bitstream is encoded with connectivity simplification and mapping updates without attribute updates as described above with respect to FIG. 23. The coded mesh bitstream can include a geometry component bitstream, an attribute component bitstream, a connectivity component bitstream, and a mapping component bitstream. [0116] At block 2404, the process 2400 involves reconstructing geometry information of the dynamic mesh from the geometry component bitstream in the coded mesh bitstream. The reconstructed geometry information includes data specifying vertices of the dynamic mesh. As discussed above in detail with respect to FIG.2, the geometry information of the dynamic mesh can be reconstructed from the geometry component bitstream by applying a video decoder to the geometry component bitstream to generate reconstructed geometry component images which are then used to reconstruct the ordered list of vertex coordinates in the geometry information. [0117] At block 2406, the process 2400 involves reconstructing connectivity information of the dynamic mesh from the connectivity component bitstream in the coded mesh bitstream. At block 2408, the process 2400 involves refining the reconstructed connectivity information based on the reconstructed geometry information to generate refined connectivity information. Blocks 2406 and 2408 are similar to block 1506 and block 1508, respectively, described with regard to FIG.15. [0118] At block 2410, the process 2400 involves reconstructing the attribute image from the attribute component bitstream and refining the reconstructed attribute image. As discussed above in detail with respect to FIG.2, the attribute component image of the dynamic mesh can be reconstructed from the attribute component bitstream by applying a video decoder to the attribute component bitstream to generate reconstructed attribute image. The reconstructed attribute image can be refined by applying an inverse transformation on the face corresponding to a degenerated face in the connectivity information. The transformation can be estimated based on the decoded vertex in the geometry information and the reconstructed projected vertex in the geometry information. [0119] At block 2412, the process 2400 involves reconstructing the dynamic mesh based on the reconstructed geometry information, the refined connectivity information, and the refined attribute image as well as other information including the decoded attribute information and so on. At block 2414, the process 2400 involves causing the reconstructed dynamic mesh to be rendered for display. For example, the reconstructed dynamic mesh can be transmitted to a device or a module configured to render the 3D object represented by the reconstructed dynamic mesh to generate rendered images or video for display. [0120] Computing System Example [0121] Any suitable computing system can be used for performing the operations described herein. For example, FIG. 25 depicts an example of a computing device 2500 that can implement the mesh encoder 100 of FIG.1, the mesh decoder 200 of FIG.2, the mesh encoder 800 of FIG.8, the mesh decoder 1100 of FIG.11, the mesh encoder 1600 of FIG.16, the mesh decoder 1800 of FIG. 18, the mesh encoder 2100 of FIG. 21, the mesh decoder 2200 of FIG. 22. In some embodiments, the computing device 2500 can include a processor 2512 that is communicatively coupled to a memory 2514 and that executes computer-executable program code and/or accesses information stored in the memory 2514. The processor 2512 may comprise a microprocessor, an application-specific integrated circuit (“ASIC”), a state machine, or other processing device. The processor 2512 can include any of a number of processing devices, including one. Such a processor can include or may be in communication with a computer-readable medium storing instructions that, when executed by the processor 2512, cause the processor to perform the operations described herein. [0122] The memory 2514 can include any suitable non-transitory computer-readable medium. The computer-readable medium can include any electronic, optical, magnetic, or other storage device capable of providing a processor with computer-readable instructions or other program code. Non-limiting examples of a computer-readable medium include a magnetic disk, memory chip, ROM, RAM, an ASIC, a configured processor, optical storage, magnetic tape or other magnetic storage, or any other medium from which a computer processor can read instructions. The instructions may include processor-specific instructions generated by a compiler and/or an interpreter from code written in any suitable computer- programming language, including, for example, C, C++, C#, Visual Basic, Java, Python, Perl, JavaScript, and ActionScript. [0123] The computing device 2500 can also include a bus 2516. The bus 2516 can communicatively couple one or more components of the computing device 2500. The computing device 2500 can also include a number of external or internal devices such as input or output devices. For example, the computing device 2500 is shown with an input/output (“I/O”) interface 2518 that can receive input from one or more input devices 2520 or provide output to one or more output devices 2522. The one or more input devices 2520 and one or more output devices 2522 can be communicatively coupled to the I/O interface 2518. The communicative coupling can be implemented via any suitable manner (e.g., a connection via a printed circuit board, connection via a cable, communication via wireless transmissions, etc.). Non-limiting examples of input devices 2520 include a touch screen (e.g., one or more cameras for imaging a touch area or pressure sensors for detecting pressure changes caused by a touch), a mouse, a keyboard, or any other device that can be used to generate input events in response to physical actions by a user of a computing device. Non-limiting examples of output devices 2522 include an LCD screen, an external monitor, a speaker, or any other device that can be used to display or otherwise present outputs generated by a computing device. [0124] The computing device 2500 can execute program code that configures the processor 2512 to perform one or more of the operations described above with respect to FIGS. 1-24. The program code can include the mesh encoder 100 of FIG.1, the mesh decoder 200 of FIG. 2, the mesh encoder 800 of FIG.8, the mesh decoder 1100 of FIG.11, the mesh encoder 1600 of FIG. 16, the mesh decoder 1800 of FIG. 18, the mesh encoder 2100 of FIG. 21, the mesh decoder 2200 of FIG. 22. The program code may be resident in the memory 2514 or any suitable computer-readable medium and may be executed by the processor 2512 or any other suitable processor. [0125] The computing device 2500 can also include at least one network interface device 2524. The network interface device 2524 can include any device or group of devices suitable for establishing a wired or wireless data connection to one or more data networks 2528. Non- limiting examples of the network interface device 2524 include an Ethernet network adapter, a modem, and/or the like. The computing device 2500 can transmit messages as electronic or optical signals via the network interface device 2524. [0126] General Considerations [0127] Numerous specific details are set forth herein to provide a thorough understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter may be practiced without these specific details. In other instances, methods, apparatuses, or systems that would be known by one of ordinary skill have not been described in detail so as not to obscure claimed subject matter. [0128] Unless specifically stated otherwise, it is appreciated that throughout this specification discussions utilizing terms such as “processing,” “computing,” “calculating,” “determining,” and “identifying” or the like refer to actions or processes of a computing device, such as one or more computers or a similar electronic computing device or devices, that manipulate or transform data represented as physical electronic or magnetic quantities within memories, registers, or other information storage devices, transmission devices, or display devices of the computing platform. [0129] The system or systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device can include any suitable arrangement of components that provide a result conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems accessing stored software that programs or configures the computing system from a general purpose computing apparatus to a specialized computing apparatus implementing one or more embodiments of the present subject matter. Any suitable programming, scripting, or other type of language or combinations of languages may be used to implement the teachings contained herein in software to be used in programming or configuring a computing device. [0130] Embodiments of the methods disclosed herein may be performed in the operation of such computing devices. The order of the blocks presented in the examples above can be varied—for example, blocks can be re-ordered, combined, and/or broken into sub-blocks. Some blocks or processes can be performed in parallel. [0131] The use of “adapted to” or “configured to” herein is meant as an open and inclusive language that does not foreclose devices adapted to or configured to perform additional tasks or steps. Additionally, the use of “based on” is meant to be open and inclusive, in that a process, step, calculation, or other action “based on” one or more recited conditions or values may, in practice, be based on additional conditions or values beyond those recited. Headings, lists, and numbering included herein are for ease of explanation only and are not meant to be limiting. [0132] While the present subject matter has been described in detail with respect to specific embodiments thereof, it will be appreciated that those skilled in the art, upon attaining an understanding of the foregoing, may readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, it should be understood that the present disclosure has been presented for purposes of example rather than limitation, and does not preclude the inclusion of such modifications, variations, and/or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art.
Next Patent: DYNAMIC MESH CODING WITH SIMPLIFIED TOPOLOGY