Tao Yu's Homepage

Audio Matters Too! Enhancing Markerless Motion Capture with Audio Signals for String Performance Capture

Yitong Jin^*, Zhiping Qiu^*, Yi Shi, Shuangpeng Sun, Chongwu Wang, Donghao Pan, Jiachen Zhao, Zhenghao Liang, Yuan Wang, Xiaobing Li, Feng Yu, Tao Yu^✝, Qionghai Dai^✝

ACM SIGGRAPH 2024 & TOG

[Project Page] [Arxiv] [Youtube] [Bilibili]

Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL

Liang An, Jilong Ren, Tao Yu, Tang Hai^✝, Yichang Jia^✝, Yebin Liu^✝

Nature Communications

[Paper] [Code] [Code for Mouse] [Project]

MMVP: A Multimodal MoCap Dataset with Vision and Pressure Sensors

He Zhang^*, Shenghao Ren^*, Haolei Yuan^*, Jianhui Zhao, Fan Li, Shuangpeng Sun, Zhenghao Liang, Tao Yu^✝, Qiu Shen^✝, Xun Cao^✝

IEEE CVPR 2024

[Project Page]

HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Models

Mengcheng Li, Hongwen Zhang, Yuxiang Zhang, Ruizhi Shao, Tao Yu^✝, Yebin Liu^✝

IEEE CVPR 2024 (Highlight)

[Project Page]

OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learning

Haiyang Ying, Yixuan Yin, Jinzhi Zhang, Fan Wang, Tao Yu, Ruqi Huang, Lu Fang,

IEEE CVPR 2024

[Project Page] [Paper]

DiffPerformer: Iterative Learning of Consistent Latent Guidance for Diffusion-based Human Video Generation

Chenyang Wang, Zerong Zheng, Tao Yu, Xiaoqian Lv, Bineng Zhong, Shengping Zhang, Liqiang Nie

IEEE CVPR 2024

[Comming Soon]

Human Pose Estimation with General Contact

He Zhang, Jianhui Zhao, Fan Li⁺, Yitian Wu, Chao Tan, Shuangpeng Sun, Yaohua Wu, You Li, Tao Yu⁺

Springer CVMJ

[Coming Soon]

ProbIBR: Fast Image-Based Rendering with Learned Probability-Guided Sampling

Yuemei Zhou, Tao Yu, Zerong Zheng, Gaochang Wu, Guihua Zhao, Wenbo Jiang, Ying Fu, Yebin Liu

IEEE TVCG

[Paper]

Neural Physical Simulation with Multi-resolution Hash Grid Encoding

Haoxiang Wang, Tao Yu^✝, Tianwei Yang, Hui Qiao^✝, Qionghai Dai

AAAI 2024 [Oral Presentation]

[Presentation]

Triangulation Residual Loss for Data-efficient 3D Pose Estimation

Jiachen Zhao, Tao Yu^✝, Liang An, Yipeng Huang, Fang Deng, Qionghai Dai^✝

NeurIPS 2023

[Paper] [Code]

PARF: Primitive-Aware Radiance Fusion for Indoor Scene Novel View Synthesis

Haiyang Ying, Baowei Jiang, Jinzhi Zhang, Jun Zhang, Hongguang Xing, Di Xu, Tao Yu^✝, Qionghai Dai, Lu Fang^✝

ICCV 2023

[Project Page] [Paper]

OPAL: Occlusion Pattern Aware Loss for Unsupervised Light Field Disparity Estimation

Peng Li*, Jiayin Zhao*, Jingyao Wu, Chao Deng, Yuqi Han, Haoqian Wang^✝, Tao Yu^✝,

IEEE-TPAMI

[Project] [Arxiv] [Code]

StyleAvatar: Real-time Photo-realistic Neural Portrait Avatar from a Single Video

Lizhen Wang, Xiaochen Zhao, Jingxiang Sun, Yuxiang Zhang, Hongwen Zhang, Tao Yu^✝, Yebin Liu^✝

ACM SIGGRAPH 2023

[Project] [Arxiv]

Learning Visibility Field for Detailed 3D Human Reconstruction and Relighting

Ruichen Zheng*, Peng Li*, Haoqian Wang, Tao Yu

IEEE CVPR 2023

[Project] [Arxiv]

Controllable Free Viewpoint Video Reconstruction Based on Neural Radiance Fields and Motion Graphs

He Zhang, Fan Li, Jianhui Zhao, Chao Tan, Dongming Shen, Yebin Liu, Tao Yu

IEEE TVCG

[Paper]

DeepCloth: Neural Garment Representation for Shape and Style Editing

Zhaoqi Su, Tao Yu, Yangang Wang, Yebin Liu

IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI 2022)

[Project] [Arxiv] [Dataset]

HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling

Zhongang Cai*, Daxuan Ren*, Ailing Zeng*, Zhengyu Lin*, Tao Yu*, Wenjia Wang*, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, Fangzhou Hong, Mingyuan Zhang, Chen Change Loy, Lei Yang^✝, Ziwei Liu^✝

European Conference on Computer Vision (ECCV 2022 Oral Presentation)

[Project] [Arxiv] [Dataset] [Code] [Demo]

Geometry-aware Single-image Full-body Human Relighting

Chaonan Ji, Tao Yu, Kaiwen Guo, Jingxin Liu, Yebin Liu

European Conference on Computer Vision (ECCV 2022)

[Project] [Arxiv]

GIMO: Gaze-Informed Human Motion Prediction in Context

Yang Zheng, Yanchao Yang, Kaichun Mo, Jiaman Li, Tao Yu, Yebin Liu, Karen Liu, Leonidas Guibas,

European Conference on Computer Vision (ECCV 2022)

[Project] [Arxiv]

HVTR: Hybrid Volumetric-Textural Rendering for Human Avatars

Tao Hu, Tao Yu, Zerong Zheng, He Zhang, Yebin Liu, Matthias Zwicker

International Conference on 3D Vision (3DV 2022)

[Project] [Arxiv]

Interacting Attention Graph for Single Image Two-Hand Reconstruction

Mengcheng Li, Liang An, Hongwen Zhang, Lianpeng Wu, Feng Chen, Tao Yu, Yebin Liu

IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2022 Oral Presentation)

[Project] [Arxiv]

Structured Local Radiance Fields for Human Avatar Modeling

Zerong Zheng, Han Huang, Tao Yu, Hongwen Zhang, Yandong Guo, Yebin Liu

IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2022)

[Arxiv] [Project Page]

DoubleField: Bridging the Neural Surface and Radiance Fields for High-fidelity Human Reconstruction and Rendering

Ruizhi Shao, Hongwen Zhang, He Zhang, Mingjia Chen, Yanpei Cao, Tao Yu, Yebin Liu

IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2022)

[Project] [Arxiv]

FaceVerse: a Fine-grained and Detail-changeable 3D Neural Face Modelfrom a Hybrid Dataset

Lizhen Wang, Zhiyuan Chen, Tao Yu, Chenguang Ma, Liang Li, Yebin Liu

IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2022)

[Project]

Robust and Accurate 3D Self-portraits in Seconds

Zhe Li*, Tao Yu*, Zerong Zheng, Yebin Liu

IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI 2022)

[Paper] [Youtube Coming Soon]

Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras

Yuxiang Zhang, Zhe Li, Liang An, Mengcheng Li, Tao Yu^✝, Yebin Liu^✝

IEEE International Conference on Computer Vision (ICCV 2021)

[Project] [Arxiv]

DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Cameras

Yang Zheng*, Ruizhi Shao*, Yuxiang Zhang, Tao Yu, Zerong Zheng, Qionghai Dai, Yebin Liu

IEEE International Conference on Computer Vision (ICCV 2021)

[Project] [Arxiv]

Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors

Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, Yebin Liu

IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2021 Oral Presentation)

[Project] [Youtube]

Deep Implicit Templates for 3D Shape Representations

Zerong Zheng, Tao Yu, Qionghai Dai, Yebin Liu

IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2021 Oral Presentation)

[Project] [Youtube]

POSEFusion: Pose-guided Selective Fusion for Single-view Human Volumetric Capture

Zhe Li, Tao Yu, Zerong Zheng, Kaiwen Guo, Yebin Liu

IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2021 Oral Presentation)

[Project] [Youtube]

PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction

Zerong Zheng, Tao Yu, Yebin Liu, Qionghai Dai

IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI 2020)

[Project] [Arxiv]

MulayCap: Multi-layer Human Performance Capture Using A Monocular Video Camera

Zhaoqi Su, Weilin Wan, Tao Yu, Lingjie Liu, Lu Fang, Wenping Wang, Yebin Liu

IEEE Transactions on Visualization and Computer Graphics (TVCG 2020)

[Project] [Arxiv]

NormalGAN: Learning Detailed 3D Human from a Single RGB-D Image

Lizhen Wang, Xiaochen Zhao, Tao Yu, Songtao Wang, Yebin Liu

European Conference on Computer Vision (ECCV 2020)

[Project] [Arxiv]

RobustFusion: Human Volumetric Capture with Data-driven Visual Cues using a RGBD Camera

Zhuo Su, Xu Lan, Zerong Zheng, Tao Yu, Yebin Liu, Lu Fang

European Conference on Computer Vision (ECCV 2020 Spotlight Presentation)

[Paper]

4D Association Graph for Realtime Multi-person Motion Capture Using Multiple Video Cameras

Yuxiang Zhang*, Liang An*, Tao Yu, Xiu Li, Kun Li, Yebin Liu

IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2020 Oral Presentation)

[Project] [Arxiv] [Youtube]

Robust 3D self-portraits in seconds

Zhe Li, Tao Yu, Zerong Zheng, Yebin Liu

IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2020 Oral Presentation)

[Project] [Arxiv] [Video]

DeepHuman: 3D Human Reconstruction from a Single Image

Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, Yebin Liu

IEEE International Conference on Computer Vision (ICCV 2019 Oral Presentation)

[Project] [Arxiv] [Video]

UnstructuredFusion: Real-time 4D Geometry and Texture Reconstruction using Commercial RGBD Cameras

Lan Xu, Zhuo Su, Lei Han, Tao Yu, Yebin Liu, Lu Fang

IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI 2019)

[Paper] [Youtube]

SimulCap: Single-View Human Performance Capture with Cloth Simulation

Tao Yu, Zerong Zheng, Yuan Zhong, Jianhui Zhao, Qionghai Dai, Gerard Pons-Moll, Yebin Liu

IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2019)

[Project] [Arxiv] [Youtube] [Video]

HybridFusion: Real-time Performance Capture Using a Single Depth Sensor and Sparse IMUs

Zerong Zheng, Tao Yu, Hao Li, Kaiwen Guo, Qionghai Dai, Lu Fang, Yebin Liu

European Conference on Computer Vision (ECCV 2018)

[Project] [Paper] [Youtube] [Video]

DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor

Tao Yu, Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai Dai, Hao Li, Gerard Pons-Moll, Yebin Liu

IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018 Oral Presentation)

[Project] [Arxiv] [Paper] [Youtube] [Video] [Oral Presentation (Youtube)] [Software] [TechCrunch] [Sohu]

BodyFusion: Real-time Capture of Human Motion and Surface Geometry Using a Single Depth Camera

Tao Yu, Kaiwen Guo, Feng Xu, Yuan Dong, Zhaoqi Su, Jianhui Zhao, Jianguo Li, Qionghai Dai, Yebin Liu

IEEE International Conference on Computer Vision (ICCV 2017)

[Project] [Paper] [Youtube] [Video]

Real-time Geometry, Albedo and Motion Reconstruction Using a Single RGBD Camera

Kaiwen Guo, Feng Xu, Tao Yu, Xiaoyang Liu, Qionghai Dai, Yebin Liu

ACM Transactions on Graphics (Present in SIGGRAPH 2017)

[Project] [Paper] [Youtube] [Video]

Tao Yu | 于涛

Publication

Audio Matters Too! Enhancing Markerless Motion Capture with Audio Signals for String Performance Capture

Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL

MMVP: A Multimodal MoCap Dataset with Vision and Pressure Sensors

HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Models

OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learning

DiffPerformer: Iterative Learning of Consistent Latent Guidance for Diffusion-based Human Video Generation

Human Pose Estimation with General Contact

ProbIBR: Fast Image-Based Rendering with Learned Probability-Guided Sampling

Neural Physical Simulation with Multi-resolution Hash Grid Encoding

Triangulation Residual Loss for Data-efficient 3D Pose Estimation

PARF: Primitive-Aware Radiance Fusion for Indoor Scene Novel View Synthesis

OPAL: Occlusion Pattern Aware Loss for Unsupervised Light Field Disparity Estimation

StyleAvatar: Real-time Photo-realistic Neural Portrait Avatar from a Single Video

Learning Visibility Field for Detailed 3D Human Reconstruction and Relighting

Controllable Free Viewpoint Video Reconstruction Based on Neural Radiance Fields and Motion Graphs

DeepCloth: Neural Garment Representation for Shape and Style Editing

HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling

Geometry-aware Single-image Full-body Human Relighting

GIMO: Gaze-Informed Human Motion Prediction in Context

HVTR: Hybrid Volumetric-Textural Rendering for Human Avatars

Interacting Attention Graph for Single Image Two-Hand Reconstruction

Structured Local Radiance Fields for Human Avatar Modeling

DoubleField: Bridging the Neural Surface and Radiance Fields for High-fidelity Human Reconstruction and Rendering

FaceVerse: a Fine-grained and Detail-changeable 3D Neural Face Modelfrom a Hybrid Dataset

Robust and Accurate 3D Self-portraits in Seconds

Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras

DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Cameras

Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors

Deep Implicit Templates for 3D Shape Representations

POSEFusion: Pose-guided Selective Fusion for Single-view Human Volumetric Capture

PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction

MulayCap: Multi-layer Human Performance Capture Using A Monocular Video Camera

NormalGAN: Learning Detailed 3D Human from a Single RGB-D Image

RobustFusion: Human Volumetric Capture with Data-driven Visual Cues using a RGBD Camera

4D Association Graph for Realtime Multi-person Motion Capture Using Multiple Video Cameras

Robust 3D self-portraits in seconds

DeepHuman: 3D Human Reconstruction from a Single Image

UnstructuredFusion: Real-time 4D Geometry and Texture Reconstruction using Commercial RGBD Cameras

SimulCap: Single-View Human Performance Capture with Cloth Simulation

HybridFusion: Real-time Performance Capture Using a Single Depth Sensor and Sparse IMUs

DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor

BodyFusion: Real-time Capture of Human Motion and Surface Geometry Using a Single Depth Camera

Real-time Geometry, Albedo and Motion Reconstruction Using a Single RGBD Camera

Experience

Tsinghua University

Tsinghua University

Tsinghua University

Tsinghua University

Beihang University

University of Southern California

Hefei University of Technolgy

Skills