Research

Machine perception aims at building a machine learning system capable of understanding the visual signals like human. Our recent research includes self-guidance for expedited masked autoencoder training, efficient video classification based on frame sampling, effective data augmentation for referring image segmentation, comprehensive experimental study on moment localization, and more.

Due to the large volume of videos, video understanding research inherently requires large-scale computing environment. VIP lab aims at building a large-scale video data processing infrastructure, where a variety of video datasets are stored and they can be efficiently processed at scale on various tasks in computer vision.

Recent publications

• Towards a Complete Benchmark on Video Moment Localization (AISTATS'24)
• Modality-Aware Representation Learning for Zero-shot Sketch-based Image Retrieval (WACV'24)
• Mr. HiSum: A Large-scale Dataset and Benchmark for Video Highlight Detection and Summarization (NeurIPS'23)

5-min presentations

• Self-guided Masked Autoencoder (Jeongwoo Shin)
• Semi-optimal Policy and Knowledge Distillation for Video Frame Sampling (Junho Lee)
• Negative-mined Mosaic Augmentation for Referring Image Segmentation (Chaeyun Kim)
• Large-scale Video Summarization (Jihoon Han & Kwanseok Kim)
• Audio-Visual Object Identification (Sunyoo Kim)
• Comparative Study on Video Moment Localization and Beyond (Donghwa Kim)
• Implicit Neural Representations (Inseo Lee)
• Modality-Aware Representation Learning for Zero-shot Sketch-based Image Retrieval (Eunyi Lyou)
• Application on Hair Damage Detection (Kwanseok Kim)

Recently, large language models (LLMs) have enabled human-level linguistic conversation on machines, and diffusion models rapidly grow the capability of generating visual scenes. We are interested in both fundamental principles of diffusion and its applications, such as image editing. We also have worked on a few 3D vision projects, including novel view synthesis and human pose estimation.

We are also interested in applying these image/video generation or editing techniques to other domains. For example, we have worked on brain MRI generation, virtual try-on, and construction scene image editing.

Recent publications

• PoseDiff: Pose-conditioned Diffusion for Sparse View Unbounded Scene Synthesis (WACV'24)
• Towards Robust and Smooth 3D Multi-Person Pose Estimation from Monocular Videos in the Wild (ICCV'23)
• Learning to Wear: Details-Preserved Virtual Try-on via Disentangling Clothes and Wearer (BMVC'22)

5-min presentations

• Isometric Representation Learning for Disentangled Diffusion Models (Junho Lee & Jaehoon Hahm)
• Multi-vector Image Representation (Suho Ryu)
• Image Verb Editing (Kihyun Kim)
• Learning Equivariant Dynamics in the Latent Space of Diffusion Models (Sunghyun Kim)
• Multi-concept Subject-guided Image Editing (Sangho Lee)
• Large-scale Image-Text Paired Dataset Creation for Identity-preserving Modeling (Junho Lee)
• Diffusion-based Brain MRI Generation (Wonyoung Jang)

Having experience both on computer vision and recommendation systems, VIP Lab is one of the best places to work on video recommendation. Taking advantage of this, we extensively work on content-based recommendation systems, particularly multimedia features on movie and music domain.

We are also interested in other recommendation research topics, including recommendation systems leveraging large language models (LLMs), recommendations for local communities, sequential recommendations, and graph-based algorithms.

Recent publications

• Content-based Graph Reconstruction for Cold-start item recommendation (SIGIR'24)
• S-Walk: Accurate and Scalable Session-based Recommendation with Random Walks (WSDM'22)
• Session-aware Linear Item-Item Models for Session-based Recommendation (WWW'21)

5-min presentations

• Adaptive Graph Convolution for Top-N Recommendation (Eungi Kim & Chanwoo Kim)
• Content-based Graph Reconstruction for Cold-start Item Recommendation (Jinri Kim & Sewon Lee)
• Recommendation based on Large Language Models (LLMs) (Yujin Jeon)

Machine learning is widely applied not just to commercial applications but to scientific problems. Recently, machine learning plays a key role in scientific discoveries beyond the scale that human brain can handle. Climate science is a good example where machine learning and computer vision technology helps natural science. We work on detecting, tracking, and forecasting extreme climate events, such as hurricanes.

We also work on molecular structure estimation and drug discovery, with collaborators both from academia and industry, developing machine learning models that aim to understand general physical rules of our world.

Recent publications

• Towards Physically Reliable Molecular Representation Learning (UAI'23)
• Learning to Focus and Track Extreme Climate Events (BMVC'19)
• Tracking and Forecasting Extreme Climate Events (WACV'19)

Presentations

• Towards Physically Reliable Molecular Representation Learning (Seunghoon Yi)