Recent Research
The goal is to establish a unified framework for video anomaly detection, advancing precision in
identifying and localizing abnormal events across dynamic scenes while enabling interpretable
analysis
of complex visual patterns.
Starting from real-world applications in surveillance and social media analysis, we introduce
Hawkeye (Zhang
et al., ACM MM'24)
, the first scene-enhanced video-language model designed for anomaly detection. Hawkeye
integrates multimodal context
(visual-textual-temporal cues) to recognize subtle anomalies and pinpoint their temporal
boundaries in untrimmed videos,
This work lays a critical foundation for event typing and spatiotemporal
localization in short video understanding.
Building on this, we investigate low-resource scenarios where annotated anomaly data is scarce.
Our Continuous
Attention Modeling method (Zhang et
al.,JOS'23) enhances adaptability by capturing long-range
dependencies
in sparse anomaly signals. Further, we extend Hawkeye with self-supervised learning to uncover
latent patterns
across unlabeled videos, improving generalization to unseen anomaly types.
To scale solutions, we construct a benchmark suite combining large-scale anomaly annotations and
instruction-tuned datasets.
This addresses the challenge of diverse event types (e.g., accidents, unusual behaviors) and
supports downstream tasks
like explainable reasoning (Liu
et al., COLING'2024).
Papers:
(Zhang
et al., ACM MM'24)
(Zhang et al.,JOS'23)
(Liu
et al., COLING'2024).
The goal is to establish an emotion-driven framework for multimodal large language model (LLM)
video generation
, achieving cross-modal semantic alignment and high-fidelity emotional action synthesis while
enhancing
controllability and realism in generated content.
Starting from the core demand of multimodal interaction, we propose LLM-Guided Emotion-Action
Synthesis
(Yu
et al., ACM MM'2024),
the first method to jointly model text emotion semantics and human motion sequences for
emotionally rich video generation
This work enables precise conveyance of complex emotions (e.g., "excited waving") in synthetic
videos, providing critical support for emotion-aware short video editing.
To address cross-modal alignment challenges, we introduce two innovations:
1) A Trustworthy Reflection Mechanism
(Liu
et al., COLING'24)
that dynamically calibrates generation intent with emotional
labels, improving semantic coherence in action outputs;
2) Latent Diffusion Models
(Luo
et al., COLING'24)
that jointly model emotion-themed distributions across text, audio,
and motion modalities in latent space, overcoming efficiency bottlenecks of pixel-level
generation.
To scale practical applications, we construct the first benchmark dataset for emotion-driven
action generation,
containing 100K+ emotion-annotated human motion sequences and multimodal instruction pairs.
Based on this, we propose
AutoEmoDirector, a framework enabling users to edit video emotion intensity and motion styles
via natural language instructions
(e.g., "transform a joyful dance into a sorrowful stroll"), significantly lowering the barrier
for video content creation.
Papers:
(Yu
et al., ACM MM'2024)
(Luo
et al., COLING'24)
(Liu
et al., COLING'24)
