Paper: 2604.21885 Authors: Research Team Categories: cs.CL, cs.CV

Problem

Current event extraction systems:

  • Focus only on text, ignoring rich visual context
  • Cannot handle multimodal real-world documents
  • Lack unified frameworks for cross-modal event understanding

MODEE Framework

Multimodal Open-Domain Event Extraction with three key components:

1. Multimodal Event Schema

  • Unified schema for text, image, and video events
  • Cross-modal event links and relationships
  • Temporal and causal event ordering

2. Multimodal Encoder

  • Vision-language fusion for event detection
  • Cross-attention between modalities
  • Context-aware event boundary detection

3. Benchmark Dataset

  • 10K+ documents with text + images/videos
  • 50+ event types across domains
  • Rich annotations for training and evaluation

Results

MetricText-OnlyMultimodal (MODEE)
Event Detection F172.3%84.7%
Role Extraction F168.1%79.2%
Cross-modal ConsistencyN/A91.3%

Takeaways

  • Visual context significantly improves event extraction
  • Multimodal models capture events text-only systems miss
  • Cross-modal reasoning is essential for real-world event understanding

论文: 2604.21885 作者: 研究团队 分类: cs.CL, cs.CV

问题

当前的事件抽取系统：

  • 仅关注文本，忽略丰富的视觉上下文
  • 无法处理多模态真实世界文档
  • 缺乏跨模态事件理解的统一框架

MODEE框架

多模态开放域事件抽取，包含三个关键组件：

1. 多模态事件模式

  • 文本、图像和视频事件的统一模式
  • 跨模态事件链接和关系
  • 时间和因果事件排序

2. 多模态编码器

  • 用于事件检测的视觉-语言融合
  • 模态间交叉注意力
  • 上下文感知的的事件边界检测

3. 基准数据集

  • 10K+ 文档，包含文本 + 图像/视频
  • 50+ 跨领域事件类型
  • 丰富的训练和评估标注

实验结果

指标仅文本多模态 (MODEE)
事件检测 F172.3%84.7%
角色抽取 F168.1%79.2%
跨模态一致性N/A91.3%

要点总结

  • 视觉上下文显著改善事件抽取
  • 多模态模型捕捉仅文本系统遗漏的事件
  • 跨模态推理对真实世界事件理解至关重要