MMAISB 2025
The 1st International Workshop for Multi-Modal AI Safety Benchmark (MMAISB)
February 9 (Sunday), 2025, Nexus Resort & Spa Karambunai, Kota Kinabalu, Malaysia
In conjunction with the IEEE BigComp 2025 - 12th IEEE International Conference on Big Data and Smart Computing
Generative AI techniques have opened new business opportunities that can change and impact our ordinary lives. However, evaluating and measuring the safety and faithfulness of the AI-generated contents remains generally under-explored. The workshop of this year introduces some research efforts to construct benchmark datasets for evaluating safety and faithfulness of generative AI, currently undergoing in South Korea by TTA (Telecommunications Technology Association) with the collaboration of researchers and practitioners from KAIST, University of Seoul, Keimyung University, SelectStar, and Kakao. Our ultimate goal is to establish a methodology for constructing benchmark datasets, evaluation metrics, protocols and annotations to evaluate the safety and faithfulness of multimodal generative AI.
Organizing Committee
General Co-chairs
- Junho Shin (TTA, Korea)
- Ho-Jin Choi (KAIST, Korea)
Organizing Co-chairs
- Joon Ho Kwak (TTA, Korea)
- Jeongyun Han (University of Seoul, Korea)
- Soohyun Cho (Keimyung University, Korea)
Program Co-chairs
- EunYoung Byun (TTA, Korea)
- Jemin Hwang (TTA, Korea)
- Joyce Jiyoung Whang (KAIST, Korea)
- Joseph Seering (KAIST, Korea)
- HwaJung Hong (KAIST, Korea)
- Uichin Lee (KAIST, Korea)
- Juho Kim (KAIST, Korea)
- Bohyun Kim (SelectStar, Korea)
- Seokyeon Ko (SelectStar, Korea)
- Taeho Kim (SelectStar, Korea)
- Kyunghoon Kim (Kakao, Korea)
- Myungsik Ha (Kakao, Korea)
Local Arrangement Co-chairs
- Chae-Gyun Lim (KAIST, Korea)
- Seung-Ho Han (KAIST, Korea)
- Yechan Hwang (KAIST, Korea)
- Eojin Joo (KAIST, Korea)
Program
13:30 - 17:50, February 9 (Sunday), 2025
• Venue: MASOKOH 2 (Nexus Resort & Spa Karambunai, Kota Kinabalu, Malaysia)
13:30 - 13:40 Opening
Session Chair Seung-Ho Han (KAIST, Korea)
- Welcoming Address
13:40 - 14:40 LLM Benchmark Datasets
Session Chair Chae-Gyun Lim (KAIST, Korea)
- Between Assistance and Reliance: Assessing the Fine Line in AI Dependence
- AI for the Vulnerable: Incorporating User Context in LLM Safety
- An Empirical Study on Assessing LMM Privacy Risks and Exploring Measures to Mitigate Privacy Concerns
14:40 - 15:40 Multi-Modal Benchmark Datasets
Session Chair Yechan Hwang (KAIST, Korea)
- Exploring LMM Risk Assessment Datasets Across Diverse Modalities
- Multi-Modal Risk Detection: Combining Video and Image Processing for Safety Analysis
- From Text to Speech: Building Multimodal Datasets for AI Safety Assessment
- A Recipe for Multilingual and Multi-Modal Red-Teaming
15:40 - 16:00 Coffee Break
16:00 - 17:00 Construction Process and Quality Management
Session Chair Soohyun Cho (Keimyung University, Korea)
- A Quality Assurance Framework for Multimodal Assessment Datasets on AI Risk Factors
- Enhancing Keyphrase Generation in AI Risk Evaluation Datasets through Prompt Engineering
- An Analysis of Unsafe Responses Across Large Language Models
- Defining and Monitoring Potential Risks of Large Action Models and AI Agents
17:00 - 17:45 Socio-Cultural Aspects and Applications
Session Chair Jeongyun Han (University of Seoul, Korea)
- Exploring Korean Socio-Cultural Factors for Database Development and AI Risk Testing
- An Empirical Study on AI Safety Assessment Using a Large Language Model
- Trustworthy AI: Ensuring Quality and Safety in Large Language Model (LLM) Application
17:45 - 17:50 Closing Statements
- Closing Remarks
Sponsors
Contact
All questions about submissions should be emailed to chairs Chae-Gyun Lim (rayote@kaist.ac.kr) or Seung-Ho Han (seunghohan@kaist.ac.kr).