Overview
Multimodal Hoax Detection is a research prototype developed in the context of the INDONERIS competition. It explores Indonesian misinformation detection using news text, related images, and information derived from audio transcripts.
The current development stage focuses on the data foundation: collection, cleaning, deduplication, normalization, and checks for potential label leakage. That foundation is necessary before multimodal fusion strategies can be evaluated fairly.
Problem and objectives
A hoax classifier can learn signals that do not represent the underlying claim, such as phrases associated with fact-checking websites or news repeated across training and test sets. These patterns can make evaluation look stronger than performance on new cases.
The project aims to:
- Integrate news and fact-checking records into a consistent schema.
- Reduce duplicates and patterns that may reveal labels.
- Prepare transcripts as an additional information source.
- Establish a traceable foundation for comparable experiments.
Data preparation pipeline
Records from several sources are aligned to a shared schema, then processed through text cleaning, label validation, missing-value checks, and deduplication.
Duplicate detection includes exact matching and approximate similarity search with MinHash/LSH. This helps identify closely related content even when wording differs.
Processing manifests retain record counts at each stage, source and label distributions, and cleaning parameters. Dataset changes can therefore be reviewed through the process that produced them.
Modality preparation
The preparation stage includes:
| Collection | Preparation output |
|---|---|
| News and TurnBackHoax | 3,068 integrated records |
| Transcripts | 992 normalized transcripts |
News text provides the main textual foundation. Images and transcripts are prepared as additional modalities for further development.
The collections are not assumed to have a one-to-one correspondence. A multimodal dataset still requires explicit mapping between claims, images, transcripts, sources, and labels.
Developed components
Development includes text dataset integration, transcript normalization, filtering of potential label leakage, quality checks, and a Streamlit exploration interface.
Python and Pandas support data processing, while Whisper is part of the audio information preparation context. Separate preparation stages make data issues easier to investigate before model complexity is added.
Evaluation and status
Evaluation at this stage focuses on data quality: label validity, input completeness, source distribution, schema consistency, and duplication. Record counts describe preparation outputs and are not classification performance metrics.
Multimodal feature extraction remains in development. Fusion training and comparative evaluation are planned next stages; final classification metrics are not presented as completed results.
Main challenges
The main challenges are source leakage, class imbalance, transcript noise, and modality alignment. Data splitting must account for content similarity so that evaluation does not depend on repeated news stories.
Misinformation detection also requires context and evidence review. Future model predictions should support an analytical process that remains open to inspection.
Next steps
The next steps are to establish leakage-resistant splits, align samples across modalities, evaluate text, image, and audio or transcript baselines, and compare them with fusion models.
Further evaluation should examine generalization across sources and time periods, with error analysis by claim type. The objective is to understand performance on cases beyond the training collection.
Code and documentation
Project scope
The current development scope covers dataset integration, deduplication, transcript normalization, leakage checks, and preparation for multimodal experiments.