Overview
IndoMemeXAI is a research project that combines text and images to classify Indonesian memes as hateful or non-hateful. It studies how the relationship between modalities, including incongruity between wording and imagery, affects model predictions.
Alongside fusion experiments, the project examines prediction behavior through attribution and attention analysis. The objective is to understand both performance and model behavior in Indonesian language and cultural contexts.
Research problem
A meme's meaning often emerges from the interaction between its wording and image. A seemingly neutral sentence can carry a different meaning alongside a particular visual. An image alone may also leave the target, irony, or social context unclear.
The research focuses on three questions:
- How should text and visual representations be combined?
- How much does topic information contribute to classification?
- Which input features contribute to a prediction, particularly in error cases?
Data and experiment preparation
Experiments use aiatums/INDOMEME with the following split:
| Split | Samples |
|---|---|
| Training | 3,415 |
| Validation | 603 |
| Test | 1,005 |
The primary task is binary classification using the hate label. Text preparation normalizes Unicode, URLs, mentions, hashtags, Indonesian slang, and whitespace. Captions and OCR text form the textual input. Images are resized and normalized for the visual encoder.
The split is preserved across comparisons. Topic information is included in selected variants and removed in an ablation experiment to assess its contribution.
Architecture and fusion strategies
IndoBERTweet forms textual representations, while ViT-B/16 forms visual representations from 224 x 224 images. The model combines that information before two-class classification.
The studied strategies include:
- Late fusion: combine global text and image representations.
- Cross-attention: use textual representations to attend to relevant visual information.
- ICAM: model cross-modal interaction and incongruity, with learned feature gates.
- No-topic ablation: measure the performance change when topic information is removed.
For topic-aware variants, a topic embedding is combined with the fusion output before the classification head. Each strategy is a separate experimental configuration.
Evaluation design
Evaluation considers macro-F1, accuracy, and hate-class recall. Macro-F1 provides a more balanced view across classes, while hate-class recall measures the proportion of relevant samples identified.
The final experiment uses an ensemble of five seeds: 42, 123, 2024, 7, and 999. The configuration without test-time augmentation records the following test results:
| Metric | Result |
|---|---|
| Macro-F1 | 0.80765 |
| Accuracy | 0.83085 |
| Hate-class recall | 0.70029 |
| Classification threshold | 0.57 |
These results apply to that split and experimental configuration. A training-to-test macro-F1 gap of approximately 0.16812 makes generalization an ongoing development concern.
Model behavior analysis
Explainability analysis uses Layer Integrated Gradients to assess the contribution of text tokens for a specific checkpoint. Visual analysis includes cross-attention maps, attention rollout, and incongruity heatmaps.
Perturbation analysis and ROAR provide further ways to investigate feature influence. Topic-level review and individual case studies connect aggregate metrics with errors on particular samples.
Attribution visualizations are inspection tools. Their interpretation still depends on sample context, result stability, and the model's response to changes in its inputs.
Findings and limitations
In the ensemble comparison, the configuration without test-time augmentation achieves a higher test macro-F1 than the augmented configuration. This finding applies to the experimental design used.
OCR quality, class imbalance, topic dependence, language variation, and distribution shift affect interpretation. Hate labels also depend on context and annotation decisions.
IndoMemeXAI remains experimental research. Use in a moderation system would require evaluation on independent data and a human review process.
Next steps
The next priorities are evaluation beyond the main split, more detailed topic-level error analysis, and explanation stability checks. Results should remain connected to their configuration, seeds, checkpoints, and samples to support reproducibility.
Code and experimental results
Project scope
The research scope covers data preparation, fusion strategy comparisons, classification evaluation, and analysis of the features influencing predictions.