Teaching Bangla AI to Say 'I Don't Know': A Dataset for Unanswerable Questions

Paper: 2603.05462 Authors: Abrar Eyasir, Tahsin Ahmed, Muhammad Ibrahim Categories: cs.CL

The Gap

Existing Bangla QA datasets assume every question has an answer in the given context. Models trained on these datasets never learn to abstain — they’ll confidently fabricate answers even when the context contains nothing relevant. SQuAD 2.0 solved this for English by adding unanswerable questions, but low-resource languages like Bangla lack equivalent resources. Prior Bangla datasets (BanglaQA, BanglaReading) contain only answerable questions, creating models that hallucinate rather than admit ignorance.

Problem: Bangla QA models hallucinate answers
   |
   v
Assumption: Models need negative examples (unanswerable questions)
   |
   v
Method: Build dataset with 43% unanswerable + adversarial distractors
   |
   v
Evidence: BERT F1 jumps 0.15 -> 0.62 after fine-tuning
   |
   v
Conclusion: Balanced answerable/unanswerable data enables abstention

The Increment

One sentence: Before NCTB-QA, Bangla QA systems would generate plausible-sounding nonsense when stumped; after, they can recognize when a question is unanswerable from context.

Core Mechanism

The dataset construction follows a three-stage pipeline. First, they extract passages from 50 official textbooks covering grades 1-12 across subjects (science, math, history, literature). Second, they generate question-answer pairs using both manual annotation and semi-automated extraction from textbook exercises. Third — and this is the critical part — they create unanswerable questions by pairing questions with passages that lack the answer but contain topically related content (adversarial distractors).

The adversarial design works like this: take a question about photosynthesis, then pair it with a passage about plant reproduction. Both discuss plants, both use botanical vocabulary, but one doesn’t answer the other. This forces models to do deep semantic matching rather than keyword spotting. They also include questions where the passage contains partial information but not enough to answer completely.

The benchmark tests three transformer architectures (BERT, RoBERTa, ELECTRA) in two settings: zero-shot (pretrained only) and fine-tuned on NCTB-QA. Performance is measured by exact match, F1 score, and BERTScore (semantic similarity between predicted and gold answers).

Textbook Corpus (50 books)
         |
         v
   [Passage Extraction]
         |
         +---> Answerable pairs (57%)
         |         |
         |         v
         |    Question + Answer + Supporting passage
         |
         +---> Unanswerable pairs (43%)
                   |
                   v
              Question + Distractor passage
                   |
                   v
            (topically related but lacks answer)
         
         All pairs
         |
         v
    [Train/Dev/Test Split]
         |
         v
    Benchmark: BERT/RoBERTa/ELECTRA
         |
         v
    Metrics: EM / F1 / BERTScore

Think of this like training a medical student with a trick exam. Most questions have answers in the textbook chapter (answerable), but some questions reference a different chapter entirely (unanswerable). The catch: the wrong chapter discusses related anatomy, uses similar terminology, and looks superficially relevant. A lazy student doing keyword matching will confidently write nonsense. A careful student learns to recognize “this passage doesn’t actually contain what I need” and writes “insufficient information.” The dataset is the exam; fine-tuning is the study process; the model’s ability to abstain is passing the trick questions.

Key Concepts

• Unanswerable Questions: Not trick questions or riddles — these are perfectly valid questions whose answers simply don’t appear in the provided context passage. Example: “What causes diabetes?” paired with a passage about heart disease symptoms. The question is clear, but the passage is the wrong source. Models must learn to detect this mismatch rather than extract plausible-sounding fragments. This differs from “hard” answerable questions where the answer exists but requires complex reasoning.

• Adversarial Distractors: Passages deliberately chosen to mislead keyword-matching approaches. If the question asks about “mitochondria function,” the distractor passage discusses “cell membrane structure” — same domain (cell biology), overlapping vocabulary (organelles, cellular processes), but fundamentally different content. This design punishes shallow pattern matching and rewards semantic understanding. Without adversarial design, models could cheat by checking if any sentence contains question keywords.

• Domain-Specific Fine-Tuning: Pretrained models know general language patterns but lack specialized knowledge. A model pretrained on web text doesn’t know Bangla educational terminology or textbook discourse structure. Fine-tuning on NCTB-QA teaches both the domain vocabulary (scientific terms, historical names) and the task structure (when to abstain). The 313% F1 improvement shows pretrained knowledge alone is insufficient — you need in-domain adaptation for low-resource languages where pretraining data is scarce.

Framework Shift

Before (existing Bangla QA):        After (NCTB-QA):

Question --> [Model] --> Answer     Question + Passage
                                           |
Always produces output                    v
(even if nonsense)                   [Model Decision]
                                      /            \
                                     /              \
                              Answerable?      Unanswerable?
                                   |                 |
                                   v                 v
                               Extract           Return NULL
                               Answer            (abstain)
                                   
Assumption: answer exists       Assumption: answer may not exist

From “always answer” to “answer when possible, abstain when not,” the core shift is teaching models that silence is sometimes the right response.

Expert Assessment

Problem choice: Real gap. Low-resource NLP consistently lags English by 3-5 years, and unanswerable question handling is a known weakness even in high-resource settings. The educational domain choice is smart — textbooks provide clean, structured text and natural question-answer pairs. This isn’t manufactured; it’s addressing a practical need for Bangla educational technology.

Method maturity: Straightforward dataset construction, no novel algorithms. The adversarial distractor design is borrowed from SQuAD 2.0. The contribution is execution (building the resource) rather than methodological innovation. That’s fine — low-resource languages need datasets more than they need new architectures. However, the paper doesn’t explore automated methods for generating unanswerable pairs at scale, relying heavily on manual curation.

Experimental integrity: Baselines are fair (standard pretrained models), but the comparison is limited. No cross-lingual transfer experiments (e.g., does English SQuAD 2.0 fine-tuning help?). No analysis of what makes questions hard — which distractor types fool models most? The 313% improvement sounds dramatic but starts from a very low baseline (0.15 F1), suggesting the pretrained models are nearly useless without fine-tuning. More error analysis would strengthen claims.

Writing quality: The dataset description is clear, but the paper lacks depth in analysis. The “Results and Discussion” section mostly reports numbers without explaining why certain models perform better or what linguistic phenomena cause failures. A case study of model errors (false positives vs false negatives on unanswerable questions) would be valuable. The related work section adequately covers Bangla NLP but misses recent work on abstention mechanisms in QA.

Verdict: weak accept — Solid resource contribution for an underserved language, but limited analytical insight beyond “fine-tuning helps.”

Takeaways

If you’re building QA systems for any low-resource language, steal the adversarial distractor design: don’t just add random unanswerable questions, pair them with topically related passages that share vocabulary with the question. This creates a much harder benchmark than random negatives.

The 43/57 unanswerable/answerable split is worth noting — not 50/50, but enough negative examples to force models to learn abstention without overwhelming the dataset. If you’re creating similar datasets, this ratio seems like a reasonable starting point.

The dramatic improvement from fine-tuning (0.15 → 0.62 F1) suggests that for low-resource languages, pretrained models are nearly useless out-of-the-box for specialized tasks. Don’t assume transfer learning will save you — budget for domain-specific fine-tuning data.