Summary

12,744

Stories

119,633

Question-Answer Pairs

616

Average Words per Article

4.13

Average Words per Answer

74.9%

Human Performance (F1)

Reasoning Statistics

Reasoning mechanisms needed to answer questions in NewsQA based on 500 examples. For each type, we show an example question with the text snippet that contains the answer span, with words relevant to the reasoning type in bold.

Reasoning Proportion Example
Word Matching 31.6%

Q: When were the findings published?

T: Both sets of research findings were published Thursday...

Paraphrasing 26.8%

Q: Who is the struggle between in Rwanda?

T: The struggle pits ethnic Tutsis, supported by Rwanda, against ethnic Hut

Synthesis 17.8%

Q: Where is Brittanee Drexel from?

T: The mother of a 17-year-old Rochester, New York high school student ... says she did not give her daughter permission to go on the trip. Brittanee Marie Drexel's mom says...

Inference 14.0%

Q: Who drew inspiration from presidents?

T: Rudy Ruiz says the lives of US presidents can make them positive role models for students.

Ambiguous/ Insufficient 9.8%

Q: Whose mother is moving to the White House?

T: ... Barack Obama's mother-in-law, Marian Robinson will join the Obamas at the family's private quarters at 1600 Pennsylvania Avenue. [Michelle is never mentioned]

Consensus Statistics

102,841

With Consensus (Including Validated)

(85.96%)

51,630

Validated Answers

Only 51,630 needed to be validated because there was agreement during second step of the collection.

45,381

Validated Answers with Consensus

(87.90%)

Story Length Distribution

Question Length Distribution

Question Type Distribution

Overall Answer Length Distribution

Average Answer Length Distribution per Question