Evaluation

If you would like to evaluate your results on the FigureQA test sets, please compile your predicted answers in the format documented below and email them to the authors.

Please send your results to Samira and Adam A. at figureqa@microsoft.com.

Predictions Format

Results must be provided in a CSV format, having the first line as a header and each row like so:

 question_index,image_index,question_id,question_string,answer 

Here  question_index  refers to the sample's index in the  qa_pairs  array within qa_pairs.json. All other fields are the same as in qa_pairs.json, documented here.

Here are example results files for test1 and test2. Simply replace the final column field  <0/1>  with a  0  or  1  in each row.

To evaluate your results faster, please ensure that your results file is UTF-8 encoded and can be read using the read_csv function from the Python Pandas library.