About ReClor

Logical reasoning is an important ability to examine, analyze, and critically evaluate arguments as they occur in ordinary language as the definition from LSAC. ReClor is a dataset extracted from logical reasoning questions of standardized graduate admission examinations. Empirical results show that the state-of-the-art models struggle on ReClor with poor performance indicating more research is needed to essentially enhance the logical reasoning ability of current models. We hope this dataset could help push Machine Reading Comprehension (MRC) towards more complicated reasoning.

Example

Context:
In jurisdictions where use of headlights is optional when visibility is good, drivers who use headlights at all times are less likely to be involved in a collision than are drivers who use headlights only when visibility is poor. Yet Highway Safety Department records show that making use of headlights mandatory at all times does nothing to reduce the overall number of collisions.
Question: Which one of the following, if true, most helps to resolve the apparent discrepancy in the information above?
Options:
A. In jurisdictions where use of headlights is optional when visibility is good, one driver in four uses headlights for daytime driving in good weather.
B. Only very careful drivers use headlights when their use is not legally required.
C. The jurisdictions where use of headlights is mandatory at all times are those where daytime visibility is frequently poor.
D. A law making use of headlights mandatory at all times is not especially difficult to enforce.
Answer: B

Download

If you agree to the following items, please download the dataset from this download link use this Download Link, and unzip it use the following password:
for_non-commercial_research_purpose_only

Use Items

1. ReClor dataset is available for non-commercial research purpose use only.
2. All passages are obtained from websites/books which are not the property of National University of Singapore. We are not responsible for the content nor the meaning of these passages.
3. We shuffle the order of answer options and randomly delete one of the wrong options for each dataset point to comply with fair use of law.
4. You agree not to reproduce, duplicate, copy, sell, trade, resell or exploit for any commercial purpose, any portion of the contexts and any portion of derived data.
5. We reserve the right to terminate your access to the ReClor dataset at any time.

Code

This GitHub repository serves as an example to show how to load the dataset and generate predicted results of testing set that can submitted to the leaderboard.

Leaderboard

Please submit the predicted results of testing set of your model to this EvalAI leaderboard for evaluation.

Citation

If the paper inspires you, please cite us:

@inproceedings{yu2020reclor,
        author = {Yu, Weihao and Jiang, Zihang and Dong, Yanfei and Feng, Jiashi},
        title = {ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning},
        booktitle = {International Conference on Learning Representations (ICLR)},
        month = {April},
        year = {2020}
    }

Authors

Weihao Yu*

Zihang Jiang*

Yanfei Dong

Jiashi Feng

* equal contribution

Contact

If you have any question, please email us.