Rafael Glater

# Learn how to use trec_eval to evaluate your information retrieval system

The trec_eval is a tool used to evaluate rankings, either documents or any other information that is sorted by relevance. The evaluation is based on two files: The first, known as "qrels" (query relevances) lists the relevance judgements for each query. The second contains the rankings of documents returned by your RI system. The evaluation is based on two files: The first, known as "qrels" (query relevances) lists the relevance judgements for each query. The second contains the rankings of documents returned by your RI system. More details about these files will be described below.

Click here to visit the official website and download the lastest version link, which is currently in version 9.0. Once downloaded, extract files to a folder and type "make" in command line prompt to compile trec_eval sourcecode. After that, the trec_eval executable will be ready to use.

## How to use

The command to run trec_eval has the following format:

$./trec_eval [-q] [-m measure] qrel_file results_file trec_eval: is the name of the executable program. -q: In addition to summary evaluation, give evaluation for each query/topic qrel_file: path of the file with the list of relevant documents for each query -m: shows only a specific measure ("-m all_trec" shows all measures, "-m official" is the default parameter which shows only main measures) results_file: path of the file with the list of documents retrieved by your application Note: trec_eval provides sample qrels and results files, which are respectively "test/qrels.test" and "test/results.test". You can test trec_eval command with them. ## Viewing a specific measure If you want to see only some specific measures, just use the -m parameter followed by measure name. Use period (".") to pass parameters to the measure and separate multiple parameters with a comma (","). The -m parameter should be passed once for each measure. For example, to show only the measures MAP and Precision at 5 and at 10 use the following command: $ .\trec_eval -m map -m P.5,10 qrels_file results_file

Details about all available measures are given below.

## The qrels file

This file contains a list of documents considered relevants for each query. This relevance judgement is made by human beings who manually select documents that should be retrieved when a particular query is executed. This file can be considered as the "correct answer" and the documents retrieved by your IR system should approximate the maximum to it. It has the following format:

query-id 0 document-id relevance

The field query-id is a alphanumeric sequence to identify the query, document-id is a alphanumeric sequence to identify the judged document and the field relevance is a number to indicate the relevance degree between the document and query (0 for non relevant and 1 for relevant). The second field "0" is not currently used, just put it in the file. The fields can be separated by a blank space or tabulation.

## The results file

The results file contains a ranking of documents for each query automaticaly generated by your application. This is the file that will evaluated by trec_eval based in the "correct answer" provided by the first file. This file has the following format:

query-id Q0 document-id rank score STANDARD

The field query-id is a alphanumeric sequence to identify the query. The second field, with "Q0" value, is currently ignored by trec_eval, just put it in the file. The field document-id is a alphanumeric sequence to identify the retrieved document. The field rank is an integer value which represents the document position in the ranking, but this field is also ignored by trec_eval. The field score can be an integer or float value to indicate the similarity degree between document and query, so the most relevants docs will have higher scores. The last field, with "STANDARD" value, is used only to identify this run (this name is also showed in the output), you can use any alphanumeric sequence.

## Measures of trec_eval

Running trec_eval with default options the following informations are outputted:

 runid Name of the run (is the name given on the last field of the results file) num_q Total number of evaluated queries num_ret Total number of retrieved documents num_rel Total number of relevant documents (according to the qrels file) num_rel_ret Total number of relevant documents retrieved (in the results file) map Mean average precision (map) gm_map Average precision. Geometric mean Rprec Precision of the first R documents, where R are the number os relevants bpref Binary preference recip_rank Reciprical Rank iprec_at_recall_0.00 Interpolated Recall - Precision Averages at 0.00 recall iprec_at_recall_0.10 Interpolated Recall - Precision Averages at 0.10 recall iprec_at_recall_0.20 Interpolated Recall - Precision Averages at 0.20 recall iprec_at_recall_0.30 Interpolated Recall - Precision Averages at 0.30 recall iprec_at_recall_0.40 Interpolated Recall - Precision Averages at 0.40 recall iprec_at_recall_0.50 Interpolated Recall - Precision Averages at 0.50 recall iprec_at_recall_0.60 Interpolated Recall - Precision Averages at 0.60 recall iprec_at_recall_0.70 Interpolated Recall - Precision Averages at 0.70 recall iprec_at_recall_0.80 Interpolated Recall - Precision Averages at 0.80 recall iprec_at_recall_0.90 Interpolated Recall - Precision Averages at 0.90 recall iprec_at_recall_1.00 Interpolated Recall - Precision Averages at 1.00 recall P_5 Precision of the 5 first documents P_10 Precision of the 10 first documents P_15 Precision of the 15 first documents P_20 Precision of the 20 first documents P_30 Precision of the 30 first documents P_100 Precision of the 100 first documents P_200 Precision of the 200 first documents P_500 Precision of the 500 first documents P_1000 Precision of the 1000 first documents