Average Precision

TL;DR

• Metric for ranking performance that averages precision values at positions where relevant items appear.
• Reflects both relevance and rank position of relevant items.
• Especially useful when the number of relevant items is small.

Definition

Average precision is a metric used in information retrieval and data mining to evaluate the performance of a model in ranking items. It measures the average of the precision values at each position in the ranking, taking into account only the items that are relevant to the query.

Explanation

Average precision computes precision at each position in a ranked list and then averages those precision values only over the positions corresponding to relevant items. Precision at a position is the number of relevant items up to that position divided by the total number of items up to that position. The resulting average summarizes how well the ranking places relevant items toward the top.

Examples

Example 1

A search engine returns the following ranked list for the query “jaguar car”:

• Jaguar XE sedan
• Jaguar XF sedan
• Jaguar XJ sedan
• Jaguar F-Type sports car
• Jaguar XK sports car
• Ford Mustang sports car
• Toyota Camry sedan
• Honda Civic sedan

The relevant items are the first four items in the list. The precision at each position is:

• precision at position 1: 1/1=1.0
• precision at position 2: 2/2=1.0
• precision at position 3: 3/3=1.0
• precision at position 4: 4/4=1.0

The average precision is the average of these precision values:

$$(1.0+1.0+1.0+1.0)/4=1.0$$

Example 2

A different search engine returns this ranked list for the same query “jaguar car”:

• Ford Mustang sports car
• Toyota Camry sedan
• Honda Civic sedan
• Jaguar XE sedan
• Jaguar XF sedan
• Jaguar XJ sedan
• Jaguar F-Type sports car
• Jaguar XK sports car

The relevant items are still the first four items in the list. The precision at each position is:

• precision at position 1: 0/1=0.0
• precision at position 2: 0/2=0.0
• precision at position 3: 0/3=0.0
• precision at position 4: 1/4=0.25

The average precision is the average of the precision values at the relevant-item positions:

$$(0.25+0.25+0.25+0.25)/4=0.25$$

Use cases

• Useful for evaluating ranking models, particularly when the number of relevant items is small, because it evaluates the precision of those relevant items rather than overall recall.

Notes or pitfalls

• Average precision considers only precision values at positions of relevant items, so it emphasizes both relevance and rank position.