Information Retrieval

Intermediate

Information retrieval (IR) is the science and practice of searching for and obtaining relevant information from large collections of data, including text documents, multimedia, and structured databases. The field addresses the fundamental challenge of connecting users with the information they need, encompassing the theories, algorithms, and systems that power modern search engines, digital libraries, and recommendation systems. At its core, IR deals with the representation, storage, organization, and access of information items, drawing on principles from computer science, linguistics, cognitive science, and library science.

The theoretical foundations of information retrieval were established in the mid-20th century, with seminal contributions from researchers such as Gerard Salton, who developed the vector space model and the SMART Information Retrieval System, and Stephen Robertson, who advanced probabilistic retrieval models. The field introduced key evaluation metrics like precision and recall, and formalized the concept of relevance as a measurable quantity. The development of the inverted index as a core data structure enabled efficient full-text search over massive document collections, paving the way for the web search revolution of the late 1990s and early 2000s.

Today, information retrieval encompasses a broad range of topics including web search, question answering, text classification, clustering, filtering, and recommendation. Modern IR systems leverage machine learning, natural language processing, and deep learning techniques such as transformer-based neural ranking models to improve search quality. Evaluation campaigns like TREC (Text REtrieval Conference) continue to drive innovation. The field is more relevant than ever as the volume of digital information grows exponentially, making effective retrieval a critical capability for individuals, businesses, and society at large.

Practice a little. See where you stand.

Ready to practice?5 minutes. No pressure.

Take the Quiz Learn First

Quiz

Reveal what you know — and what needs work

Adaptive Learn

Responds to how you reason, with real-time hints

Start here

Flashcards

Build recall through spaced, active review

Cheat Sheet

The essentials at a glance — exam-ready

Glossary

Master the vocabulary that unlocks understanding

Learning Roadmap

A structured path from foundations to mastery

Book

Deep-dive guide with worked examples

Key Concepts

A data structure that maps each term in a vocabulary to a list of documents (or positions within documents) where that term appears, enabling fast full-text search. It is the fundamental building block of most modern search engines.

Example

When Google indexes billions of web pages, it builds an inverted index so that searching for 'climate change' instantly retrieves all pages containing those words, rather than scanning every page sequentially.

A numerical statistic that reflects the importance of a term in a document relative to a collection. Term frequency measures how often a term appears in a document, while inverse document frequency reduces the weight of terms that appear in many documents.

Example

In a collection of news articles, the word 'the' has a high term frequency but very low IDF (it appears everywhere), so its TF-IDF score is low. The word 'cryptocurrency' may appear less often but has a higher TF-IDF score in articles where it is discussed, making it more useful for identifying relevant documents.

Two fundamental evaluation metrics in IR. Precision is the fraction of retrieved documents that are relevant, while recall is the fraction of all relevant documents that are retrieved. Together they capture the trade-off between returning only relevant results and returning all relevant results.

Example

If a search engine returns 10 results for a query and 7 are relevant (precision = 70%), but there are 20 relevant documents in the collection total, then recall is 7/20 = 35%. Improving one metric often comes at the expense of the other.

A mathematical model for representing documents and queries as vectors in a high-dimensional space, where each dimension corresponds to a term. Relevance is computed as the similarity (often cosine similarity) between the query vector and document vectors.

Example

A query about 'machine learning algorithms' is represented as a vector, and documents whose vectors point in a similar direction in term-space are ranked as more relevant, even if they do not contain the exact query words.

A probabilistic ranking function used to estimate the relevance of documents to a given query. It extends TF-IDF by incorporating document length normalization and term saturation, and is widely used as a strong baseline in modern search systems.

Example

Elasticsearch and Apache Solr use BM25 as their default ranking algorithm. When you search a product catalog, BM25 scores each product description against your query, ranking shorter and more focused descriptions appropriately against longer ones.

A technique where the system uses user judgments on initially retrieved documents to refine the query and improve subsequent retrieval results. It can be explicit (user marks relevant documents) or implicit (inferred from click behavior).

Example

After a user searches for 'java' and clicks only on programming-related results (ignoring results about coffee or the island), the system infers that the user wants programming content and adjusts the results accordingly.

The process of automatically adding additional terms to a user's original query to improve retrieval effectiveness. Terms can be drawn from thesauri, user feedback, or co-occurrence statistics in the document collection.

Example

A user searches for 'heart attack' and the system automatically expands the query to include 'myocardial infarction,' 'cardiac arrest,' and 'coronary event,' retrieving medical documents that use clinical terminology.

An algorithm developed by Larry Page and Sergey Brin that ranks web pages based on the structure of hyperlinks. A page receives a higher score if it is linked to by many pages, especially by pages that themselves have high PageRank scores.

Example

Wikipedia pages tend to have high PageRank because thousands of other websites link to them, signaling to the algorithm that these pages are authoritative and trustworthy sources of information.

The application of deep learning and neural network models to information retrieval tasks, including learned dense representations (embeddings), cross-encoders for re-ranking, and end-to-end retrieval models that move beyond traditional term-matching approaches.

Example

A BERT-based re-ranker can understand that a query about 'how to fix a leaking faucet' is semantically related to a document titled 'plumbing repair guide for dripping taps,' even though few query terms appear verbatim in the document.

One concept at a time.

Explore your way

Choose a different way to engage with this topic — no grading, just richer thinking.

Explore your way — choose one:

Explore with AI →

Curriculum alignment— Standards-aligned

Grade level

College+

Learning objectives

•Analyze ranking algorithms including TF-IDF, BM25, and learning-to-rank models for relevance optimization in search systems
•Evaluate precision, recall, F-measure, and normalized discounted cumulative gain as metrics for retrieval system effectiveness
•Apply indexing techniques including inverted indexes, query expansion, and relevance feedback to improve search performance
•Design evaluation experiments using test collections, pooling methods, and statistical significance testing for retrieval benchmarking

Recommended Resources

This page contains affiliate links. We may earn a commission at no extra cost to you.

Books

Introduction to Information Retrieval

by Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schutze

Search Engines: Information Retrieval in Practice

by W. Bruce Croft, Donald Metzler, and Trevor Strohman

Modern Information Retrieval: The Concepts and Technology behind Search

by Ricardo Baeza-Yates and Berthier Ribeiro-Neto