How Search Engines Work
BeginnerA search engine has three core systems: a crawler that continuously visits billions of web pages; an indexer that organises what it finds into a giant lookup table; and a ranking algorithm that, given a search query, retrieves and orders the most relevant results in milliseconds. Google's crawler visits billions of pages daily, its index stores hundreds of billions of documents, and its ranking system considers over 200 signals to return results within 200 milliseconds of your search.
Think of a search engine like a librarian who has read every book
Imagine a librarian who has personally read every book and magazine ever published (crawling), catalogued every sentence by topic (indexing), and memorised which books are cited most by other books (PageRank). When you ask a question, they instantly recall not just relevant books, but the most authoritative and fresh ones — ranked by a combination of relevance, trustworthiness, and how well they match the words you used. Google is this librarian, at internet scale.
Step by Step
Key Concepts
Web Crawler
An automated bot that systematically browses the web to discover and fetch pages. Follows hyperlinks from page to page. Googlebot is the most sophisticated crawler, handling JavaScript rendering, respecting crawl budgets and robots.txt, and prioritising fresh and high-quality content. Crawl frequency depends on page importance and how often content changes.
Inverted Index
The core data structure of a search engine — a mapping from words (or tokens) to the documents containing them. Allows retrieval of all pages containing a query word in O(1) time. Stores not just which pages contain a word, but where (title, body, anchor text), how often, and in what proximity to other query terms.
PageRank
Google's foundational link analysis algorithm, treating hyperlinks as votes of confidence. Pages linked by many authoritative pages rank higher. The score distributes across the web like a random walk: imagining a user clicking random links, PageRank is the probability of landing on each page. Still relevant but now one of 200+ ranking factors, with AI-based signals increasingly dominant.
E-E-A-T
Experience, Expertise, Authoritativeness, Trustworthiness — Google's quality framework for evaluating content. Used by human "search quality raters" to evaluate ranking quality, which in turn trains ranking algorithms. Particularly important for YMYL (Your Money Your Life) queries — health, finance, legal topics where bad information causes real harm.
Core Web Vitals
Google's page experience metrics used as ranking signals since 2021. LCP (Largest Contentful Paint): how quickly main content loads (target <2.5s). FID (First Input Delay) / INP (Interaction to Next Paint): responsiveness to clicks (target <200ms). CLS (Cumulative Layout Shift): visual stability, no content jumping around (target <0.1). Poor scores negatively impact rankings.
BERT / Transformer Models
Since 2019, Google uses BERT (Bidirectional Encoder Representations from Transformers) to understand query and document meaning beyond keyword matching. BERT captures context (the word "bank" means different things in "river bank" and "bank account") and long-range dependencies. MUM (2021) is even more powerful, understanding text across 75 languages and multimedia content.
robots.txt
A file at the root of a website (example.com/robots.txt) that instructs search engine crawlers which pages they may and may not crawl. Used to exclude duplicate content, private pages, and admin areas from the index. Googlebot respects robots.txt instructions. "Disallow: /" would block all crawling; "Allow: /" permits full crawling.
Key Facts
- Google processes approximately 8.5 billion searches per day (2024) — roughly 99,000 searches per second. Google has over 90% of the global search market.
- Google's index contains hundreds of billions of web pages. The entire indexed web is estimated at 4-5 billion pages — a tiny fraction of the ~1 trillion URLs Google has discovered.
- The original PageRank paper ("The Anatomy of a Large-Scale Hypertextual Web Search Engine") by Larry Page and Sergey Brin, published in 1998, is the most cited computer science paper of all time.
- Google's first data centre (1998) was built from cheap commodity hardware in Larry Page's dorm room — 10 4GB hard drives in a custom rack built from LEGO. It could handle ~10,000 queries/day.
- AI Overviews (Google's generative AI summaries in search results, launched 2024) are trained on search index data and use large language models to synthesise answers, changing the fundamental "ten blue links" paradigm of search.
- India is Google Search's second-largest market by query volume after the US. Hindi, Tamil, Telugu, and other Indian languages are increasingly important, driving Google's investment in multilingual AI models.
Real-World Applications
Information Access
Search engines are the primary way humanity accesses information — from medical symptoms to historical facts to technical documentation. Google has indexed knowledge that would take a human millions of lifetimes to read, and makes it accessible in milliseconds to anyone with internet access.
Search Engine Optimisation (SEO)
Businesses invest heavily in SEO — optimising content, technical performance, and link profiles to rank higher. SEO is a ~$80 billion industry globally. Higher rankings directly translate to traffic, leads, and revenue. Understanding how search engines work is fundamental to digital marketing.
Local Search
Google Maps and local search ("restaurants near me") combines traditional web indexing with business data, reviews, location signals, and real-time information. Local search drives foot traffic for millions of small businesses. "Near me" searches have grown 150%+ in recent years as mobile usage dominates.
Vertical Search Engines
Specialised search engines index specific content types: YouTube (video), Google Scholar (academic papers), LinkedIn (professional profiles), Booking.com (hotels), Amazon (products). These "vertical" search engines often out-rank Google for queries within their domain because of superior depth and structured data.
AI-Powered Search
Microsoft Bing's ChatGPT integration (2023) and Google's AI Overviews (2024) are transforming search from a list of links to a direct answer engine. Perplexity AI offers citation-based AI search. This shift is the most significant change to search since PageRank — potentially reducing clicks to websites and concentrating answer-giving in a single AI response.
Frequently Asked Questions
How does Google know when a new page is published?
Several mechanisms: (1) Sitemaps — website owners submit XML sitemaps listing all URLs for Googlebot to crawl; (2) Following links — when a new page is linked from an already-indexed page, Googlebot discovers it while crawling; (3) URL submission — Google Search Console allows direct URL submission; (4) Crawl patterns — frequently-updated sites are crawled more often. New pages can appear in Google's index within hours if they're on authoritative domains, or weeks for new sites.
Can you pay Google to rank higher in search results?
For organic (non-ad) results: no. Google explicitly separates paid and organic results and protects the integrity of organic rankings. You can pay for Google Ads to appear above organic results (labelled "Sponsored"), but these are clearly identified and don't affect organic ranking. This separation is fundamental to Google's business model — users trust the organic results because they know they can't be bought.
Why does the same search return different results on different devices?
Google personalises results based on: search history and Google account data, geographic location (IP and GPS), device type (desktop vs mobile — different results for "near me"), language settings, and time of day. Two people in different cities searching the same thing will get significantly different results. You can see personalisation-free results using incognito mode and a VPN, or appending "&pws=0" to the Google search URL.
What is the "dark web" and why isn't it indexed?
The dark web is content on networks (primarily Tor) that require special software to access, using .onion addresses not accessible via regular browsers. Search engines can't index it because: (1) crawlers can't access Tor network addresses, (2) most dark web sites don't want to be indexed, (3) content changes rapidly. The "deep web" is different — it's simply unindexed content like your email inbox, private databases, and login-protected pages. The deep web is vast; the dark web is much smaller.