How Search Engines Work · Crawling, Indexing & Ranking Explained

A search engine is a tool that helps people find information on the internet. You type in a question or some words, and it shows you a list of web pages it thinks are most relevant to what you are looking for.

That sounds simple. But behind the scenes, to be able to respond to billions of searches every day in a fraction of a second, a search engine like Google must first read and store a copy of a huge portion of the entire internet — and then have a system for deciding which pages best answer each possible query.

Google is by far the dominant search engine in the UK and most of the world — accounting for approximately 90% of search queries. Bing (Microsoft), Yahoo, and DuckDuckGo make up most of the remainder. When marketers talk about "SEO," they almost always mean Google SEO specifically, though the principles apply broadly.

Google uses automated programmes called crawlers (also called spiders or bots) to systematically browse the internet. The most famous is called Googlebot. Crawlers follow links — they find a page, read it, follow the links on that page to other pages, read those, follow their links, and so on continuously.

This is why links matter so much in SEO: if no other page links to yours, Googlebot may never find it. A new website with no links pointing to it is like a house with no road leading to it — it exists, but the postman can never find it to deliver the mail.

Google's crawlers visit pages regularly — how often depends on how frequently your content changes and how important Google thinks your site is. A major news site might be crawled every few minutes. A small business website might be crawled every few weeks.

Website owners can influence crawling through a file called robots.txt — a simple text file that tells crawlers which pages they are and are not allowed to visit. You can also use a sitemap — a list of all the pages on your site — to help Google find all your content even if it is not well-linked internally.

When Googlebot crawls a page, it processes the content and stores a version of it in Google's index — a massive database of every page Google knows about. Think of the index as Google's filing system: it stores what each page is about so it can retrieve it quickly when someone searches for a related term.

During indexing, Google analyses:

• The text on the page — What words are used? What is the page about? Is the content clear and well-organised?
• The HTML structure — Is there a clear heading (H1)? Are subheadings used logically? Are images described with alt text?
• The links — What pages does this page link to? What pages link to this one?
• Page speed and technical quality — How fast does the page load? Does it work on mobile?
• The freshness — When was the content last updated? Is it current?

Not every crawled page gets indexed. Google may choose not to index a page if it thinks the content is too thin (not enough useful information), duplicate (the same content exists elsewhere), or low quality. You can check whether your pages are indexed by searching site:yourwebsite.com in Google.

Ranking is the most complex part of how search engines work — and the part that SEO is designed to influence. When someone types a query, Google retrieves all the pages in its index that are relevant to that query, then ranks them in order of how useful and trustworthy it thinks each one is for that particular search.

Google uses hundreds of signals to rank pages — it does not publicly share the complete list, but it has documented the most important ones. The ranking algorithm has one goal: show the user the most helpful, reliable answer to their question as quickly as possible.

Google has documented several categories of signals that influence rankings. Here are the most important ones a beginner should know:

Relevance — Does the page actually answer the query? Google is very good at understanding language, so it is not just about keyword matching. A page about "running trainers" might rank for "best shoes for jogging" even if it does not use those exact words. The page must be clearly and genuinely about the topic.

Quality and helpfulness — Google has explicitly stated it wants to rank pages that are helpful to real people, not pages written primarily to rank in search engines. The E-E-A-T framework (Experience, Expertise, Authoritativeness, Trustworthiness) describes what Google looks for in high-quality content.

Backlinks (links from other sites) — When other websites link to your page, Google treats it as a vote of confidence. A link from the BBC or a respected industry publication carries much more weight than a link from a random blog. This is why "link building" — earning links from other websites — is a major part of SEO.

Page experience — Google uses a set of metrics called Core Web Vitals to measure how good the experience of loading and using a page is. Pages that load fast, do not shift around as they load, and respond quickly to user interaction are rewarded.

Mobile-friendliness — Google primarily uses the mobile version of your website to rank your pages (this is called "mobile-first indexing"). If your site does not work well on a phone, it will rank poorly — even for people searching on a desktop.

A Google Search Engine Results Page (SERP) is not just a list of blue links anymore. There are several different types of results that appear, and understanding them helps you understand what SEO can and cannot achieve:

This matters because it changes what "ranking number one" means. For some queries, the organic result in position one is actually the fifth thing a user sees on the page — below paid ads, a featured snippet, and a local pack. Understanding the full SERP for your target queries helps you set realistic SEO goals.

Search intent is the reason behind a search query — what the person actually wants to find or do. It is the single most important concept in SEO because Google's entire mission is to match results to intent. If your page does not match the intent of the query, it will not rank — even if it is beautifully written and technically perfect.

There are four main types of search intent:

Informational — The person wants to learn something. "How do I tie a bowline knot?" "What is inflation?" "Why is the sky blue?" These queries are best answered by guides, articles, and explainers. The person is not trying to buy anything.

Navigational — The person wants to get to a specific website. "Facebook login," "HMRC self-assessment," "Amazon UK." They already know where they want to go and are using Google as a shortcut. There is nothing to optimise here — they want the specific brand, not alternatives.

Commercial — The person is researching before buying. "Best running shoes 2026," "iPhone 16 vs Samsung S25," "Cheapest broadband deals." They are gathering information to make a purchase decision. These are valuable queries for businesses — the person is on their way to buying.

Transactional — The person is ready to buy or take action. "Buy Nike Pegasus 40 UK8," "Book dentist appointment Manchester," "Download Spotify." They want to complete a specific action now.

Google makes thousands of changes to its ranking algorithm every year — most of them small, some of them major. The major updates — called core updates — can significantly shift which pages rank for important queries. This is why SEO rankings are never permanent: what works today may be less effective after the next update.

Google's publicly documented major updates include: Panda (targeting thin, low-quality content), Penguin (targeting manipulative link building), Hummingbird (improving understanding of natural language queries), RankBrain (introducing machine learning to ranking), BERT (better understanding of query context), and Helpful Content (targeting content written primarily for search engines rather than for people).

The through-line in all these updates: Google is progressively getting better at identifying what is genuinely useful to searchers and rewarding it — and identifying what is manipulative or low-quality and demoting it. Building pages that genuinely help people is more durable than trying to game the algorithm.

Now that you understand how search engines work, here are the practical implications:

The next guide in this track — SEO for Beginners — takes everything you have just learned and turns it into practical steps for improving your own website's search visibility.

What to Read Next

You have finished this guide. Here are the best places to go deeper — all written for beginners, all from the Digital Codex reference library.

Sources & Further Reading