Web
SEO for people who would rather be building
Everything I know about SEO
7 min readVedas Dixit
I’ve been building websites for a while now, and somewhere along the way I started noticing the same things every site needs to get discovered. So I made an Agent Checklist for the websites I build.
2 years of me building websites alongside Claude | other agents unknowingly made this Mental Model. I expected some dark art & it turned out to be a short list, and most of it is just being a good host :) tell people where the door is, let them in, and don't leave them waiting... that's it!
I also didn't want to write another list of tips. Google's own docs are right there, and they're good. So I made that video above about a new haunted house in town & tried my best to lay that mental model down. Below is the slow version.
Search is like a pipeline
This is one way to put it, but it's more like a series of operations optionally done on a website. A page goes through six stages before anyone sees it in a search result, and every SEO technique you've ever heard of is a fix for one of them.
01 · Discover
Google first has to find out that your URL exists. A sitemap can tell it, or another page can link to it.
Breaks onIf nothing points to the page, Google might never find it.
02 · Crawl
Once Google knows the URL exists, Googlebot actually makes a request to your server and asks for the page.
Breaks onrobots.txt can block the crawl & so can things like server errors or a messy chain of redirects.
03 · Render
Getting the HTML isn't always the end of it. Google can also run JavaScript and build the page to see what actually appears on it.
Breaks onIf important content only appears after some client-side request or user interaction, Google might not see it the way you expect.
04 · Index
Now Google has to decide whether this page is worth keeping in its index, and which URL it should treat as the main version.
Breaks onA noindex tag, duplicate pages, or a page that says 'not found' while returning a 200 status can all cause problems here.
05 · Rank
If the page makes it into the index, Google can consider it when someone searches. This is where it figures out whether your page is actually useful for that particular search.
Breaks onHaving a page indexed doesn't mean it'll rank well. The content still needs to be useful and relevant, and things like performance and links can matter too.
06 · Display
Finally, Google has to turn the page into a search result that someone can actually see and decide to click.
Breaks onA bad title, an unhelpful description, or broken preview information can make a perfectly good page look pretty bad in the results.
Did you make a note of the order above? When something is wrong, find the earliest stage that's broken and fix that first. Doing stuff like polishing a title is pointless on a page that was never indexed :/
If you write the code, you pretty much own almost all of stages one to four, and six! Stage five is mostly about whether the page is actually worth reading, which is a different kind of problem...
There are two versions of every page
The first is the response HTML: the exact string your server sends back.
The second is the rendered DOM | Document Object Model: what the page turns into after JavaScript has run.
You can see the first with View Source or curl. You can see the second in the Elements panel. And they can look surprisingly different.
Anything that's already in the response HTML is there for a crawler to read. Anything that's only added after JavaScript runs depends on that crawler actually running JavaScript and waiting for it to finish.
Google does run JavaScript, but not immediately. It reads the HTML first, extracts things like links, and then renders the page later using a headless version of Chrome. That renderer isn't a person browsing your site: it doesn't click around or scroll through the page, and it doesn't keep your cookies around between visits.
And Google isn't the only crawler you have to think about. Link previews from places like WhatsApp, Slack and LinkedIn may not run your JavaScript at all. The same is true for many other crawlers, including a lot of AI-related ones.
Try this on something you've built. It takes a minute.
# What does the server really send?
curl -s https://blog.vedasdixit.com/seo | head -c 3000
# Status and headers only. Try a URL that shouldn't exist.
curl -sI https://blog.vedasdixit.com/nope
# The two files every crawler asks for first.
curl -s https://blog.vedasdixit.com/robots.txt
curl -s https://blog.vedasdixit.com/sitemap.xmlThen open DevTools, run Disable JavaScript from the command menu, and reload. What's left on screen is roughly what those bots get. On a plain React app, that's an empty <div id="root">. Which is the whole argument for rendering on the server.
The setup
So the list is short, I promise :) I'll go through it in pipeline order. The code below is Next.js (the App Router), but the ideas are the same anywhere.
Links and a sitemap
One way Google finds pages is by following links from pages it already knows. Like the page vedasdixit.com has a link to blog.vedasdixit.com like this page, but navigation has to be real <a href> links. A div with a click handler that calls the router is invisible to a crawler.
A sitemap is the map of all the pages on your site. I generate it from the list of posts, so there is nothing to keep in sync by hand.
export default function sitemap(): MetadataRoute.Sitemap {
return [
{ url: absoluteUrl("/") },
{ url: absoluteUrl("/author") },
...getPosts().map((post) => ({
url: absoluteUrl(`/${post.slug}`),
lastModified: new Date(post.updated ?? post.date),
})),
];
}Two notes for my fellow Next.js developers. Google ignores `priority` and `changeFrequency`, so I left them out. And `lastModified` has to be true. Stamp today's date on everything and Google learns to ignore it.
A sitemap helps a page get found. A page you can only reach through the sitemap is an orphan, and it tends to perform like one ¯\_(ツ)_/¯
The sign in the yard | robots.txt
robots.txt tells crawlers what they're allowed to fetch. A good default says come in, and points at the map. Here's the example:
export default function robots(): MetadataRoute.Robots {
return {
rules: { userAgent: "*", allow: "/" },
sitemap: absoluteUrl("/sitemap.xml"),
};
}Also, the status code of robots.txt matters more than what's in it. A 404 is fine, Google just reads that as no rules. But a 500 (or a 429) makes Google stop crawling your whole site for 12 hours, and after that it falls back to an old copy for up to 30 days. So a robots.ts that throws can quietly stop all crawling :/
These two are easy to mix up because they sound like they're doing the same thing. They're not. One controls whether Google can crawl a page; the other tells Google not to put the page in its search index.
| robots.txt: Disallow | noindex | |
|---|---|---|
| What it means | Don't crawl this URL. | You can crawl it, but don't put it in the index. |
| What it controls | Crawling | Indexing |
| Can Google read the page? | No, if the URL is blocked. | Yes. |
| Will it disappear from search? | Not necessarily. Google can still know the URL exists from other pages and show it without crawling the page. | Yes, after Google crawls the page and sees the noindex. |
| The important catch | Because Google can't crawl the page, it can't see a noindex tag placed on that page. | The page must be crawlable for Google to see the noindex. |
So if you actually want a page out of search, let Google crawl it and use noindex. Disallow is different: you're telling crawlers not to fetch the page in the first place. And since robots.txt is public, it's not a way to hide anything sensitive.
So if you actually want a page out of search, let Google crawl it and use noindex. Disallow is different: you're telling crawlers not to fetch the page in the first place. And since robots.txt is public, it's not a way to hide anything sensitive.
And if you're thinking about AI bots: most AI companies have separate bots for search and for training. Like OpenAI has GPTBot for training and OAI-SearchBot for ChatGPT search. Anthropic has ClaudeBot and Claude-SearchBot. Blocking one doesn't block the other, so you can say no to training and still show up in their answers.
If your site is behind Cloudflare, curl the live robots.txt instead of trusting the one in your repo. Cloudflare can add its own AI bot rules to it.
Words in the HTML
The easiest way to make a page readable to crawlers is also the least exciting one: put the content in the HTML you send back. If I request this blog post, the response already contains the words you're reading right now.
You don't need to avoid JavaScript completely. JavaScript can add interactions, animations, menus, and all sorts of other things after the page loads. The important part is whether the content you're trying to get indexed exists before that happens.
For example, imagine a product page that sends back an empty <div> and then fetches the product name and description with JavaScript. A crawler that doesn't run JavaScript sees an empty box. A crawler that does run it might eventually see the product. Same URL, very different page depending on what the crawler can execute.
And there's a useful distinction here: JavaScript being involved doesn't automatically mean the content is missing from the HTML. A page can be rendered on the server, send that HTML to the browser, and then use JavaScript to make it interactive afterward.
That's what happens with this blog too. The posts are generated as HTML ahead of time, so the first response already contains the actual article. The JavaScript is mostly there to make the page nicer to use.
In React | Next.js, for example, even a component marked with 'use client' can still have its initial HTML generated on the server. The directive mainly changes where the component's JavaScript runs and what gets sent to the browser. It doesn't mean 'hide this from crawlers'.
The situations that cause trouble are more specific:
- Content that's fetched only after the page loads.
- Content that's only added when something happens in the browser.
- Components deliberately rendered only on the client.
- Content hidden behind an interaction that the crawler never performs.
- Content that isn't present until the user keeps scrolling.
One more thing about the HTML: Googlebot only reads roughly the first 2 MB of it. Anything after that just gets dropped. Most pages are nowhere near that, but a huge blob of inline JSON or a base64 image can push you over.
One address per page
The same page can end up being reachable at more than one URL: with and without www, with a trailing slash, or with something like ?utm_source= stuck on the end. You want Google to know which URL you consider the main one. That's what the canonical tag is for.
It's just one line in the page metadata, and every page can have its own.
export async function generateMetadata({ params }) {
const { slug } = await params;
const post = getPost(slug);
return {
title: post.title,
description: post.dek,
alternates: { canonical: `/${post.slug}` },
};
}One thing I had to be careful about: don't put a page-specific canonical in the root layout. Anything you put there gets inherited by the pages underneath it. You don't want every post telling Google that its canonical URL is the home page.
Also, the canonical only counts if it's inside the <head>. If something puts a div, img or iframe in the head above it, Google treats that as the end of the head and stops reading. Anything after it, including your canonical, gets missed. Tag managers & cookie banners are the usual suspects. And if a page ends up with two canonicals, Google may just ignore both.
The other half of this is returning the right status code. If someone visits a URL that doesn't exist, the server should actually return a 404. Don't render your normal 'not found' page while quietly returning a 200.
In Next.js, dynamicParams = false on the post route can handle this for a statically generated set of posts: a slug that isn't one of the real posts gets a 404 instead of pretending the page exists.
One catch if you use loading.tsx or <Suspense>: once the page starts streaming, the status is already 200 & Next.js can't change it anymore. So a notFound() that runs after that gives you a 200 with a noindex tag instead of a real 404. Google won't index it, but it's still not a real 404. Do the check before anything streams (or use dynamicParams like above), then confirm with curl -sI.
How the result looks
The title is probably the most obvious bit of the search result that I get to control. I keep mine unique, put the topic first, and the site name at the end.
The description is a little different. It doesn't directly affect ranking, but it can affect whether someone actually clicks the result. Google can also ignore it and write its own snippet, but I still write one for every page.
I also put a small block of JSON-LD on each post. It's basically some structured information for machines: this is a blog post, here's its headline, when it was published, who wrote it, and so on.
const jsonLd = {
"@context": "https://schema.org",
"@type": "BlogPosting",
headline: post.title,
description: post.dek,
datePublished: post.date,
dateModified: post.updated ?? post.date,
author: { "@type": "Person", name: "Vedas Dixit", url: "/author" },
};This isn't some magic ranking boost. It gives Google structured information about the page and can make it eligible for certain search features. The important part is that the data actually matches what's on the page.
The weird part
I originally set the share image once in the root layout. Then I gave individual posts their own openGraph metadata so I could set things like the post title and publish date.
That's where I ran into a slightly annoying Next.js detail.
Metadata doesn't merge openGraph field by field the way I initially expected. If a page defines its own openGraph, it can replace the layout's openGraph object instead of just adding the fields I changed.
So everything still builds, the page itself looks completely fine, and then you paste the link somewhere and realise the nice share image has disappeared.
I ended up keeping the fields that are shared by every page in one object and spreading those into any page that needs its own openGraph values.
export const OG_BASE = {
siteName: SITE_NAME,
locale: LOCALE,
images: [OG_IMAGE],
};
// and in any page that sets its own openGraph
openGraph: {
...OG_BASE,
type: "article",
title: post.title,
publishedTime: post.date,
},It's one extra line on the pages that need it. Not a big deal, but it took me a while to figure out why the image had suddenly vanished.
Things I'm ignoring
There's a lot of SEO advice floating around that I just don't think is worth spending time on. These are the things I decided to leave alone.
- The keywords meta tag. Google has ignored it for well over a decade.
- Sitemap priority and changeFrequency. Also ignored.
- FAQ schema. FAQ rich results stopped showing in May 2026. Rich result types get retired, so check what's still live before building for one.
- llms.txt and other AI-only files. Google Search doesn't read them. Its AI answers are built on the normal index, so a page that's indexed and readable is already in the running.
- A perfect Lighthouse score. Core Web Vitals are a real ranking signal and a small one. Google measures real users, not my laptop. And a fast page with a weak answer still loses to a slow page with the best one.
My test for any SEO claim now is pretty simple: which part of the pipeline does it actually affect, and does Google's documentation say that it does? If I can't answer either, I don't bother with it.
My Checklist
This is what I check for every website I ever make :)
That's the whole list. None of it is rocket science!
If you want to go deeper
- How Google Search works (opens in a new tab). The pipeline, officially. Fifteen minutes.
- Google's SEO Starter Guide (opens in a new tab). The complete basics on one page.
- JavaScript SEO basics (opens in a new tab). How Googlebot treats JavaScript.
- Rendering on the Web (opens in a new tab). The best single article on where your HTML gets made.
- generateMetadata (opens in a new tab) in the Next.js docs. Skim it now, come back often.