I Built My Own Search Engine Because Google Stopped Being One

Searching the Web Stopped Meaning Finding Things

Somewhere in the last few years, searching the web stopped meaning finding things. You get ads, SEO sludge, and the same twelve domains regardless of what you asked. If you work with AI agents, it's worse — they inherit all of that noise as "ground truth."

So I built Odin: a self-hosted search engine that runs entirely on my own hardware.

A Map, Not a Cache

The core idea is different from a traditional crawler. Odin doesn't try to mirror the internet — that race is unwinnable from a home server. Instead it maintains a map of hubs: the pages that reliably point to good answers for a topic.

Ask it about self-hosted photo management and it doesn't return a stale snapshot — it knows which community wikis, forums, and repositories are authoritative for that space, then fetches the current content live.

The index right now: over 545,000 pages across 205,000+ domains, with 476,000 vector embeddings, over 500,000 extracted entities, and a 479-hub curated taxonomy. Small by Google standards. But it's my 545,000 pages — curated toward the topics I actually research, with no advertiser thumb on the scale.

Teaching a Machine to Read the Web

Indexing URLs is table stakes. The hard problem is knowing what each page is about without a human reading it. Odin solves this with a content enrichment pipeline: every page gets fed through a local language model that extracts structured metadata — what the page offers, what category it belongs to, what search queries it answers.

The model doing this work is Qwen 4B Crow, an Opus 4.6 distill running in IQ4_XS quantization. It chews through about 103 pages per minute across 8 parallel workers, using 4.8 GB of VRAM. The pipeline is simple in concept: raw HTML gets cleaned into readable text, the model reads it and outputs a description, a category label, relevant tags, and a handful of search queries the page would answer. That structured metadata is what makes semantic search actually work.

There's a nice optimization too. Pages from the same domain usually share the same topic, so once one page on a domain has been enriched, other pages inherit that metadata automatically — no LLM needed. 63% of pages are enriched via inheritance, the remaining 37% via the model. It's the kind of boring efficiency that adds up when you're processing half a million pages on a single GPU.

What you get from all this: when you search for "self-hosted photo backup," the result isn't just a URL and a title. It's a URL, a one-sentence summary, a relevance category, and tags that tell you what's on that page before you click. Every result has been read by a machine that actually understood it.

The Unglamorous Hard Part: Actually Fetching Pages

Everyone thinks search is about ranking. Half the real work is that the modern web actively resists being read. Cloudflare walls, JavaScript-only rendering, bot detection that blocks anything without a full browser fingerprint.

Odin's fetcher runs a four-level fallback chain — from a plain HTTP request all the way up to a full browser session — and currently lands about a 95% success rate on pages that a naive fetch fails on. Video extraction has its own stack of 13 strategies for the sites that bury media behind players.

The fetcher alone is arguably the most sophisticated component. Getting content out of the modern web is an arms race, and Odin is winning more engagements than it loses.

What Google Won't Show You

Search engines make editorial decisions dressed as algorithmic ones. Adult content, torrent indexes, privacy tools, security research — entire categories of the web get quietly de-indexed or buried past page ten. It's not that the content is illegal. It's that it doesn't generate ad revenue, or it makes advertisers uncomfortable.

Odin indexes everything. The taxonomy spans from academic research and code repositories through to torrent trackers and adult platforms. Not because every category is equally important, but because the search engine shouldn't be the one deciding what you're allowed to find. That's your call.

In practice, this means Odin is genuinely useful for categories the big engines have abandoned. Looking for privacy tools that aren't on Google's whitelist? Odin has them. Researching VPN providers with real technical analysis instead of affiliate-link roundups? Odin surfaces the forums and wikis that actually test this stuff. The censorship isn't a feature you opt out of — it's a filter that was never installed.

Why This Matters for AI

Odin doubles as an MCP tool server — it speaks the protocol Claude and other AI assistants use. When my agent researches something, it doesn't get "what a search engine's ad model wants me to see." It gets live content from sources chosen for authority.

But you don't need MCP. Odin works as a native CLI tool — plain Node.js commands you can call from any script, shell, or agent harness that can spawn a process. That means you can wire it into QuetzaCodetl, OpenCode, a cron job, or a Bash script without any protocol overhead. The MCP layer is there if your framework speaks it; the CLI is there for everything else.

Research quality went up noticeably, especially for developer topics, self-hosting, and anything the big engines quietly bury. In side-by-side tests against Google across categories I care about — AI tooling, self-hosted software, developer documentation, privacy tools — Odin wins more categories than it loses. Not because it's smarter. Because it isn't fighting me.

The Numbers

This thing started as an experiment on a single RTX 4090. Here's where it stands:

• 545,044 pages indexed across 205,774 unique domains
• 476,000 vector embeddings at 87% coverage — semantic search across the full index
• 505,000 named entities extracted from page content
• 479 curated hubs in the taxonomy — authoritative sources mapped by topic
• 91% enrichment coverage — nearly every page has structured metadata
• 103 pages/min enrichment speed on a single 4B model at 4.8 GB VRAM
• 95% fetch success rate on pages a naive HTTP request can't touch

It's not competing with Google's 400 billion pages. It doesn't need to. 545,000 pages where every single one was chosen, categorized, and enriched by a machine I control — that's a different product entirely. It's the difference between a library that buys every book and a librarian who actually reads them.

Getting Odin

If you want to run your own instance — the full crawler + indexing pipeline, the MCP server for AI agent integration, a starter hub map so you're not indexing from zero, and the four-level fetch fallback that handles bot-protected pages — I've packaged it as a one-time download with a guided setup.

Get Odin — $75 AUD