Why your site is not found: technical obstacles that announce nothing

There is a layer that operates before content and before links: whether the site permits being found at all. What makes it treacherous is that it signals nothing. There is no error message, the site opens, it looks tidy and may be beautifully built. It is simply absent from the results. This is about the common obstacles of that kind, and how to check for them without a supplier.

In short

• The most common cause is not competition — it is a leftover instruction on the site itself telling search engines not to index it.
• Robots.txt and noindex are two different things, and they are frequently confused in a way that makes one cancel the other.
• Several pages about the same thing divide each other's weight; the fix is usually to merge rather than to write another.
• URL structure is the decision most expensive to change later, and it gets taken earliest.
• The effect of changes is measured in weeks and as a trend, not in daily figures.

The first place to look

The order is dull but it matters: check first whether the site has forbidden its own indexing. During a build this is done deliberately, so an unfinished site stays out of results. The directive has to be removed at launch, and that is precisely what sometimes does not happen. From the outside the site looks entirely normal.

Alongside it lives a second mistake born of good intentions. Robots.txt and a noindex directive work differently: the first restricts crawling, the second restricts appearing in results. If a page is blocked in robots.txt, Google may never see the noindex directive, because seeing it would require crawling the page. The outcome is the opposite of the intent: the URL can remain in results, and you have no way to change that until the block is lifted.

Pages that compete with each other

The second most common problem is not a technical fault but a structural one. The same service is described on two or three pages — an older one, a newer one, and a landing page made at some point. All three partly answer the same question, so links and signals divide, and it is not obvious to a search engine which to show.

The usual response — write another, better page — makes it worse. The correct fix is normally to merge into one and redirect the rest. That is uncomfortable, because somebody has to decide which page survives, and decisions of that kind stall inside companies for longer than any technical work.

What URL structure tells you

URLs look cosmetic until they need changing. Then it turns out they are one of the few decisions whose cost grows with time: every change requires redirects, every redirect is somewhere to make a mistake, and the mistake is not noticed immediately.

• One URL per piece of content. If the same page is reachable at several addresses, choose the primary one and declare it.
• Normalise protocol and domain form with a single rule, not a chain of them.
• Redirect straight to the final address: chains of redirects are where the most is lost.
• If content is gone and there is no equivalent, an error code is more honest than a redirect to the home page.
• Avoid URLs carrying meaningless parameters — they create duplicates for no benefit.

Speed: where it genuinely matters

Speed is often presented as the main cause, and that is usually overstated. A very slow site does get in the way, but a site that opens in a couple of seconds is rarely the reason it cannot be found. It is more useful to treat speed as a user metric rather than a search one: if someone closes the page before seeing the content, no amount of visibility helps.

The one case where speed is a real technical obstacle is a server that responds unreliably or with long delays. Then crawling becomes inefficient, and that is already visible in Search Console.

Illustrative scenario, not a description of a client project

Imagine a new site that does not appear in results after launch. Content is suspected, a rewrite is commissioned, links are planned. Two months later somebody notices that one template still carries the do-not-index directive — the same one switched on during the build. None of the work done was bad; it simply could not have any effect while the obstacle was in place. This is why the technical check comes before content work rather than after it. An example of how we weigh sequence, not a promise about an outcome.

A check you can run this week

• Connect Search Console if it is not connected. Without it every other step is guesswork.
• Check a few of your most important pages: are they indexed, and if not, what reason is given.
• Check for a leftover do-not-index directive, and whether robots.txt blocks anything you want visible.
• Find two pages that talk about the same thing and decide which one survives.
• Check where old URLs lead, if the site has ever been rebuilt.

These five will not solve visibility, but they answer the more important question: whether working on content makes sense yet. If an obstacle is in place, the best writing changes nothing — and that costs far more than the check itself.