Problem
My personal website pages built with NextJs v14.x.x hosted by Vercel were not indexed at all for ages somewhere down the line ... very shocking...
Details in Google Search Console
When you visit Sitemap page, it says Couldn't fetch.
When I visited URL Inspection page, it says Site-wide availability issues.
Investigating Process
However I could see the content https:mypage/sitemap.xml on browser.
curl -I https:mypage/sitemap.xml shows 200 response status properly.
Then I checked more details in Site-wide availability issues on https://support.google.com/webmasters/answer/9012289#will_i_be_indexed&zippy=%2Csite-wide-availability-issues
and then found this one:
Robots.txt unreachable: Google won't crawl a website if the robots.txt file is present but not reachable. You can check your robots.txt availability in the Crawl Stats report.
Then I realised thathttps:mypage/robots.txt returns 500 status despite the fact that I didn't prepare the robots.txt in my NextJS directory.
Running production mode in NextJS hides the error details while dev mode shows you the details.
Voila, my poor error handling returns 500 status even though /robots.txt does not exist which in turn confused Google Indexer and put it the label of Not Indexable Page.
The case is closed.
Top comments (0)