First, Video Killed the Radio Star, and now Google Killed Their Crawl Errors Report. While you may be wondering what the world is coming to (or why we made that obscure 80s reference), you may have seen the warnings:
Site managers and SEOs alike have relied on Google’s Crawl Errors report to identify and fix broken pages and links for years. Now that has changed.
The Crawl Errors report will soon be removed and replaced with the Index Coverage report. The Index Coverage report is meant to show issues with URLs that are submitted for indexing (i.e., those submitted via sitemaps). This is meant to focus in on those issues that matter — according to Google.
While it is stated in a Google help article that “404s are only considered an error if you specifically asked us to crawl a URL, such as through a sitemap,” this may not capture all key issues. There are other scenarios where 404 errors are not captured but are still problematic.
How You Might Miss 404 Errors
Here are a few scenarios where detrimental 404s would not be captured in the Index Coverage report.
Scenario 1: Website Migration
Suppose during a website migration (like from Magento to BigCommerce, for example), URLs are not matched and are not properly redirected. This could result in a situation where a large amount of your inbound links are now pointed at URLs that no longer exist. As there can be a pretty significant lag between the time the 404 error is encountered and when it appears on the report, they can go unnoticed for a significant period of time.
Suppose you were migrating to BigCommerce from a platform like Volusion which has multiple URLs for the same product details page. Only the canonical, indexed URL would possibly see a reported 404 error. If not redirected, the other versions could possibly return a 404 error for every item in your catalog. Ouch.
Scenario 2: Broken / Incorrect Links
Suppose a key website chooses to link to your site content (yay!) but mistypes a single character in the URL (boo!). As a result, what would have been a valuable link to a specific page returns a 404 error. As this broken URL is not what was submitted for indexing, it would not be included in the report. The link specifically referencing particular content on your site does not pass the link equity and offers a poor user experience — and you would not even know!
Why This Matters
These exceptions are important to note, because many websites and ecommerce stores could experience critical issues because of them. You may be able to capture errors directly related to the indexing of URLs in your sitemap, but how many errors will you really have when your sitemap updates automatically? And, while some obscure or spammy URLs that return 404s do not matter, others do. Here’s a few key examples of why they matter.
Poor User Experience
While sometimes necessary, 404 errors never present a good user experience. Upon encountering a Page Not Found, many users will bounce and not return. Others will be left with a bad impression about your site and your business.
It is important to identify and fix 404 errors to keep future customers happy.
Loss of Link Juice / Equity
You work hard to get noticed. While SEO continues to change, links and references continue to be important. An inbound link pointing to one of your products from a reputable site could offer a boost in traffic and page authority. However, if that link ends up pointing to a 404 page, what happens to the link equity?
It can be very profitable to identify and fix these links, either by contacting the originating site or by creating a 301 redirect. While each 301 redirect causes a bit of link equity to be lost, it’s better than losing all of it by allowing a 404 to remain unfixed.
If your site is filled with broken pages (regardless of where they originate), this poor experience will not only be noticed by users but by search engines as well. This may not directly impact the way a page ranks, but could definitely impact the way your site is viewed.
What to Do About it
Website managers have long relied on Google Search Console for this data. Now that it has changed, you may be looking for new data sources.
We still recommend using the new Search Console and the Index Coverage report. Of course, Search Console will have value far beyond the immediate reporting of crawl errors. It will also be valuable to utilize the new Index Coverage report for indexing related errors.
For broken links that exist within your site, you can use tools like Screaming Frog SEO Spider or an online broken link tool. However, these will only capture internal links (originating from within your site).
When we identified this information gap and its importance to ecommerce store managers, we created the Swerve Redirects app for BigCommerce, which includes a 404 tracking tool. As 404 errors are tracked in real time, you can fix them as fast as they occur.
While it is unclear how Google will adapt Search Console in the future, we hope this information is helpful in ensuring how you can maintain a healthy, user-friendly online store today.
[Editor’s note: Evelyn Bai takes responsibility for the reference to The Buggles. She just couldn’t help herself.]