Enhance your site's crawlability and indexability by search spiders.
Keywords and content are the two main pillars upon which most search engine optimization strategies are built, but they are far from the only two that are important.
Discover your website less commonly but just as importantly - not just for users but for search bots.
There are approximately 50 billion web pages on 1.93 billion websites on the Internet. This is too much for any human team to explore, so these robots, also called spiders, play an important role.
These bots determine the content of each page by following links from website to website and from page to page. This information is compiled into an extensive database, or index, of URLs, which is then put together by a search engine's algorithm for ranking.
This two-step process for navigating and understanding your site is called crawling and indexing.
As an SEO professional, you've undoubtedly heard these terms before, but let's define them just for the sake of clarity:
Crawlability refers to how well these search engine robots can scan and index your web pages.
Indexability measures the ability of a search engine to analyze your web pages and add them to their index.
As you can probably imagine, both are an essential part of SEO.
If your site has poor crawlability, for example, many broken links and dead ends, then search engine crawlers will not be able to access your content, which will result in it being excluded from the index.
On the other hand, indexability is vital because pages that have not been indexed will not appear in search results. How can Google rank a page that is not included in its database?
Crawling and indexing are a bit more complicated than what we've discussed here, but that's the basic overview.
If you're looking for a more in-depth discussion of how it works, Dave Davies has an excellent article on crawling and indexing.
How to improve crawling and indexing
Now that we've looked at how important these two processes are, let's look at some of the elements of your website that affect crawling and indexing — and discuss ways to optimize your site for them.
1. Improve page loading speed
With billions of web pages being indexed, web spiders don't have all the day to wait for your links to load. This is sometimes referred to as crawl budgeting.
If your site does not load within the specified time frame, it will leave the site, which means you will remain uncrawled and not indexed. And as you can imagine, this is not good for SEO purposes.
Thus, it is a good idea to regularly evaluate your page speed and improve it wherever you can.
You can use Google Search Console or tools like Screaming Frog to check the speed of your website.
If your site is running slowly, take steps to mitigate the problem. This may include upgrading the server or hosting platform, enabling compression, reducing CSS, JavaScript, and HTML, and removing or reducing redirects.
Find out what's slowing down loading time by reviewing the Core Web Vitals report. If you want more accurate information about your goals, especially from a user-centric point of view, Google Lighthouse is an open source tool that you may find very useful.
2. Strengthen Internal Link Structure
A good site structure and internal linking are foundational elements of a successful SEO strategy. A disorganized website is difficult for search engines to crawl, which makes internal linking one of the most important things a website can do.
But don’t just take our word for it. Here’s what Google’s search advocate John Mueller had to say about it:
“Internal linking is super critical for SEO. I think it’s one of the biggest things that you can do on a website to kind of guide Google and guide visitors to the pages that you think are important.”
If your internal linking is poor, you also risk orphaned pages or those pages that don’t link to any other part of your website. Because nothing is directed to these pages, the only way for search engines to find them is from your sitemap.
To eliminate this problem and others caused by poor structure, create a logical internal structure for your site.
Your homepage should link to subpages supported by pages further down the pyramid. These subpages should then have contextual links where it feels natural.
Another thing to keep an eye on is broken links, including those with typos in the URL. This, of course, leads to a broken link, which will lead to the dreaded 404 error. In other words, page not found.
The problem with this is that broken links are not helping and are harming your crawlability.
Double-check your URLs, particularly if you’ve recently undergone a site migration, bulk delete, or structure change. And make sure you’re not linking to old or deleted URLs.
Other best practices for internal linking include having a good amount of linkable content (content is always king), using anchor text instead of linked images, and using a “reasonable number” of links on a page (whatever that means).
Oh yeah, and ensure you’re using follow links for internal links.
3. Submit Your Sitemap To Google
Given enough time, and assuming you haven’t told it not to, Google will crawl your site. And that’s great, but it’s not helping your search ranking while you’re waiting.
If you’ve recently made changes to your content and want Google to know about it immediately, it’s a good idea to submit a sitemap to Google Search Console.
A sitemap is another file that lives in your root directory. It serves as a roadmap for search engines with direct links to every page on your site.
This is beneficial for indexability because it allows Google to learn about multiple pages simultaneously. Whereas a crawler may have to follow five internal links to discover a deep page, by submitting an XML sitemap, it can find all of your pages with a single visit to your sitemap file.
Submitting your sitemap to Google is particularly useful if you have a deep website, frequently add new pages or content, or your site does not have good internal linking.
4. Update Robots.txt Files
You probably want to have a robots.txt file for your website. While it’s not required, 99% of websites use it as a rule of thumb. If you’re unfamiliar with this is, it’s a plain text file in your website’s root directory.
It tells search engine crawlers how you would like them to crawl your site. Its primary use is to manage bot traffic and keep your site from being overloaded with requests.
Where this comes in handy in terms of crawlability is limiting which pages Google crawls and indexes. For example, you probably don’t want pages like directories, shopping carts, and tags in Google’s directory.
Of course, this helpful text file can also negatively impact your crawlability. It’s well worth looking at your robots.txt file (or having an expert do it if you’re not confident in your abilities) to see if you’re inadvertently blocking crawler access to your pages.
Some common mistakes in robots.text files include:Robots.txt is not in the root directory.
- Poor use of wildcards.
- Noindex in robots.txt.
- Blocked scripts, stylesheets and images.
- No sitemap URL.
5. Check Your Canonicalization
Canonical tags consolidate signals from multiple URLs into a single canonical URL. This can be a helpful way to tell Google to index the pages you want while skipping duplicates and outdated versions.
But this opens the door for rogue canonical tags. These refer to older versions of a page that no longer exists, leading to search engines indexing the wrong pages and leaving your preferred pages invisible.
To eliminate this problem, use a URL inspection tool to scan for rogue tags and remove them.
If your website is geared towards international traffic, i.e., if you direct users in different countries to different canonical pages, you need to have canonical tags for each language. This ensures your pages are being indexed in each language your site is using.
6. Perform A Site Audit
Now that you’ve performed all these other steps, there’s still one final thing you need to do to ensure your site is optimized for crawling and indexing: a site audit. And that starts with checking the percentage of pages Google has indexed for your site.
Check Your Indexability Rate
Your indexability rate is the number of pages in Google’s index divided by the number of pages on our website.
You can find out how many pages are in the google index from Google Search Console Index by going to the “Pages” tab and checking the number of pages on the website from the CMS admin panel.
There’s a good chance your site will have some pages you don’t want indexed, so this number likely won’t be 100%. But if the indexability rate is below 90%, then you have issues that need to be investigated.
You can get your no-indexed URLs from Search Console and run an audit for them. This could help you understand what is causing the issue.
Another useful site auditing tool included in Google Search Console is the URL Inspection Tool. This allows you to see what Google spiders see, which you can then compare to real webpages to understand what Google is unable to render.
Audit Newly Published Pages
Any time you publish new pages to your website or update your most important pages, you should make sure they’re being indexed. Go into Google Search Console and make sure they’re all showing up.
If you’re still having issues, an audit can also give you insight into which other parts of your SEO strategy are falling short, so it’s a double win. Scale your audit process with tools like:Screaming Frog
Semrush
Ziptie
Oncrawl
Lumar
7. Check For Low-Quality Or Duplicate Content
If Google doesn’t view your content as valuable to searchers, it may decide it’s not worthy to index. This thin content, as it’s known could be poorly written content (e.g., filled with grammar mistakes and spelling errors), boilerplate content that’s not unique to your site, or content with no external signals about its value and authority.
To find this, determine which pages on your site are not being indexed, and then review the target queries for them. Are they providing high-quality answers to the questions of searchers? If not, replace or refresh them.
Duplicate content is another reason bots can get hung up while crawling your site. Basically, what happens is that your coding structure has confused it and it doesn’t know which version to index. This could be caused by things like session IDs, redundant content elements and pagination issues.
Sometimes, this will trigger an alert in Google Search Console, telling you Google is encountering more URLs than it thinks it should. If you haven’t received one, check your crawl results for things like duplicate or missing tags, or URLs with extra characters that could be creating extra work for bots.
Correct these issues by fixing tags, removing pages or adjusting Google’s access.
8. Eliminate Redirect Chains And Internal Redirects
As websites evolve, redirects are a natural byproduct, directing visitors from one page to a newer or more relevant one. But while they’re common on most sites, if you’re mishandling them, you could be inadvertently sabotaging your own indexing.
There are several mistakes you can make when creating redirects, but one of the most common is redirect chains. These occur when there’s more than one redirect between the link clicked on and the destination. Google doesn’t look on this as a positive signal.
In more extreme cases, you may initiate a redirect loop, in which a page redirects to another page, which directs to another page, and so on, until it eventually links back to the very first page. In other words, you’ve created a never-ending loop that goes nowhere.
Check your site’s redirects using Screaming Frog, Redirect-Checker.org or a similar tool.
9. Fix Broken Links
In a similar vein, broken links can wreak havoc on your site’s crawlability. You should regularly be checking your site to ensure you don’t have broken links, as this will not only hurt your SEO results, but will frustrate human users.
There are a number of ways you can find broken links on your site, including manually evaluating each and every link on your site (header, footer, navigation, in-text, etc.), or you can use Google Search Console, Analytics or Screaming Frog to find 404 errors.
Once you’ve found broken links, you have three options for fixing them: redirecting them (see the section above for caveats), updating them or removing them.
10. IndexNow
IndexNow is a relatively new protocol that allows URLs to be submitted simultaneously between search engines via an API. It works like a super-charged version of submitting an XML sitemap by alerting search engines about new URLs and changes to your website.
Basically, what it does is provides crawlers with a roadmap to your site upfront. They enter your site with information they need, so there’s no need to constantly recheck the sitemap. And unlike XML sitemaps, it allows you to inform search engines about non-200 status code pages.
Implementing it is easy, and only requires you to generate an API key, host it in your directory or another location, and submit your URLs in the recommended format.
Wrapping Up
By now, you should have a good understanding of your website’s indexability and crawlability. You should also understand just how important these two factors are to your search rankings.
If Google’s spiders can crawl and index your site, it doesn’t matter how many keywords, backlinks, and tags you use – you won’t appear in search results.
And that’s why it’s essential to regularly check your site for anything that could be waylaying, misleading, or misdirecting bots.
So, get yourself a good set of tools and get started. Be diligent and mindful of the details, and you’ll soon have Google spiders swarming your site like spiders.