Make your website a better business tool in 30 minutes or less: Get Instant Access »


A spider web covered in dew
Jul 09 2020

Getting Search Engine Spiders to Crawl Your Website

I know, I know, the idea of having eight-legged beasts crawl my website gives me the heebie-jeebies, too. But organizing your site like a well-structured web is honestly the key to the SEO prize you want: getting seen when your potential customers query a search engine.

I apologize for the arachnid symbology, as I often have to when discussing SEO. 

Let’s talk about crawl accessibility! If you read my last blog “What is SEO Exactly?”, you might remember that I talked about something called “indexing.” That’s the process where search engine bots find your website, then “crawl” through and add every page to the digital library that makes up the internet.

Build a Crawlable Web with Your Site Structure

You are responsible for making sure these crawlers don’t have to jump from page to page with no links for them to follow. Every page on your website that you want to be easily found should easily link back to your homepage URL without too many other pages in between. You can think of these as “internal links”—and the more you have of them, the more likely that the web crawlers won’t miss a page.

(Note: This doesn’t mean that you have to put everything on one page! In fact, we don’t recommend doing that from an SEO or a content strategy standpoint. It just means you’ll want to make getting from here to there as simple as possible.)

Let’s say you’re selling a peplum top on your store’s site. Linking from the homepage can be as easy as… [Homepage] → [Shop] → [Tops] → [Peplum]

That’s three clicks, and the highest recommended number is usually four. And depending on the way your homepage menu is set up, it could be even simpler than that. You might have a “Shop” menu item that drops down to show all the different clothing-item categories customers can choose from. That means you get from homepage to peplum in two clicks.

A good standard to keep in mind: the more vital the content, the fewer clicks it should take to get there. If an item or topic is more specific, then it’s okay if it takes more clicks to get there.

Make things easy for the robot spiders, and people will be more likely to see your sassy shirt in search results.

Search Engine Crawlers Don’t “Get” Images or Animation

What you’ve heard is true: your audience loves video! Here at Upswept, we also love to use photos to give all our readers a feast for the eyes. But while the presence of these photos make for diverse content and a great user experience, getting a search engine to pay attention to them requires some extra setup.

When a search engine spider crawls your site, it doesn’t see the images that many of us do. They understand HTML, which is made up of a lot of words. There are two ways to communicate the meaning of your images, video, or animation with words: file names and alt-text.

If a photo you want to use has a filename like “IMG1212.png”, that doesn’t give the crawler any useful information. Using a filename that is quick and relevant to the content of your page, like “Purple-Peplum-Top.png”, is going to bump up the relevance of your image a lot more, especially if the name of the page is in the file name, too. Just make sure you don’t use more than 5 words.

Nearly every website content management system has a field for alt-text when you load up a photo. Leaving that space blank is a missed opportunity. For alt-text, you can use a few more words, but you should keep them honest and concise. 

Another important, non-SEO reason to do this is to make your site more accessible to people with visual processing disabilities, or anyone navigating your site using assistive technologies. Setting up alt-text is a winning idea all around.

Other SEO Opportunities to Consider

There are a couple other places where people miss easy SEO content opportunities. For example, in addition to images, the crawlers can’t read PDFs. If you have information in a PDF that’s important to a search engine, consider copying that content to an actual web page, and save that PDF for your more direct marketing efforts. 

Similarly, search engines also can’t parse audio files! So, if you want the content of a video or audio file to contribute to your searchability, consider springing for a transcription and including it on your website.

There are also important ways to boost your crawl accessibility that, if you’re not a web developer, you might need some help with, including page redirects, security certificates, and submitting sitemaps. And if you’re looking for some help with that, I happen to know somebody who can take a look. 😉

One thought on “Getting Search Engine Spiders to Crawl Your Website

Leave a Reply

Your email address will not be published.