Google: Paginate Your Infinite Scroll So It’s Captured By Crawlers
Infinite scroll is a fabulous way to deliver content to users while they browse your news feed or peruse your pinboard. The problem is that Google’s search-engine spiders—which can’t always emulate the same behavior that users would produce when viewing your page—may have issues with infinite scroll.
To circumvent this, Google recommends that you create a paginated version of your infinite scroll. The good news is that you can do so without affecting the look or feel of your website while allowing search-engine spiders as much access as they need to crawl your items. If your content is paginated correctly, you can ensure that duplicate content and other errors that could potentially hurt you won’t exist.
Think of pagination like you would the pages of a book that lists a certain number of items per page. You can reliably flip to any page and know just how many items are there without repeating previous items or missing others.
Your website’s infinite scroll should work like this. It’s advisable that you can carve all of your content into pages that can be loaded manually with plain URLs or with dynamic scripts. As the graphic above illustrates, it’s also important to insert rel=”next” and rel=”prev” markups to alert search engines that there are multiple pages of the same thing.
Optimal URL Structures
When reconfiguring your infinite scroll website to work with a basic version of pagination, it’s important that your URLs work by themselves. You shouldn’t have to rely on POST data or anything that would require more than a URL.
When creating a paginated version of your infinite scroll, you should avoid using data in the URL that may be considered relative. This includes relative time offsets, such as specifying data that was relevant three days ago, and parameters like a local search radius that would otherwise be non-specific to search engines, which in turn can cause duplicate results and other undesirable consequences.
The last step is to test your infinite scroll page to ensure it works properly. Verify that the “bare-bones” version of your infinite scroll page works with scripts disabled in your browser.
To ensure that search-engine spiders know when to stop, always make pages that are out-of-bounds return a 404 error. This will minimize the occurrences of harmful factors like duplicate content penalties or perceived missing pages in search engines.