Track Search Engine Robot Activity With SEO Crawlytics
By default the plugin works with Google and Bing however via the options area you can track other search engines by simply adding the robot name, mask and reverse DNS for the site. The plugin can send you daily reports about the search engine robots that crawled your site via email.
You can also receive instant notification via email whenever a search engine robot has visited your site, though doing so wouldn’t be very practical as your inbox would be full of notifications very quickly. Additional email addresses can be set in the notification area, which is useful if there is more than one admin for your website.
One of the best features of SEO Crawlytics is the ability to track search engine robots from other parts of your website. For example, by adding the integration code into the footer of your forum, you can also track the activity of robots in your discussion forum. This is a welcome addition as most WordPress plugins don’t offer any functionality outside of WordPress itself.
SEO Crawlytics provides a variety of reports about search engine crawling activity on your site. In each graph, a different colour is used for the search engine robots that you have configured in the options area. This helps you distinguish stats for each search engine in the reports.
The hourly visits graph is a great way of seeing hourly activity over a two week period for each of the search engines. The graph will highlight spikes in crawling activity and the times of the day most search engines visit.
The same data can be shown on a daily basis. This graph is a bit more user friendly and shows the activity for each search engine robot on a per day basis rather than showing activity spikes on an hourly basis.
One benefit of using SEO Crawlytics over non-WordPress solutions is it’s ability to track activity on a per post basis. The plugin shows you the posts and pages that are being crawled most on your site. It also shows you the post categories that are being crawled the most by search engines.
The peak crawl times graph shows you the times of the day search engine robots have crawled your website over the last 24 hours. It’s essentially the same information as the hourly visits graph except data is shown from the last day rather than the last two weeks. Crawl times are shown on a collective basis too rather than being broken down for each search engine robot. It’s a useful graph for understanding why your server load increases at certain times of the day.
The crawl spread pie chart shows you what search engines are crawling your website the most (for most websites, Google will have the biggest share).
There’s a lot to like about SEO Crawlytics. It’s easy to setup, produces some useful reports about search engine crawling activity on your website and best of all – it’s free to download. I recommend trying it out if you want to track what search engines are crawling your website every day.