Google Crawl Rate Problems With Sitemap Indexing

Over the past few days we have seen reviews from Webmasters that changing the crawl rate in Google Webmaster Tools has prevented Google from indexing their Sitemap documents.

A Google Webmaster Help thread has reports of this problem dating back five days with many webmasters complaining thereafter.

The error webmasters are seeing read:

We were not able to down load your Sitemap file due to the crawl rate we’re using for your hosting server. For information on increasing crawl rate, please see our Help Center.
Why happening? It seems like to only happen if you modify the crawl rate to the manual setting but I cannot ensure that for sure.

One particular website owner effected with that reported:

I am having the same problem too. the message yesterday, and transformed my crawl rate to manual and transferred the bar all the way up on the right to permit the quickest crawl rate, but I still the view the same “crawl rate problem” message as I saw yesterday.

So far Google has not resolved these issues in the forum.

Community forum discussion at Google Webmaster Help.

Update: Googler, JohnMu replied to the thread with more information. He explained:

This message generally signifies that we’d like to crawl much more (in this case, your Sitemap data file) from the site if your server (or crawl-rate-setting) would allow it.
When you have a manually arranged crawl-rate in Webmaster Tools, you may desire to reset that back to “let Google determine my crawl rate,” so that our systems could try and immediately raise it to match your server’s availability (the manual setting is mostly to limit it even lower). Relatively simplified: really should we see that crawling a lot of causes your server to slow down, we will generally reduce our crawl rate to avoid creating issues.
When you notice that Googlebot is regularly crawling less than you’d want, then you may be considering most of these points:

* Work to reduce the number of crawlable URLs on your web site. For instance, when you have session-IDs in your Web addresses, or utilize complex, dynamically generated URLs, that will generate a lot of crawlable URLs. Typical canonicalization techniquest can easily typically help in a case like this: http://www.google.com/support/webmasters/bin/answer.py?answer=139066

* Check your Webmaster Tools crawl-stats to ascertain if crawling of the website (and even — when you have access — other sites on the very same server) is particularly slow, and then your hoster and/or web-developer to ascertain if that can be enhanced.

* Make use of the “Report an issue with Googlebot” form behind the “learn more” link next to the crawl-rate settings. Keep in mind that if Googlebot isn’t crawling just as much as you’d want due to technological problems (too many URLs being crawled and/or server troubles), then we’d actually recommend correcting those 1st.

Expect this helps! You can post back in case you have any queries.