XML sitemaps for large websites

Our online web links submission website smaart.in has grown quite rapidly and quite big. Our team was out to find the best solution to generate a large sitemap. We did some research on the sitemaps and found the followings.


04Oct

XML sitemaps for large websites

Posted By:- Rocky Singh

Our online web links submission website smaart.in has grown quite rapidly and quite big. Our team was out to find the best solution to generate a large sitemap. We did some research on the sitemaps and found the followings.



 

What was the issue:
XML-sitemaps.com allows only 500 URLs in an XML file. If your website is large then you have to go for paid package. There might be many other online services which allow XML sitemap generation for Google but we always prefer xml-sitemaps.com as this is easy to use.



 

In our research we came to know the following about XML sitemap files:

 
  • XML Sitemap file should not contain more than 50,000 links. If your website has more than 50,000 links, create multiple XML files.
  • You can create an index XML file with links to individual sitemap files. And this file should not contain more than 1000 file links.


 

Solution:
We tested a lot of open source scripts and desktop based website sitemap generation software but none of them worked for us. (I am not going to name the softwares we tested but going to list some resources below where we found many scripts/softwares which we tested.)



 

The main reasons why they did not work for us were the followings:

 
  • Some of the scripts are listed as best scripts but they are just not working.
  • Some scripts which work, were taking too long (more than 5 hours) to generate a large sitemap.
  • Some of the scripts do not have the ability to create multiple sitemap scripts and index sitemap file.


 

Our conclusion:
We decided not use any third party script but go for our own database based XML sitemap script. The custom database based sitemap script works in 2 parts.

 
  • It generates multiple sitemap files, each file having not more than 50,000 links.
  • It generates index sitemap script file which we submit to the Google.

 

The solution is working perfectly fine.




Resources:




Sitemap scripts & softwares
http://code.google.com/p/sitemap-generators/wiki/SitemapGenerators

XML Sitemap format
http://www.sitemaps.org/protocol.html

XML index file format
http://www.sitemaps.org/protocol.html#index

https://support.google.com/webmasters/answer/71453?hl=en



BACK TO BLOG