You are allowed to crawl the dblp website. We do even provide a very simple XML query API which is documented here:

• Michael Ley: DBLP XML Requests . Appendix to a paper from VLDB 2009.

However, if the number of requests from a remote location becomes so high that it threatens the operation of the website, please understand that we might have to time-out or block your queries temporarily. For bulk queries, please consider "sleeping" your process for a second or two between two queries.

If you know that you need to do a lot of queries to the dblp data stock, we encourage you to please download the whole dblp dataset as a single XML file instead and do your queries locally. The big XML file is in sync with the data on the websites.

robots.txt

Sometimes, we do have a robots.txt in place to limit crawling on some sections of the site. This is never done to hide any of our information from you or your crawlers, but rather to avoid confusing crawlers with duplicate or testing branches of the website. Hence, please respect our robots.txt.