?
Also what does user-agent signify?
if they don't have robots.txt then you can crawl
user-agent specifies which crawlers the following rules apply to
Обсуждают сегодня