Answers: As far as my knowledge say it is we have to upload a robot.txt record on the root of your server and you have commonly following on it:
User-agent: *
Disallow:
This means allow adjectives user agent of search engine to crawl your website, disallow non. If you want Google not crawl your website you obligation to disallow Google