Individuals use DA to check out which Sites are trusted inside their field and to figure out where by to acquire superior backlinks. txt file is then parsed and will instruct the robot concerning which web pages will not be to become crawled. To be a online search engine crawler https://www.youtube.com/watch?v=0USI3vTZ0aw