instructions around the usage of meta robot tags and robots.txt files · CMSgov/price-transparency-guide@bc8e96e

Original file line number	Diff line number	Diff line change
`@@ -46,6 +46,13 @@ These machine-readable files post made available to the public without restricti`
`46`	`46`
`47`	`47`	`The location of the these URLs must be provided over HTTPS to ensure the integrity of the data.`
`48`	`48`
	`49`	`+Robots.txt`
	`50`	`+----------`
	`51`	`+`
	`52`	+To allow for search engine discoverability, neither a `robots.txt` file nor `meta` tag on the page where the files are hosted will have rules such that give instructions to web crawlers to not index the page.
	`53`	`+`
	`54`	+This is typically follows the format of `<meta name="robots" content="noindex, nofollow>"` or for a `robots.txt` file the usable of the `Disallow` directive.
	`55`	`+`
`49`	`56`	`Special Data Types`
`50`	`57`	`------------------`
`51`	`58`