Determining Bias to Search Engines from Robots.txt

College of Information Sciences and Technology
The Pennsylvania State University
University Park, PA 16801, USA