com.norconex.collector.http.robot
Classes 
RobotsMeta
RobotsTxt