Uses of Class
com.norconex.collector.http.robot.RobotsTxt

Packages that use RobotsTxt
com.norconex.collector.http.crawler   
com.norconex.collector.http.handler   
com.norconex.collector.http.handler.impl   
 

Uses of RobotsTxt in com.norconex.collector.http.crawler
 

Methods in com.norconex.collector.http.crawler with parameters of type RobotsTxt
 void IHttpCrawlerEventListener.documentRobotsTxtRejected(HttpCrawler crawler, String url, IURLFilter filter, RobotsTxt robotsTxt)
           
 void HttpCrawlerEventAdapter.documentRobotsTxtRejected(HttpCrawler crawler, String url, IURLFilter filter, RobotsTxt robotsTxt)
           
 

Uses of RobotsTxt in com.norconex.collector.http.handler
 

Methods in com.norconex.collector.http.handler that return RobotsTxt
 RobotsTxt IRobotsTxtProvider.getRobotsTxt(org.apache.http.impl.client.DefaultHttpClient httpClient, String url)
           
 

Methods in com.norconex.collector.http.handler with parameters of type RobotsTxt
 void IDelayResolver.delay(RobotsTxt robotsTxt, String url)
           
 

Uses of RobotsTxt in com.norconex.collector.http.handler.impl
 

Methods in com.norconex.collector.http.handler.impl that return RobotsTxt
 RobotsTxt DefaultRobotsTxtProvider.getRobotsTxt(org.apache.http.impl.client.DefaultHttpClient httpClient, String url)
           
 

Methods in com.norconex.collector.http.handler.impl with parameters of type RobotsTxt
 void DefaultDelayResolver.delay(RobotsTxt robotsTxt, String url)
           
 



Copyright © 2009-2013 Norconex Inc.. All Rights Reserved.