#Sitesucker login download
In addition to server-wide robot control using robots.txt, Web page creators can also use the Robots META tag to specify that the links on a page should not be followed by robots.Ĭheck this box to have SiteSucker automatically start the download after you drag a URL into the Web URL text field.Ĭheck this box to have SiteSucker append an "html" file extension to downloaded HTML files that lack an "html" or "htm" file extension. Web administrators can disallow access to cgi and private and temporary directories, for example, because they do not want pages in those areas downloaded. The robots.txt file allows the Web site administrator to define what parts of a site are off-limits to specific robots, like SiteSucker. Robot exclusions are usually put in place for a good reason and should be obeyed.īy default, SiteSucker honors robots.txt exclusions and the Robots META tag. Warning: Ignoring robot exclusions is not recommended. For more information on authentication, see Password-protected Sites.Ĭheck this box to have SiteSucker ignore robots.txt exclusions and the Robots META tag. If nothing is found in the Keychain, it displays the Login Dialog.Ĭheck this box to suppress display of the Login Dialog and skip the download of any pages that require authentication. Whenever SiteSucker encounters a page that requires authentication, it first looks for the appropriate credentials in the Keychain. The General tab in the Download Settings dialog provides the following settings:
0 kommentar(er)
