URL Filters Sections

URL filters allow you to easily control Project downloads by setting which files should be loaded and which should be skipped.

URL Filters are divided into four parts:

• Protocol - restrict loading by protocol (http://www.server.com/dir1/dir2/file.htm);
• Server - restrict loading by server name (http://www.server.com/dir1/dir2/file.htm);
• Directory - restrict loading by path (http://www.server.com/dir1/dir2/file.htm);
• Filename - restrict loading by filename and link names (http://www.server.com/dir1/dir2/file.htmand text between tags).
  

Please note that if a URL doesn’t contain a filename (such as http://www.srv.com/path) it should be ended with “/” slash. Otherwise "path" will be recognized as a filename.

The URL Filter sections work as a group. If one filter specifies that a file is not to be downloaded, it will not be downloaded even if subsequent filters allow it.

Each filter has three modes:

• Load files from all... - load files without any restriction.
• Load files only within the starting... - load file only if its URL corresponds to the starting URL.
  • The file is allowed for downloading only if the corresponding URL segment is the same, as in the starting URL (Project | Address (URL)). For the
  • ... domain - the file is allowed for downloading only if the domain is the same, like in the starting URL (Project | Address (URL)). For example, if the starting server is www.server.com, loading will be enabled from cgi.server.com, images.server.com, etc.
  • ... server - the file is allowed for downloading only if the server name is the same, as in the starting URL (Project | Address (URL)). For example, if the starting server is www.server.com, loading will be enabled only from www.server.com. Loading from all other servers will be disabled.
  • Load up to X links from other servers - Pages outside of the starting server or domain may be downloaded by using the other servers limit. Loading up to one links means that Offline Explorer Enterprise will load all external links but will not go any further. Setting it to two will load pages by external links and all pages that are linked to them.
  • ... directory - the file directory will be downloaded if its path is equal to or begins with the path in the starting URL. Please note that if a URL doesn’t contain a filename (such as http://www.srv.com/path) it should be ended with ”/” slash. Otherwise "path" will be recognized as filename.
• Custom configuration - include or exclude files using keywords to filter URLs.
  • The Custom URL filters mode uses keywords to filter URLs.
  • There are two types of keywords - to enable and to disable file loading if the keyword matches. There are two lists for each type of keyword.
  • Offline Explorer Enterprise compares the corresponding URL segment of every file to be downloaded to the keywords of each type. If any of keywords in the Include list matches then the file is allowed for download. If none of the keywords match then the file will not be loaded. If any of keywords in the Exclude list matches then the file is not allowed for download. If none of the keywords match then the file will be loaded. The keyword may include any symbol. To match to a keyword means that the corresponding URL segment (server name, path or filename with extension) contains the keyword.
  • Examples: server name www.zdnet.com contains keyword zdnet. Path /~user015/images/ does not contain keyword user705.
  • The two special symbols "^" and "$" indicate the beginning and end of the URL segment.
  • For example, keyword ^www.cnn means that the URL segment should begin from www.cnn. Keyword .htm$ means that URL segment should end with the .htm.
  • To define a set of possible symbols on a position, use [] characters. Thus, keyword g[eo]t matches both get and got words. The symbol "-" means a range from one symbol to another.
  • For example, [a-z] means all symbols from a to z (including.) [0-9]means all numeric symbols.
  • Use an asterisk "*" for any number of any symbols, such as ^a*.gif$, which means all files that begin with an a and end with .gif.
  • If you want to exclude a directory, such as /ads/, don’t forget to place "/" slashes around it. Otherwise other directories, such as /leads/, will also be excluded from downloading.
  • You may specify the server name (starting with http://, ftp://, https://, rtsp://, pnm://, mms:// or mmst://) in the Directory keywords.
  • You may also specify full URLs in Filename keywords.
  • For example, the following keyword in the URL Filters | Filename | Custom configuration is allowed: http://www.srv.com/directory/img[0-9]/*.gif URL Filters | Directory | Custom configuration can contain the following keyword: http://www.zdnet.*/newfiles/
  • You can also specify a keyword to look in link names (text between tags): link:someword
  • This keyword will work only in the URL Filters | Filename section.
  • The "tree" icon next to the keyword field shows the map of the downloaded site to choose a filename, directory or server as a keyword. Note: the keyword check is case insensitive.
  • Tip: To filter URLs that end with a slash (no filename specified), use default.htm in URL Filters | Filename section.
  • With the exception of the Protocol URL filter, the Custom Configuration sections of each URL Filter contain an Include and Exclude list, and you may enter any number of keywords into each of these lists. If a keyword matches the corresponding URL segment in an Include list, then the file will be downloaded. If a keyword matches the corresponding URL segment in an Exclude list, then the file will not be downloaded.

Server keyword examples:

Directory keyword examples:

Filename keyword examples:

The Protocol URL Filter Custom Configuration section allows selection or deselection of each protocol directly.

Some useful keys to manage URL Filters keywords lists:

• Ctrl-L loads keywords from a text file to the currently selected list.
• Ctrl-M moves all keywords in the current list to another (like from Included to Excluded).
• Ctrl-S adds the starting server, directory or filename to the selected keywords list. For example, when you want to allow loading only from the starting directory in the URL Filters | Directories | Custom configuration.
• Ctrl-C copies all keywords to Windows clipboard.
• Ctrl-A checks all keywords in the list.
• Ctrl-N unchecks all keywords.
• Press F2 key on a selected keyword to edit it in-place.

Note: URL Filters settings do not apply to files, which File Filters category doesn't have "Load using URL Filters setting" selected in its Location box.

You can also use URL Macros in the URL Filters keywords. For example:

filename{:0day}.htm

Tip: To filter URLs that end with a slash (no filename specified), use default.htm in URL Filters | Filename section.

Note: Offline Explorer Enterprise skips a URL if any of the URL Filters or File Filters setting doesn't allow it to be loaded (even if all other Project settings allow the URL to be loaded).