10.6. URL lists

On this page, you can create various lists of URLs and then use them as black and white lists for the content filtering rules.

Note that UserGate offers its own updatable lists. You will also need the corresponding license to use them. For more details on product licensing, please refer to UserGate licensing.

Name

Description

UserGate black list

This list contains URLs prohibited by some national laws.

Phishing black list

This list contains URLs of known phishing websites.

UserGate white list

This list contains URLs of known trusted websites and portals.

Search engines without safesearch capability

This list contains known search engines which do not provide safe search (family filter). We recommend to block such search engines for parental control, as they provide ability to get adult content.

UserGate black list (Custom code)

This list contains URLs prohibited by some national laws. The GS1 suffix code for UserGate custom black/white lists comply with the national laws of the country. See http://www.gs1.org/company-prefix for details

To set up filtering based on lists of URLs, perform the following steps:

Name

Description

Step 1. Create a new list of URLs

Click Add and specify the name for the new list

Step 2. Add the necessary records to your list

Add the necessary URLs to your list. You can use special characters ^, $ and * in the lists:

* stands for an arbitrary number of characters

^ denotes the start of the current line

$ denotes the end of the current line

Note that characters ? and # are not allowed here

Step 3. Create a new content filtering rule containing one or more lists

Please refer to Content filtering.

All records that start with http:// , https://, ftp:// or contain one or more "/" characters are handled as URLs with the HTTP(S) filtering, but not applied to the DNS filtering. Otherwise, such string will be treated as domain name and therefore will be applied for both DNS and HTTP(S) filtering.

To block a website by exact address, use the "^" and "$" characters:
To block an exact URLs of all subfolders, use the "^" character:
To block a domain with all its URLs, use the following record:
domain.com

Example of URL record interpretation:

Sample record

Handling of DNS requests

Handling of HTTP requests

yahoo.com

or

*yahoo.com*

Blocks the entire domain with its third-level domains, e.g.:

sport.yahoo.com

mail.yahoo.com

and

qweryahoo.com

Blocks the entire domain with all its URLs and third-level domains, e.g.:

http://sport.yahoo.com

http://mail.yahoo.com

https://mail.yahoo.com

http://sport.yahoo.com/123

^mail.yahoo.com$

Blocks only mail.yahoo.com

Blocks only http://mail.yahoo.com

https://mail.yahoo.com

^mail.yahoo.com/$

Nothing is blocked

Nothing is blocked, since the last "/" defines an URL, but neither "https" nor "http" are specified

^http://finance.yahoo.com/personal-finance/$

Nothing is blocked

Blocks only

http://finance.yahoo.com/personal-finance/

^yahoo.com/12345/

Nothing is blocked

Blocks

http://yahoo.com/12345/whatever/

https://yahoo.com/12345/whatever/

Network administrators can create custom lists and distribute them from a single center to all UserGate servers. To create a new list, perform the following steps:

Name

Description

Step 1. Create a new file with the necessary list of URLs

Create a new text file called list.txt with URLs presented in the following format:

www.site1.com/url1

www.site2.com/url2

...

www.siteend.com/urlN

Step 2. Put this file into a new archive

Zip the file into a new archive called list.zip

Step 3. Create a new file with the necessary version of your list

Create a new file called version.txt and specify the list version (e.g. "3") in it. Make sure to increment this value each time you update the list

Step 4. Publish files on your web server

Publish list.zip and version.txt on your website and make them available for download

Step 5. Create a new list of content types and provide an URL for installing updates

Create a new list of URLs on every UserGate server. When creating a new database, make sure to provide an URL for installing updates. UserGate will be checking for a new version on your website every 4 hours and automatically update your list once a newer version is released