url_cache.pl -- Cache the content of external URLs in local files

This library provides a cache for data stored in extenal URLs. The content of each URL is kept in a file and described by a meta-file that remembers the mime-type, the original URL, when it was fetched and -if provided by the server- the last-modified stamp.

To be done: - The current implementation does not validate the cache content, nor does it honour the HTTP cache directives.

url_cache(+URI:atom, -Path:atom, -MimeType:atom) is det

Return the content of URI in a file at Path. MimeType is the Mime-type returned by the server.

Errors: - existence_error(url, URL) Server did not respond with 200 OK; - existence_error(source_sink, url_cache(.)) Cache directory does not exist
bug: - Does not check modification time and cache validity

url_cache_delete(+URL) is det

Delete an URL from the cache. Succeeds, even if the cache files do not exist.

Errors: - Throws exceptions from delete_file/1 other than existence errors.

url_cache_dir(-Dir) is det[private]

Return or create the URL caching directory

make_directory_path(+Dir) is det[private]

Create Dir and all required components.

fetch_url(+URL:atom, +Path:atom, -MimeType:atom) is det[private]

Errors: - existence_error(url, URL)

url_cache_reset_server_status is det

url_cache_reset_server_status(+Server) is det

Reset the status of the given server or all servers.

fetch_url_raw(+URL:atom, +Path:atom, -MimeType:atom, -Modified) is det[private]

Fetch data from URL and put it into the file Path. MimeType is unified with the MIME-type as reported by the server or text/plain if the server did not provide a MIME-Type.

Errors: - existence_error(url, URL)

ssl_verify(+SSL, +ProblemCert, +AllCerts, +FirstCert, +Error)

Currently we accept all certificates.

url_cache_file(+URL, +Dir, +Ext, -Path) is det

Determine location of cache-file for the given URL in Dir. If Ext is provided, the returned Path is ensured to have the specified extension.

url_to_file(+URL, +Ext, -File) is det[private]

File is a filename for storing URL and has extension Ext. We use a cryptographic hash to ensure consistent naming, a name that is guaranteed to fit in every sensible filesystem and ensure a good distribution of the cache directories.

url_cached(?URL, ?Property) is nondet

url_cached(+Dir, ?URL, ?Property) is nondet

True if URL is in the cache represented by the directory Dir and has Property. Defined properties are:

file(-File): File is the cache-file for the given URL
mime_type(-Mime): Mime is the mime-type of the URL as reported by the server
fetched(-Stamp:integer): Timestamp that specifies when the URL was fetched
last_modified(-Modified:atom): If present, this is the modification time as provided by the server.

Re-exported predicates

The following predicates are exported from this file while their implementation is defined in imported modules or non-module files loaded by this module.