|1Proxy Services||
|^ A proxy server acts as an intermediary between Web clients and Web servers.
It listens for requests from the clients and forwards these to remote servers.
The proxy server then receives the responses from the servers and returns them
to the clients. Why go to this trouble? There are several reasons, the most
common being:
|bullet|
|item| To allow internal clients access to the Internet from behind a firewall.
Browsers behind the firewall have full Web access via the proxy system.
|item| To provide controlled access to internal resources for external clients.
The proxy server provides a managed gateway through a firewall into an
organisation's Web resources.
|item| Many proxy servers provide caching, or local storage, of responses. For
frequent or commonly accessed resources this can not only significantly reduce
apparent network latency but also greatly reduce the total traffic downloaded
by a site.
|item| For anonymity. Although often related directly to firewall security
considerations, it can also sometimes be an advantage to just not reveal the
exact source of Web transactions from within your local network.
|!bullet|
|0Proxy Serving Quick-Start||
|^ No additional software needs to be installed to provide proxy serving.
|^ Proxy servering is essentially configured using a combination of
configuration directives in WASD_CONFIG_GLOBAL and WASD_CONFIG_SERVICE to
enable proxy serving both globally and then for allow a specific service to
make outgoing connections, along with mapping directives in WASD_CONFIG_MAP
to control and direct those outgoing connections.
|^ The following steps provide a brief outline of proxy configuration.
|number|
|item| Enable proxy serving and specify which particular services
are to be proxies (|link|Enabling A Proxy Service| and
|link%|../config/##Service Configuration++of++WASD Configuration||)
|item| If proxy caching is required (most probably, see |link|Proxy Caching||)
|bullet|
|& Decide on a cache device, create the cache root directory, modify
server startup procedures to include the WASD_CACHE_ROOT logical name
(|link|Proxy Cache Device||).
|& Enable caching on required services (|link|Proxy Enabling Caching||).
|& Adjust relevant cache management configuration parameters if required
(|link|Proxy Cache Management||).
|& If required adjust cache retention parameter (|link|Proxy Cache Retention||).
|!bullet|
|item| If providing SSL tunneling (proxy of Secure Sockets Layer transactions)
add/modify a service for that (|link|CONNECT Serving||).
|item| Add WASD_CONFIG_MAP mapping rules for controlling this/these services
(|link|Controlling Proxy serving||, |link|Controlling CONNECT Serving||, and
|link|FTP Proxy Serving||).
|item| Restart server (HTTPD/DO=RESTART).
|!number|
|9Proxy Error Messages|
|0Error Messages|
|^ When proxy processing is enabled and WASD_CONFIG_GLOBAL directive
[ReportBasicOnly] is disabled it is necessary to make adjustments to the
contents of the WASD_CONFIG_MSG message configuration file [status] item
beginning "Additional Information". Each of the "/httpd/-/status|/nxx||.html"
links
|code|
1xx
2xx
3xx
4xx
5xx
help
|!code|
should be changed to include a local host component
|code|
1xx
2xx
3xx
4xx
5xx
help
|!code|
|^ If this is not provided the links and any error report will be interpreted
by the browser as relative to the server the proxy was attempting to request
from and the error explanation will not be accessible.
|2HTTP Proxy Serving|
|^ WASD provides a proxy service for the HTTP scheme (prototcol).
|^ Proxy serving generally relies on DNS resolution of the requested host name.
DNS lookup can introduce significant latency to transactions. To help
ameliorate this WASD incorporates a host name cache. To ensure cache
consistency the contents are regularly flushed, after which host names must use
DNS lookup again, refreshing the information in the cache. The period of this
cache purge is contolled with the [ProxyHostCachePurgeHours] configuration
parameter.
|^ When a request is made by a proxy server is is common for it to add a line
to the request header stating that it is a forwarded request and the agent
doing the forwarding. With WASD proxying this line would look something like
this:
|code|
Forwarded: by http://host.name.domain (HTTPd-WASD/8.4.0 OpenVMS/IA64 SSL)
|!code|
It is enabled using the [ProxyForwarded] configuration parameter.
|^ An additional, and perhaps more widely used facility, is the Squid extension
field to the proxied request header supplying the originating client host name
or IP address.
|code|
X-Forwarded-For: client.host.name
|!code|
It is enabled using the [ProxyXForwardedFor] configuration parameter.
|3Enabling A Proxy Service|
|^ Proxy serving is enabled on a global basis using the WASD_CONFIG_GLOBAL file
[ProxyServing] configuration parameter. After that each virtual service must
have proxy functionality enabled as a per-service configuration.
|^ WASD can configure services using the WASD_CONFIG_GLOBAL [service]
directive, the WASD_CONFIG_SERVICE configuration file, or even the /SERVICE=
qualifier.
|0WASD_CONFIG_SERVICE||
|^ Using directives listed in
|link%|../config/##Service Configuration++of++WASD Configuration||)
this example illustrates configuring a non-proxy server (the
|/disabled| is the default and essentially redudant) and a proxy
service.
|code|
[[http://alpha.example.com:80]]
[ServiceProxy] disabled
[[http://alpha.example.com:8080]]
[ServiceProxy] enabled
|!code|
|3Proxy Affinity|
|^ High performance/highly available proxy server configurations require more
than one instance configured and running. Whether this is done by running
multiple instances on the same host or one instance on multiple hosts, it leads
to situations where successive requests will be processed by different
instances. As those instances don't share a common name to IP address cache,
they will eventually use different IP addresses when trying to connect to an
origin server running on multiple hosts.
|^ This may result in the following, user visible, issues:
|bullet|
|& multiple requests for authentication (one from each origin host)
|& loss of icons, images, javascripts, CSS because requests for these files,
although they return a 401 status, will not trigger a browser authentication
dialog
|& loss of context and performance issues where scripts/environments need to
be started on a new host (php, python, webware,...)
|!bullet|
|^ For these reasons, the proxy server will make every effort to relay
successive requests from a given client to the same origin host as long as this
one is available (built-in failover capability will ultimately trigger the
choice of a new host). This is known as client to origin affinity or proxy
affinity capability.
|^ Proxy to origin server affinity is enabled using the following service
configuration directive.
|code|
[[http://alpha.example.com:8080]]
[ServiceProxy] enabled
[ServiceProxyAffinity] enabled
|!code|
|0Uses HTTP Cookies||
|^ Obviously the use of cookies must be enabled in the browser or this facility
will not operate for that client. After the first successful connection to an
origin host, the proxy server will send a cookie indicating the IP address used
to the client browser. Upon subsequent requests, this cookie will be used to
select the same host. The cookie is named
|/WasdProxyAffinity_origin.host.name| and the value simply the IP address in
dotted decimal. This cookie is not propagated beyond the proxy service but may
be WATCHed by checking the |/Proxy Processing| item.
|3Proxy Bind|
|^ It is possible to make the outgoing request appear to originate from a
particular source address. The Network Interface must be able to bind to the
specified IP address (i.e. it cannot be an arbitrary address).
|code|
[[http://alpha.example.com:8080]]
[ServiceProxy] enabled
[ServiceProxyBind] 131.185.250.1
|!code|
|^ The same behaviour may be accomplished with an WASD_CONFIG_MAP mapping rule.
|code|
SET http://*.example.com proxy=bind=131.185.250.1
|!code|
|3Proxy Chaining|
|^ Some sites may already be firewalled and have corporate proxy servers
providing Internet access. It is quite possible to use WASD proxying in this
environment, where the WASD server makes the proxied requests via the next
proxy server in the hierarchy. This is known as |/proxy chaining||.
|code|
[[http://alpha.example.com:8080]]
[ServiceProxy] enabled
[ServiceProxyChain] next.proxy.host
|!code|
|^ Chaining may also be controlled on a virtual service or path basis using an
WASD_CONFIG_MAP mapping rule.
|code|
SET http://*.com proxy=chain=next.proxy.host:8080
|!code|
|0Chain Authorization||
|^ If the upstream proxy server requires authorization this may be supplied
using a per-service directive
|code|
[[http://alpha.example.com:8080]]
[ServiceProxy] enabled
[ServiceProxyChain] next.proxy.host
[ServiceProxyChainCred] basic:|/:||
|!code|
or via mapping rule
|code|
SET http://*.com proxy=chain=next.proxy.host:8080 \\
proxy=chain=cred=|/basic::||
|!code|
|^ The |/basic:| keyword allows WASD to appropriately encode the credentials.
Basic authentication is the only scheme currently supported.
|3Controlling Proxy Serving|
|^ Requests at a service enabled for proxy processing are directed to proxy
processing using a fundamental rule which terminates rule processing and
initiates the outgoing connection.
|code|
pass * http://
|!code|
This rule and variant equivalents for FTP and CONNECT processing, and
in combination with other rules to purpose, are seen in the examples in this
section on proxy.
|^ Controlling both access-to and access-via proxy serving is possible.
|0Proxy Password||
|^ Access to the proxy service can be directly controlled through the use of
WASD authorization. Proxy authorization is distinct from general access
authorization. It uses specific |/proxy authorization| fields provided by
HTTP, and by this allows a proxied transaction to also supply transaction
authorization for the remote server. In the WASD_CONFIG_SERVICE configuration
file.
|code|
[[http://alpha.example.com:8080]]
[ServiceProxy] enabled
[ServiceProxyAuth] proxy
|!code|
|^ In addition to the service being specified as requiring authorization it is
also necessary to configure the source of the authentication. This is done
using the WASD_CONFIG_AUTH configuration file. The following example shows all
requests for the proxy virtual service must be authorized (GET and well as
POST, etc.), although it is possible to restrict access to only read (GET),
preventing data being sent out via the server.
|code|
[[alpha.example.com:8080]]
["Proxy Access"=PROXY_ACCESS=id]
http://* read+write
|!code|
|0Chain Password||
|^ An up-stream, chained proxy server (|link|Proxy Chaining||) may be permitted
to receive proxy authentication from the client via a WASD proxy server using
the |=CHAIN| keyword. Unconfigured, WASD does not propagate HTTP |/proxy
authorization| fields. Only one proxy server in a chain can be authenticated
against.
|code|
[[http://alpha.example.com:8080]]
[ServiceProxy] enabled
[ServiceProxyAuth] chain
|!code|
|0Local Password||
|^ It is also possible to control proxy access via local authorization,
although this is less flexible by removing the ability to then pass
authorization information to the remote service. In other repects it is set up
in the same way as proxy authorization, but enabled using the |=LOCAL| keyword.
|code|
[[http://alpha.example.com:8080]]
[ServiceProxy] enabled
[ServiceProxyAuth] local
|!code|
|0Access Filtering||
|^ Extensive control of how, by whom and what a proxy service is used for may
be exercised using WASD general and conditional mapping
|link%|../config/##Request Processing Configuration++of++WASD Configuration||)
and
|link%|../config/##Conditional Mapping++of++WASD Configuration||)
possibly in the context of a virtual service specification for the particular
connect service host and port (see
|link%|../config/##Virtual Servers++of++WASD Configuration||).
The following examples provide a small indication of how mapping could be used
in a proxy service context.
|number|
|item| It is possible, though more often not practical, to regulate which hosts
are connected to via the proxy service. For example, the following rule
forbids accessing any site with the string "hacker" in it (for the proxy
service "alpha|...|:8080".
|code|
[[alpha.example.com:8080]]
pass http://*hacker*/* "403 Proxy access to this host is forbidden."
pass http://*
|!code|
|item| Or as in the following example, only allow access to specific sites.
|code|
[[alpha.example.com:8080]]
pass http://*.org/*
pass http://*.digital.com/*
pass http://* "403 Proxy access to this host is forbidden."
|!code|
|item| It is also possible to restrict access via the proxy service to selected
hosts on the internal subnet. Here only a range of literal addresses plus a
single host in another subnet are allowed access to the service.
|code|
[[alpha.example.com:8080]]
pass http://* "403 Restricted access." ![ho:131.185.250.* ho:131.185.200.10]
pass http://*
|!code|
|item| In the following example POSTing to a particular proxied servers is not
allowed (why I can't imagine, but hey, this is an example!)
|code|
[[alpha.example.com:8080]]
pass http://subscribe.sexy.com/* "403 POSTing not allowed." [me:POST]
pass http://*
|!code|
|item| It is possible to redirect proxied requests to other sites.
|code|
[[alpha.example.com:8080]]
redirect http://www.sexy.com/* http://www.disney.com/
pass http://*
|!code|
|item| A proxy service is just a specialized capability of a general HTTP
service. Therefore it is quite in order for the one service to respond to
standard HTTP requests as well as proxy-format HTTP requests. To enforce the
use of a particular service as proxy-only, add a final rule to a virtual
service's mapping restricting non-proxy requests.
|code|
[[alpha.example.com:8080]]
pass http://*
pass /* "403 This is a proxy-only service."
|!code|
|item| This example provides the essentials when supporting |/reverse
proxying||. Note that mappings may become quite complex when supporting access
to resources across multiple internal systems (e.g. access to directory icons).
|code|
[[main.corporate.server.com:80]]
pass /sales/* http://sales.corporate.server.com/*
pass /shipping/* http://shipping.corporate.server.com/*
pass /support/* http://support.corporate.server.com/*
pass * "403 Nothing to access here!"
|!code|
|!number|
|note|
To expedite proxy mapping is it recommended to have a final rule for the proxy
virtual service that explicitly |/pass||es the request. This would most
commonly be a permissive pass as in example 1, could quite easily be an
restrictive pass as in example 2, or a combination as in example 6.
|!note|
|0Request Modification||
|^ Using path mapping rules (see
|link%|../config/##Request Processing Configuration++of++WASD Configuration||).
it is possible to remove or specifically set selected proxied request headers.
Many headers are critical to server processing but some are informational or
otherwise amenable to change. This can be undertaken using the SET mapping
rule |/proxy=header=||.
|^ For example, to have a proxy service suppress the "Referer:" request header:
|code|
# WASD_CONFIG_MAP
set * proxy=header=referer
|!code|
|^ To modify the "Referer:" request header to a fixed URL:
|code|
set * proxy=header=referer=https://whatever/
|!code|
|^ To modify the "User-Agent:" request header to a specific string:
|code|
set * "proxy=header=user-agent=None of your business!"
|!code|
|9Proxy Caching|
|2Caching|
|^ Caching involves using the local file-system for storage of responses that
can be reused when a request for the same URL is made. The WASD server does
not have to be configured for caching, it will provide proxied access without
any caching taking place.
|^ When a proxied request is processed, and the characteristics would allow
the response to be cached, a unique identifier generated from the URL is used
to create a corresponding file name. The response header and any body are
stored in this file. This may be the data of an HTML page, a graphic, etc.
|^ When a proxied request is being processed, and the characteristics
would allow the request to be cached, the unique identifier generated allows
for a previously created cache file to be checked for. If it exists, and is
current enough, the response is returned from it, instead of from the remote
server. If it exists and is no longer current the request is re-made to the
remote server, and the response if still cacheable is re-cached, keeping the
contents current. If it does not exist the response is delivered from the
remote server.
|0Not all responses can be cached!|
|^ The main critera are for the response to be successful (200 status),
general (i.e. one not in response to a specialized query or action), and not
too volatile (i.e. the same page may be expected to be returned more than once,
preferably over an extended period).
|bullet|
|item| Proxied |/requests| can only be cached if |...|
|bullet#|
|& uses the GET method
|& does not contain a query string
|& is HTTP/1.n compliant (i.e. not HTTP/0.9)
|& does not contain an "Authorization:" header field
|!bullet|
|item| Proxied |/success responses| will only be cached if |...|
|bullet#|
|& is HTTP/1.n compliant (i.e. not HTTP/0.9)
|& HTTP status code
200 (success),
203 (non-authoritative),
300 (multiple choice),
301 (moved permanently),
410 (gone)
|& contains a |/Last-Modified:| header field
|& one or more hours since the last modification
|& any |/Expires:| date/time is still in the future
|& does not contain restrictive cache control
|^- "Pragma: no-cache" field (HTTP/1.0)
|^- "Cache-Control: no-cache, no-store, private" (/1.1)
|& any "Vary:" header field does not contain a "*" or "accept[-...]""
|& does not exceed a configuration parameter in size
|!bullet|
|item| Proxied |/negative responses| will be cached if |...|
|bullet#|
|& [ProxyCacheNegativeSeconds] is non-zero
|& status code
204 (no content),
305 (use proxy),
400 (bad request),
403 (forbidden),
404 (not found),
405 (method not allowed),
414 (request URI too large),
500 (internal server error),
501 (not implemented),
502 (bad gateway),
503 (service unavailable),
504 (gateway timeout),
|& does not contain restrictive cache control
|^- "Pragma: no-cache" field (HTTP/1.0)
|^- "Cache-Control: no-cache, no-store, private" (/1.1)
|!bullet|
|!list|
|^ The [ProxyCacheFileKbytesMax] configuration parameter controls the maximum
size of a response before it will not be cached. This can be determined from
any "Content-Length:" response header field, in which case it will proactively
not be cached, or if during cache load the maximum size of the file increases
beyond the specified limit the load is aborted.
|0Not all sites may benefit from cache!|
|^ As many transactions on today's Web contain query strings, etc., and
therefore cannot be meaningfully cached, it should not be assumed the
cost/benefit of having a proxy cache enabled is a forgone conclusion. Each
site should monitor the proxy traffic reports and decide on a local policy.
|^ The facilities described in |link|Reporting and Maintenance| allow
a reasonably informed decision to be made. Items to be considered.
|bullet#|
|& The ratio of cache reads to network accesses.
|& The number of non-cacheable requests and responses, particularly as a
percentage of total proxy traffic.
|& The ratio of network to cache traffic, although this may be skewed by
having a high ratio of 304 (not-modified) responses from cache (which contain
few bytes). Check the cache 304 reporting item.
|!bullet|
|^ Last, but by no means least, understanding the characteristics of local
usage. For example, are there a small number of requests generating lots of
non-cacheable traffic? For instance, a few users accessing streaming content.
|9Proxy Cache Device|
|3Cache Device|
|^ Selection of a disk device for supporting the proxy cache should not be
made without careful consideration, doubly so if significant traffic is
experienced. Here are some common-sense suggestions.
|bullet#|
|& |*avoid| locating it as a subdirectory of WASD_ROOT:[000000]
|& use a disk with as little other activity as possible (both I/O and space
usage)
|& use a disk with as much free space as possible
|& use the fastest disk available
|!bullet|
|^ Initially the directory will need to be created. This can be done manually
as described below, or if using the supplied server startup procedures
(see |link%|../config/##Server Startup++of++WASD Configuration||)
it is checked for and if it does not exist is automatically created during
startup. The directory must be owned by the HTTP$SERVER account and have full
read+write+execute+delete access. It is suggested to name it [WASD_CACHE] and may
be created manually using the following command.
|code|
$ CREATE /DIR /OWN=HTTP$SERVER /PROT=(O:RWED,G,W) device:[WASD_CACHE]
|!code|
|^ It is a relatively simple matter to relocate the cache at any stage. Simply
create the required directory in the new location, modify the startup
procedures to reflect this, shut the server down completely then restart it
using the procedures (|*not| a /DO=RESTART!). The contents of the previous
location could be transfered to the new using the BACKUP utility if desired.
|0WASD_CACHE_ROOT Logical|
|^ It is required to define the logical name WASD_CACHE_ROOT if any proxy
services are specified as using cache in the server configuration. The server
will not start unless it is correctly defined. The logical should be a
|/concealed device| logical specifying the top level directory of the
cache tree. The following example shows how to define such a logical name.
|code|
$ DEFINE /SYSTEM /EXEC /TRANSLATION=CONCEALED WASD_CACHE_ROOT device:[WASD_CACHE.]
|!code|
|^ If example startup procedure is in use then it is quite straight-forward to
have the logical created during server startup (see
|link%|../config/##STARTUP.COM++of++WASD Configuration||).
|9Proxy Enabling Caching|
|3Enabling Caching|
|^ Caching may enabled on a per-service basis. This means it is possible to
have a caching proxy service and a non-caching service active on the one
server. Caching is enabled by appending the |/cache| keyword to the
particular service specification. The following example shows a non-proxy and
a caching proxy service.
|code|
[[http://alpha.example.com:80]]
[ServiceProxy] disabled
[[http://alpha.example.com:8080]]
[ServiceProxy] enabled
[ServiceProxyCache] enabled
|!code|
|^ Proxy caching may be selectively disabled for a particular site, sites or
paths within sites using the |/SET nocache| mapping rule. This rule, used to
disable caching for local requests, also disables proxy file caching for that
subset of requests. This example shows a couple of variations.
|code|
[[alpha.example.com:8080]]
# disable caching for local site's servers that respond fairly quickly
set http://*.local.domain/* nocache
# disable caching of log files
set http://*.log nocache
pass http://*
|!code|
|note|
It is also recommended to place the cache directory under some authorization
control to prevent casual browsing and access of the cache contents. Something
local, similar in intention to
|code|
[[alpha.example.com:8080]]
["WASD Admin"=WASD_ADMIN=id]
/wasd_cache_root/* ~webadmin,131.185.250.*,r+w ;
|!code|
|!note|
|9Proxy Cache Management|
|3Cache Management|
|^ As the proxy cache is implemented using the local file system, management
of the cache implies controlling the number of, and exactly which files remain
in cache. Essentially then, management means when and which to delete. The
[ProxyReportLog] configuration parameter enables the server process log
reporting of cache management activities.
|^ Cache file deletion has three variants.
|number|
|item| |*ROUTINE|
|^ This ensures files that have not been accessed within specified limits are
periodically and regularly deleted. The [ProxyCacheRoutineHourOfDay]
configuration parameter controls this activity.
|^ The ROUTINE form occurs once per day at the specified hour. The cache
files are scanned looking for those that exceed the configuration parameter
for maximum period since last access, which are then deleted (the largest
number of [ProxyCachePurgeList], as described below).
|item| |*BACKGROUND|
|^ Setting the [ProxyCacheRoutineHourOfDay] configuration parameter to 24
enables background purging.
|^ In this mode the server continuously scans through the cache files in the
same manner as for ROUTINE purging. The difference is it is not all done a
single burst once a day, pushing disk activity to the maximum. The background
purge regulates the period between each file access, pacing the scan so that
the entire cache is passed through once a day. It adjusts this pace according
the the size of the cache.
|item| |*REACTIVE|
|^ This is a remedial action, when cache device usage is reaching its
configuration limit and files need to be deleted to free up space. The
following parameters control this behaviour.
|simple#|
|& [ProxyCacheDeviceCheckMinutes]
|& [ProxyCacheDeviceMaxPercent]
|& [ProxyCacheDevicePurgePercent]
|& [ProxyCachePurgeList]
|!simple|
|^ The cache device space usage is checked at the specified interval.
|^ If the device reaches the specified percentage used a cache purge is
initiated and by deleting files until the specified reduction is attained, the
total space in use on the disk is reduced.
|^ The cache files are scanned using the [ProxyCachePurgeList] parameter
described below, working from the greatest to least number of hours in the
steps provided. At each scan files not accessed within that period are
deleted. At each few files deleted the device free space is checked as having
reached the lower purge percentage limit, at which point the scan terminates.
|^ This parameter has as its input a series of comma-separated integers
representing a series of hours since files were last accessed. In this
way the cache can be progressively reduced until percentage usage targets are
realized. Such a parameter would be specified as follows,
|code|
[ProxyCachePurgeList] 168,48,24,8,0
|!code|
meaning the purge would first delete files not accessed in the last week,
then not for the last two days, then the last twenty-four hours, then eight,
then finally all files. The largest of the specified periods (in this case
168) is also used as the limit for the ROUTINE scan and file delete.
|^ Once the target reduction percentage is reached the purge stops. During the
purge operation further cache files are not created. Even when cache files
cannot be created for any reason proxy serving still continues transparently to
the clients.
|note|
Cache files can be manually deleted at any time (from the command line) without
disturbing the proxy-caching server and without rebuilding any databases. When
deleting, the /BEFORE=|/date/time| qualifier can be used, with /CREATED being
the document's last-modified date, /REVISED being the last time it was loaded,
and /EXPIRED the last time the file was accessed (used to supply a request).
Be aware that on an active server it is quite possible some files may be locked
at time of attempted deletion.
|!note|
|!bullet|
|0From The Command-Line|
|^ If [ProxyCacheRoutineHourOfDay] is empty or non-numeric the automatic,
once-a-day routine purge of the cache by the server is disabled and it is
expected to be performed via some other mechanism, such as a periodic batch
job. This allows routine purging more or less frequently than is provided-for
by server configuration, and/or the purge activity being performed by a process
or cluster node other than that of the HTTPd server (reducing server and/or
node impact of this highly I/O intensive activity). Progress and other
messages are provided via SYS$OUTPUT, and if configured in the [Opcom|...|]
directives to the operator log and designated operator terminal as well. If a
process already has the cache locked the initiated activity aborts.
|^ The following example shows a routine purge being performed from the
command-line. This form uses the hours from [ProxyCachePurgeList].
|code|
$ HTTPD /PROXY=PURGE=ROUTINE
|!code|
|^ A variant on this allows the maximum age to be explicitly specified.
|code|
$ HTTPD /PROXY=PURGE=ROUTINE=168
|!code|
|^ Reactive purging and statistic scans may also be initiated from the command
line. For a reactive purge the first number can be the device usage percentage
(indicated by the trailing "%"), if not the configuration limit is used.
|code|
$ HTTPD /PROXY=PURGE=REACTIVE=80%,168,48,24,8,0
$ HTTPD /PROXY=CACHE=STATISTICS
|!code|
|^ Any in-progress scan of the cache (i.e. reactive or routine purges, or a
statistics scan) can be halted from the command line (and online Server
Admininistration facility).
|code|
$ HTTPD /PROXY=STOP=SCAN
|!code|
|9Proxy Cache Invalidation|
|3Cache Invalidation|
|^ For the purposes of this document, cache invalidation is defined as the
determination when a cache file's data is no longer valid and needs to be
reloaded.
|^ The method used for cache validation is deliberately quite simple in
algorithm and implementation. In this first attempt at a proxy server the
overriding criteria have been efficiency, simplicity of implementation, and
reliability. Wishing to avoid complicated revalidation using behind-the-scenes
HEAD requests the basic approach has been to just invalidate the cache item
upon exiry of a period related to the "Last-Modified:" age or upon a
|/no-cache| request, both described further below.
|bullet|
|item| If a "Pragma: no-cache" request header field is present (as is
generated by Netscape Navigator when using the |/reload| function)
then the server should completely reload the response from the remote server.
(Too often the author seems to have received incomplete responses where the
proxy server caches only part of a response and has seemed to refuse to
explicitly re-request.) OK, it's a bit more expensive but who's to say the
proxy server is right all the time! The response is still cached ... the next
request may not have the |/no-cache| parameter.
|item| When a response is cached the file |/creation| date/time is set to
the local equivalent of the "Last-Modified:" GMT date and time supplied
with the response. In this manner the file's absolute age can be determined
quickly and easily from the file header. This is used as described in
|link|Proxy Cache Retention||.
|item| When a file is cached, the |/revision| and |/expires||
date/times are set to current. The revision date/time is used when assessing
when the file was last loaded/validated/reloaded. Once a file is cached the
RMS |/expires| date/time is updated every time it is subsequently
accessed. In this way recency of usage of the item can be easily tracked,
allowing the routine and reactive purges to operate by merely checking the file
header.
|!bullet|
|^ The |/revision count| (automatically updated by VMS) tracks the
absolute number of accesses since the file was created (actually a maximum of
65535, or an unsigned short, but that should be enough for informational
purposes).
|9Proxy Cache Retention|
|3Cache Retention|
|^ The [ProxyCaheReloadList] configuration parameter is used to control when a
file being accessed is reloaded from source.
|^ This parameter supplies a series of integers representing the hours after
which an access to a cache file causes the file to be invalidated and reloaded
from the source during the proxied request. Each number in the series
represents the lower boundary of the range between it and the next number of
hours. A file with a last-loaded age falling within a range is reloaded at the
lower boundary of that particular range. The following example
|code|
[ProxyCacheReloadList] 1,2,4,8,12,24,48,96,168
|!code|
would result in a file 1.5 hours old being reloaded every hour, 3.25 hours
old every 2 hours, 7 hours old every 4 hours, etc. Here "old" means since
last (or of course first) loaded. Files not reloaded since the final integer,
in this example 168 (one week), are always reloaded.
|3Reporting and Maintenance|
|^ The HTTPDMON utility allows real-time monitoring of proxy serving activity
(|link|HTTPd Monitor||).
|^ Proxy reports and some administrative control may be exercised from the
online Server Administration facility (|link|Server Administration||). The
information reported includes:
|bullet#|
|& some proxy serving statistics
|& current cache device status
|& whether cache space is available
|& if a purge is in progress
|& the results from the last routine and reactive purges
|& the results from the last scan of the cache
|& contents of the host name/address cache
|!bullet|
|^ The following actions can be initiated from this menu. Note that three of
these relate to proxy file cache and so may take varying periods to complete,
depending on the number of files. If the cache is particularly large the
scan/purge |*may take some considerable time||.
|bullet#|
|& generate proxy cache statistics by scanning the entire cache
|& perform a routine purge
|& perform a reactive purge
|& purge the proxy host name/address cache
|!bullet|
|^ Also available from the Server Administration facility is a dialog allowing
the proxy characteristics of the |/running| server to be adjusted on an ad hoc
basis. This only affects the executing server, to make changes to permanent
configuration the WASD_CONFIG_GLOBAL configuration file must be changed.
|^ This dialog can be used to modify the device free space percentages
according to recent changes in device usage, alter the reload or purge hour
list characteristics, etc. After making these changes a routine or reactive
purge will automatically be initiated to reduce the space in use by the proxy
cache if implied by the new settings.
|3PCACHE Utility|
|^ It is often useful to be able to list the contents of the proxy cache
directory or the characteristics or contents of a particular cache file.
Cache files have a specific internal format and so require a tool capable of
dealing with this. The
|link%|/wasd_root/src/utils/pcache.c|WASD_ROOT:[SRC.UTILS]PCACHE.C||
program provides a versatile command-line utility as well as CGI(plus) script,
making cache file information accessible from a browser. It also allows cache
files to be selected by wildcard filtering on the basis of the contents of the
associated URL or response header. For detailed information on the various
command-line options and CGI query-string options see the description at the
start of the source code file.
|0Command-Line Use||
|^ Make the WASD_EXE:PCACHE.EXE executable a foreign verb. It is then possible
to
|bullet#|
|& list the basic characteristics of all/selected files in the cache directory
tree
|& list the characteristics plus the HTTP response header of a single file
|& extract the response header
|& extract the response body (text, graphic, file, etc.)
|& do all of the above while filtering on URL or response header contents,
number of hits, when last accessed, last loaded, and last modified (in hours)
|!bullet|
|0Script Use||
|^ To make the PCACHE script available to the server ensure the following line
exists in the HTTP$CONFIG configuration file in the [AddType] section.
|code|
.HTC application/x-script /cgiplus-bin/pcache WASD proxy cache file
|!code|
|^ The following rule needs to be in the WASD_CONFIG_MAP configuration file.
|code|
pass /wasd_cache_root/*
|!code|
|note|
It is also recommended to place the utility and the cache directory under some
authorization control to prevent casual browsing and access of the cache
contents. Something local, similar in intention to
|code|
[[alpha.example.com:8080]]
["WASD Admin"=WASD_ADMIN=id]
/pcache/* ~webadmin,131.185.250.*,r+w ;
/wasd_cache_root/* ~webadmin,131.185.250.*,r+w ;
|!code|
|!note|
|^ Once available the following is then possible.
|bullet|
|item| From a directory listing ("Index Of") access a cache file and be
presented with the following information:
|bullet#|
|& blocks used/allocated
|& last modification date/time of the response
|& date/time the response was (re)loaded into cache
|& date/time the cache file was last accessed
|& number of time since first created the cache file has been accessed
|& the URL the cache file represents (as a link)
|& the full response header (as received from the proxied server)
|& a series of "buttons" allowing
|bullet|
|& the cache content (response body) to be viewed (note that self-relative
embedded graphics, etc., probably will not be displayed in such documents)
|& the cache file to be VMS DUMPed
|& the cache file to be VMS ANALYZE/RMSed
|& the cache file to be VMS DELETEd
|!bullet|
|!bullet|
|^ If the configuration changes described above have been made the following
link will return such an index.
|^+ |link%|/wasd_cache_root/|
|item| Have the utility generate a form providing a convenient interface to the
various capabilities and filters available. If the configuration changes
described above have been made the following link will return this form.
|^+ |link%|/cgiplus-bin/pcache|
|item| The utility's form does not have to be used. By supplying the appropriate
query string components, either from a custom form or forms, or directly
embedded into links, profiles, listings, deletion may be generated.
|!bullet|
|note|
Cache directory trees have the potential to become heavily populated, so the
use of the script to generate listings of the cache contents could return
extremely large listing documents.
|!note|
|2CONNECT Serving|
|^ The |/connect| service provides firewall proxying for any
connection-oriented TCP/IP access. Essentially it provides the ability to
tunnel any other protocol via a Web proxy server. In the context of Web
services it is most commonly used to provide firewall-transparent access for
Secure Sockets Layer (SSL) transactions. It is a special case of the more
general tunneling provided by WASD, see |link|Tunneling Using Proxy||.
|3Enabling CONNECT Serving|
|^ As with proxy serving in general, CONNECT serving may enabled on a
per-service basis using the WASD_CONFIG_GLOBAL [service] directive, the WASD_CONFIG_SERVICE
configuration file, or even the /SERVICE= qualifier.
|^ The actual services providing the CONNECT access (i.e. the host and port)
are specified on a per-service basis. This means it is possible to have
CONNECT and non-CONNECT services deployed on the one server, as part of a
general proxy service or standalone. CONNECT proxying is enabled by appending
the |/connect| keyword to the particular service specification. The following
example shows a non-proxy and proxy services, with and without additional
connect processing enabled.
|code|
[[http://alpha.example.com:80]]
[[http://alpha.example.com:8080]]
[ServiceProxy] enabled
[[http://alpha.example.com:8081]]
[ServiceProxyTunnel] connect
[[http://alpha.example.com:8082]]
[ServiceProxy] enabled
[ServiceProxyTunnel] connect
|!code|
|3Controlling CONNECT Serving|
|^ The connect service poses a significant security dilemma when in use in a
firewalled environment. Once a CONNECT service connection has been accepted
and established it essentially acts as a relay to whatever data is passed
through it. Therefore |*any transaction whatsoever| can occur via the connect
service, which in many environments may be considered undesirable.
|^ In the context of the Web and the use of the connect service for proxying
SSL transactions it may be well considered to restrict possible connections to
the well-known SSL port, 443. This may be done using conditional directives,
as in the following example:
|code|
[[alpha.example.com:8080]]
if (request-method:CONNECT)
pass *:443
pass * "403 CONNECT only allowed to port 443."
endif
|!code|
All of the comments on the use of general and conditional mapping made in
|link|Controlling Proxy Serving| can also be applied to the connect service.
|2FTP Proxy Serving|
|^ WASD provides a proxy service for the FTP scheme (prototcol). This
provides the facility to list directories on the remote FTP server, download
and upload files.
|^ The (probable) file system of the FTP server host is determined by examining
the results of an FTP PWD command. If it returns a current working directory
specification containing a "/" then it assumes it to be Unix(-like), if ":["
then VMS, if a "\\" then DOS. (Some DOS-based FTP servers respond with a
Unix-like "/" so a second level of file-system determination is undertaken with
the first entry of the actual listing.) Anything else is unknown and reported
as such. WASD (for the obvious reason) is particularly careful to perform well
with FTP servers responding with VMS file specifications.
|^ Note that the content-type of the transfer is determined by the way the
proxy server interprets the FTP request path's "file" extension. This may or
may not correspond with what the remote system might consider the file type to
be. The default content-type for unknown file types is
"application/octet-stream" (binary). When using the |/alt| query string
parameters then for any file in a listing the icon provides an alternate
content-type. If the file link provides a text document then the icon will
provide a binary file. If the link returns a binary file then the icon will
return a file with a plain-text content-type.
|^ In addition to content-type the FTP mode in which the file transfer occurs
can be determined by either of two conditions. It the content-type is
"text/.." then the transfer mode will be ASCII (i.e. record carriage-control
adjusted between systems). If not text then the file is transfered in Image
mode (i.e. a binary, opaque octet-stream). For any given content-type this
default behaviour may be adjusted using the [AddType] directive (see
|link%|../config/##Alphabetic Listing++of++WASD Configuration||)
or the "#!+" MIME.TYPES directive (see
|link%|../config/##MIME.TYPES++of++WASD Configuration||).
|^ Rules required in WASD_CONFIG_MAP for mapping FTP proxy. This is preferably
made against the virtual service providing the FTP proxy. The service
explicitly must make the icon path used available or it must be available to
the proxy service in some other part of the mappings. Also the general
requirement for error message URLs applies to FTP proxying
(|link|Proxy Error Messages||).
|code|
[[proxy.host.name:8080]
pass http://* http://*
pass ftp://* ftp://*
pass /*/-/* /wasd_root/runtime/*/*
|!code|
|3FTP Query String Keywords|
|^ Keywords added to an FTP request query string allow the basic FTP action to
be somewhat tailored. These case-insensitive keywords can be in the form of a
query keys or query form fields and values. This allows considerable
flexibility in how they are supplied, allowing easy use from a browser URL
field or for inclusion as form fields.
|table|
|~_ |: Keyword|: Description
|~
|~#* |. alt |. Adds alternate access (complementary content-type
at the icon) for directory listings.
|~ |. ascii |. Force the file transfer type to be done as ASCII
(i.e. with carriage-control conversion between systems with different
representations).
|~ |. content |. Explicitly specify the content type for the
returned file (e.g. "content:text/plain", or
"content=image/gif").
|~ |. dos |. When generating a directory listing force the
interpretation to be DOS.
|~ |. email |. Explicitly specify the |/anonymous||
access email address (e.g. "email:daniel@wasd.vsm.com.au" or
"email=daniel@wasd.vsm.com.au").
|~ |. image |. Force the file transfer type to be done as an
opaque binary stream of octets.
|~ |. list |. Displays the actual directory plain-text listing
returned by the remote FTP server. Can be used for problem analysis.
|~ |. login |. Results in the server prompting for a username
and password pair that are then used as the login credentials on the remote FTP
server.
|~ |. octet |. Force the content-type of the file returned to be
specified as "application/octet-stream".
|~ |. text |. Force the content-type of the file returned to be
specified as "text/plain".
|~ |. unix |. When generating a directory listing force the
interpretation to be Unix.
|~ |. upload |. Causes the server to return a simple file
transfer form allowing the upload of a file from the local system to the remote
FTP server.
|~ |. vms |. When generating a directory listing force the
interpretation to be VMS.
|!table|
|3."login" Keyword|
|^ The usual mechanism for supplying the username and password for access to a
non-anonymous proxied FTP server area is to place it as part of the request
line (i.e. "ftp://username:password@the.host.name/path/"). This has the
obvious disadvantage that it's there for all and sundry to see.
|^ The "login" query string is provided to work around the more obvious
of these issues, having the authentication credentials as part of the request
URL. When this string is placed in the request query string the FTP proxy
requests the browser to prompt for authentication (i.e. returns a 401 status).
When request header authentication data is present it uses this as the remote
FTP server username and password. Hence the remote username and password never
need to appear in plain-text on screen or in server logs.
|2Gatewaying Using Proxy|
|^ WASD is fully capable of mapping non-proxy into proxy requests, with
various limitations on effectiveness considering the nature of what is being
performed.
|^ Gatewaying between request schemes (protocols)
|simple#|
|item| HTTP to HTTP (a gateway |/of sorts| - standard proxy)
|item| HTTP TO HTTP-over-SSL (non-secure to secure)
|item| HTTP to FTP
|item| HTTP-over-SSL to HTTP (secure to non-secure)
|item| HTTP-over-SSL to HTTP-over-SSL (secure to secure)
|item| HTTP-over-SSL to FTP
|!simple|
|^ and also gatewaying between IP versions
|simple#|
|item| IPv4 to IPv6
|item| IPv6 to IPv4
|!simple|
|^ All can be useful for various reasons. One example might be where a script
is required to obtain a resource from a secure server via SSL. The script can
either be made SSL-aware, sometimes a not insignificant undertaking, or it can
use standard HTTP to the proxy and have that access the required server via
SSL. Another example might be accessing an internal HTTP resource from an
external browser securely, with SSL being used from the browser to the proxy
server, which the accesses the internal HTTP resource on its behalf.
|0Request Redirect||
|^ The basic mechanism allowing this gatewaying is "internal"
redirection. The |/redirect| mapping rule (see
|link%|../config/##REDIRECT Rule++of++WASD Configuration||)
either returns the new URL to the originating client (requiring it to
reinitiate the request) or begins reprocessing the request internally
(transparently to the client). It is this latter function that is obviously
used for gatewaying.
|3Reverse Proxy|
|^ The use of WASD proxy serving as a firewall component assumes two configured
network interfaces on the system, one of which is connected to the internal
network, the other to the external network. (Firewalling could also be
accomplished using a single network interface with router blocking external
access to all but the server system.) Outgoing (internal to external) proxying
is the most common configuration, however a proxy server can also be used to
provide controlled external access to selected internal resources. This is
sometimes known as |/reverse proxy| and is a specific example of WASD's
general |/non-proxy to proxy| request redirection capability
(|link|Gatewaying Using Proxy||).
|^ In this configuration the proxy server is contacted by an external browser
with a standard HTTP request. Proxy server rules map this request onto a
proxy-request format result. For example:
|code|
redirect /sales/* /http://sales.server.com/*?
|!code|
|^ Note that the trailing question-mark is required to propagate any query
string (see
|link%|../config/##REDIRECT Rule++of++WASD Configuration||).
|^ The server recognises the result format and performs a proxy request to a
system on the internal network. Note that the mappings required could become
quite complex, but it is possible. See example 7 in
|link|Controlling Proxy Serving||.
|0Redirection Location Field||
|^ If a reverse proxied server returns a redirection response (302) containing
a "Location: |/url||" field with the host component the same reverse-proxied-to
server it can be rewritten to instead contain the proxy server host. If these
do not match the rewrite does not occur. Using the redirection example above,
the SET mapping rule |/proxy=reverse=location| specifies the path that will be
prefixed to the path component in the location field URL. Usually this would
be the same path used to map the reverse proxy redirect (in this example
"/sales/"), though could be any string (presumably detected and processed by
some other part of the mapping).
|code|
set /sales/* proxy=reverse=location=/sales/
redirect /sales/* /http://sales.server.com/*?
|!code|
This could be simplified a little by using a postfix SET rule along with the
original redirect.
|code|
redirect /sales/* /http://sales.server.com/*? proxy=reverse=location=/sales/
|!code|
|^ If the |/proxy=reverse=location=| ends in an asterisk the entire 302
location field URL is appended (rather than just the path) resulting in
something along the lines of
|code|
Location: http://proxy.server.com/sales/http://sales.server.com/path/
|!code|
which once redirected by the client can be subsequently tested for and some
action made by the proxy server according to the content (just a bell or
whistle ;-).
|0Authorization Verification||
|^ WASD can authorize reverse proxy requests locally (perhaps from the SYSUAF)
and rewrite that username into the proxied requests "Authorization: |...|"
field. The proxied-to server can then verify that the request originated from
the proxy server and extract and use that username as authenticated.
|^ This functionality is described in the
|link%|/wasd_root/src/httpd/proxyverify.c|WASD_ROOT:[SRC.HTTPD]PROXYVERIFY.C|
module.
|0proxyMUNGE Utility||
|^ This utility (CGIplus script) can be used to rewrite HTTP response
"Location:" fields, "Set-Cookie:" path and domain components and URLs in HTML
and CSS content.
|^ This functionality is described in the prologue to the code
|link%|/wasd_root/src/utils/proxymunge.c|WASD_ROOT:[SRC.UTILS]PROXYMUNGE.C|
|note|
The proxyMUNGE Utility handles all response rewriting and so when employing it
to perform reverse-proxy processing it is unnecessary to use the
|/proxy=reverse=location=| mapping rule described
in |link|Redirection Location Field||.
|!note|
|3One-Shot Proxy|
|^ This looks a little like reverse proxy, providing access to a non-local
resource via a standard (non-proxy) request. The difference allows the client
to determine which remote resource is accessed. This works quite effectively
for non-HTML resources (e.g. image, binary files, etc.) but
non-self-referential links in HTML documents will generally be inaccessible to
the client. This can provide provide scripts access to protocols they do not
support, as with HTTP to FTP, HTTP to HTTP-over-SSL, etc.
|^ Mappings appropriate to the protocols to be support must be made against
the proxy service. Of course mapping rules may also be used to control whom or
to what is connected.
|code|
[[the.proxy.service:port]]
# support "one-shot" non-proxy to proxy redirect
redirect /http://* http://*
redirect /https://* https://*
redirect /ftp://* ftp://*
# OK to process these (already, or now) proxy format requests
pass http://* http://*
pass https://* https://*
pass ftp://* ftp://*
|!code|
|^ The client may the provide the desired URL as the path of the request to
the proxy service. Notice that the scheme provided in the desired URL can be
any supported by the service and its mappings.
|code|
http://the.proxy.service:port/http://the.remote.host/path
http://the.proxy.service:port/https://the.remote.host/path
http://the.proxy.service:port/ftp://the.remote.host/pub/
|!code|
|3DNS Wildcard Proxy|
|^ This relies on being able to manipulate host record in the DNS or local name
resolution database. If a "*.the.proxy.host" DNS (CNAME) record is resolved it
allows any host name ending in ".the.proxy.host" to be resolved to the
corresponding IP address. Similarly (at least the Compaq TCP/IP Services) the
local host database allows an alias like "another.host.name.proxy.host.name"
for the proxy host name. Both of these would allow a browser to access
"another.host.name.proxy.host.name" with it resolved to the proxy service. The
request "Host:" field would contain "another.host.name.proxy.host.name".
|^ Using this approach a fully functioning proxy may be implemented for the
browser without actually configuring it for proxy access, where returned HTML
documents contain links that are always correct with reference to the host used
to request them. This allows the client an |/ad hoc| proxy for selected
requests. For a wildcard (CNAME) record the browser user may enter any host
name prepended to the proxy service host name and port and have the request
proxied to that host name. Entering the following URL into the browser
location field
|code|
http://the.host.name.the.proxy.service:8080/path
|!code|
would result in a standard HTTP proxy request for "/path" being made to
"the.host.name:80". With the URL
|code|
https://the.host.name.the.proxy.service:8443/path
|!code|
an SSL proxy request. Note that normally the well-known port would be used to
connect to (80 for http: and 443 for https:). If the final, period-separated
component of the wildcard host name is all digits it is interpreted as a
specific port to connect to. The example
|code|
http://the.host.name.8001.the.proxy.service:8080/path
|!code|
would connect to "the.host.name:8001", and
|code|
https://the.host.name.8443.the.proxy.service:8443/path
|!code|
to "the.host.name:8443".
|note|
It has been observed that some browsers insist that an all-digit host name
element is a port number despite it being prefixed by a period not a colon.
These browsers then attempt to contact the host/port directly. This obviously
precludes using an all-digit element to indicate a target port number with
these browsers.
|!note|
|^ This wildcard DNS entry approach is a more fully functional analogue to
common proxy behaviour but is slightly less flexible in providing gatewaying
between protocols and does require more care in configuration. It also relies
on the contents of the request "Host:" field to provide mapping information
(which generally is not a problem with modern browsers). The mappings must be
performed in two parts, the first to handle the wildcard DNS entry, the second
is the fairly standard rule(s) providing access for proxy processing.
|code|
[[the.proxy.service:port1]]
if (host:*.the.proxy.service:port1)
redirect * /http://*
else
pass http://* http://*
endif
|!code|
|^ The obvious difference between this and one-shot proxy is the desired host
name is provided as part of the URL host, not part of the request path. This
allows the browser to correctly resolve HTML links etc. It is less flexible
because a different proxy service needs to be provided for each protocol
mapping. Therefore, to allow HTTP to HTTP-over-SSL proxy gatewaying another
service and mapping would be required.
|code|
[[the.proxy.service:port2]]
if (host:*.the.proxy.service:port2)
redirect * /https://*
else
pass https://* https://*
endif
|!code|
|3Originating SSL|
|^ This proxy function allows standard HTTP clients to connect to Secure
Sockets Layer (|link|Transport Layer security||) services. This is very
different to the CONNECT service (|link|CONNECT Serving||), allowing scripts
and standard character-cell browsers supporting only HTTP to access secure
services.
|^ Standard username/password authentication is supported (as are all other
standard HTTP request/response interactions). The use of X.509 client
certificates (|link|Authorization Using X.509 Certification)||) to establish
outgoing identity is not currently supported.
|0Enabling SSL|
|^ Unlike HTTP and FTP proxy it requires the service to be specifically
configured using the [ServiceClientSSL] directive.
|^ There are a number of Secure Sockets Layer related service parameters that
should also be considered (see
|link%|../config/##Service Configuration++of++WASD Configuration||).
Although most have workable defaults unless [ServiceProxyClientSSLverifyCA] and
[ServiceProxyClientSSLverifyCAfile] are specifically set the outgoing
connection will be established without any checking of the remote server's
certificate. This means the host's secure service could be considered unworthy
of trust as the credentials have not been established.
|code|
[[http://alpha.example.com:8080]]
[ServiceProxy] enabled
[ServiceClientSSL] enabled
|!code|
|2Tunneling Using Proxy|
|^ WASD supports the CONNECT method which effectively allows tunneling of
raw octets through the proxy server. This facility is most commonly used to
allow secure SSL connections to be established with hosts on the 'other side'
of the proxy server. This basic mechanism is also used by WASD to provide an
extended range of tunneling services. The term |/raw| is used here
to indicate an 8 bit, bidirectional, asynchronous exchange of octets between
two entities, as a protocol family, not necessarily as an application (but can
be so). Global proxy serving must be enabled (|link|Enabling a Proxy
Service||) and then each service must be configured and mapped according to the
desired mode of tunneling. Disabling or setting timeouts appropriately on the
mapped service is important if connections are not to be disrupted by general
server timeouts on output and non-progress (quiescent connections).
|3.[ServiceProxyTunnel] CONNECT|
|^ A service with this configuration is used as a target for CONNECT proxying
(usually SSL through a firewall). The client expects an HTTP success (200)
response once the remote connection is established, and HTTP error response if
there is a problem, and once established just relays RAW octets through the
proxy server (classic CONNECT behaviour).
|code|
# WASD_CONFIG_SERVICE
[[http://*:8080]]
[ServiceProxy] enabled
[ServiceProxyTunnel] connect
|!code|
|code|
# WASD_CONFIG_MAP
[[*:8080]]
if (request-method:connect)
pass *:443 *:443
pass * "403 CONNECT only allowed to port 443."
endif
|!code|
|^ This configuration enables CONNECT processing and limits any connect to SSL
tunneling (i.e. port 443 on the remote system).
|3.[ServiceProxyTunnel] RAW|
|^ This allows any raw octet client (e.g. telnet) to connect to the port and
by mapping be tunnelled to another host and port to connect to its service
(e.g. a telnet service). The usual HTTP responses associated with CONNECT
processing are not provided.
|code|
# WASD_CONFIG_SERVICE
[[http://*:10023]]
[ServiceProxy] enabled
[ServiceProxyTunnel] raw
|!code|
|code|
# WASD_CONFIG_MAP
[[*:10023]]
if (request-method:connect)
pass *:0 raw://another.host:23 timeout=none,none,none
endif
pass "403"
|!code|
|^ Telnet is used in the example above but the principle equally applies to
any protocol that uses a raw 8 bit, bidirectional, asynchronous exchange of
octets. Another example might be an SMTP service (port 25).
|0SSL to RAW|
|^ Using a tunnel it is possible to put a TLS/SSL (https://) front-end service
to an otherwise plaintext-only service (http://).
|code|
# WASD_CONFIG_SERVICE
[[https://tls-host:443]]
[ServiceNonSSLRedirect] https://tls.host:443
[ServiceProxy] enabled
[ServiceProxyTunnel] raw
|!code|
|code|
# WASD_CONFIG_MAP
[[*:443]]
if (request-method:connect)
pass *:0 raw://non-tls.host:80
endif
pass "403"
|!code|
|0Chaining RAW|
|^ It is possible to have a raw tunnel establish itself through a proxy chain
(|link|Proxy Chaining||) by transparently generating an intermediate CONNECT
request to the up-stream proxy server. Note that not all CONNECT proxy will
allow connection to just any specified port. For security reasons it it is
quite common to restrict CONNECT to port 443.
|code|
# WASD_CONFIG_SERVICE
[[http://*:10025]]
[ServiceProxy] enabled
[ServiceProxyTunnel] raw
|!code|
|code|
# WASD_CONFIG_MAP
[[*:10025]]
if (request-method:connect)
pass *:0 raw://another.host:25 proxy=chain=proxy.host:8080
endif
pass "403"
|!code|
|^ Any error in connecting to the chained proxy, making the request,
connecting to the destination, etc. (i.e. any error at all) is not reported.
The network connection is just dropped. Use WATCH to establish the cause if
necessary.
|3.[ServiceProxyTunnel] FIREWALL|
|^ With this configuration a service expects that the first line of text from
the client contains a host name (or IP address) and optional port (e.g.
"the.host.name" or "the.host.name:23"). This allows a variable destination to
be mapped. The usual HTTP responses associated with CONNECT processing are not
provided.
|code|
# WASD_CONFIG_SERVICE
[[http://*:10023]]
[ServiceProxy] enabled
[ServiceProxyTunnel] FIREWALL
|!code|
|code|
# WASD_CONFIG_MAP
[[*:10023]]
if (request-method:connect)
pass *:* raw://*:23 timeout=none,none,none
pass * raw://*:23 timeout=none,none,none
endif
pass "403"
|!code|
|^ The pass rules force the supplied domain name (and optional port) to be
mapped to the telnet port (23). Of course the mapping rules could allow the
supplied port to be mapped into the destination if desired.
|0Chaining FIREWALL|
|^ As with [ServiceProxyTunnel] RAW it is possible to chain FIREWALL services
to an up-stream proxy server. See |link|Chaining RAW||.
|3Encrypted Tunnel|
|^ Up to this point the tunnels have merely been through the proxy server. It
is possible to establish and maintain ENCRYPTED TUNNELS between WASD servers.
SSL is used for this purpose. This is slightly more complex as both ends of
the tunnel need to be configured.
|draw|
+------------+ +------------+
<--unencrypted-->| WASD proxy |<--ENCRYPTED-->| WASD proxy |<--unencrypted-->
+------------+ +------------+
|!draw|
|^ This arrangement may be used for any stream-oriented, network protocol
between two WASD systems. As it uses standard CONNECT requests (over SSL) it
MAY also be possible to be configured between WASD and non-WASD servers.
|^ The following example is going to maintain an encrypted tunnel between WASD
servers running on systems KLAATU and GORT. It is designed to allow a user on
KLAATU to connect to a specified port using a telnet client, and have a telnet
session created on GORT, tunnelled between the two systems via an SSL encrypted
connection.
|^ Source of tunnel:
|code|
# KLAATU WASD_CONFIG_SERVICE
[[http://*:10023]]
[ServiceProxy] enabled
[ServiceClientSSL] ENABLED
[ServiceProxyTunnel] RAW
|!code|
|code|
# KLAATU WASD_CONFIG_MAP
[[*:10023]]
# if the client is on the local subnet
if (remote-addr:192.168.0.0/24 && request-method:connect)
pass *:0 https://gort.domain:10443 timeout=none,none,none
endif
pass "403"
|!code|
|^ Destination of tunnel:
|code|
# GORT WASD_CONFIG_SERVICE
[[https://*:10443]]
[ServiceProxy] enabled
[ServiceProxyTunnel] CONNECT
|!code|
|code|
# GORT WASD_CONFIG_MAP
[[*:10443]]
# limit the connection to a specific host
if (remote-addr:192.168.0.10 && request-method:connect)
pass *:0 raw://gort.domain:23 timeout=none,none,none
endif
pass "403"
|!code|
|^ When a client connects to the service provided by port 10023 on system
KLAATU the connection is immediately processed using a pseudo CONNECT request
header. The service on this port is a proxy allowed to initiate SSL
connections (client SSL). This service is mapped to system GORT port 10443, an
SSL service that allows the CONNECT method (tunneling). KLAATU's proxy
initiates an SSL connection with GORT. When established and the CONNECT
request from KLAATU is received, it is mapped via a raw tunnel (8 bit, etc.) to
its own system port 23 (the telnet service). Telnet is in use at both ends
while encrypted by SSL inbetween! Note the use of network addresses and
general fail rules used to control access to this service, as well as the
disabling of timers that might otherwise shutdown the tunnel.
|3Encrypted Tunnel With Authentication|
|^ This arrangement is essentially a variation on example 4. It provides a
cryptographic authentication of the originator (source) of the tunnel.
|^ Source of tunnel:
|code|
# KLAATU WASD_CONFIG_SERVICE
[[http://*:10023]]
[ServiceProxy] enabled
[ServiceClientSSL] enabled
[ServiceProxyTunnel] RAW
[ServiceClientSSLcert] WASD_ROOT:[LOCAL]HTTPD.PEM
|!code|
|code|
# KLAATU WASD_CONFIG_MAP
[[*:10023]]
# if the client is on the local subnet
if (remote-addr:192.168.0.0/24 && request-method:connect)
pass *:0 https://gort.domain:10443 timeout=none,none,none
endif
pass "403"
|!code|
|^ Destination of tunnel:
|code|
# GORT WASD_CONFIG_SERVICE
[[https://*:10443]]
[ServiceProxy] enabled
[ServiceProxyTunnel] CONNECT
[ServiceProxyAuth] PROXY
|!code|
|code|
# GORT WASD_CONFIG_MAP
[[*:10443]]
# we'll be relying on X509 authentication
if (request-method:connect)
pass *:0 raw://gort.domain:23 timeout=none,none,none
endif
pass "403"
|!code|
|code|
# GORT WASD_CONFIG_AUTH
[[*:10443]]
[X509]
* r+w,param="[VF:OPTIONAL]",~4EAB3CBC735F8C7977EBB41D45737E37
|!code|
|^ This works by configuring the destination service to insist on proxy
authorization. The authorization realm is X509 which causes the destination to
demand a certificate from the source (|link|Authorization Using X.509
Certification||). The fingerprint of this certificate is checked against the
authorization rule before the connection is a allowed to procede.
|3Shared SSH Tunnel|
|^ The objective of this |/raw| tunnel variant (see
|link|[ServiceProxyTunnel] RAW||) is to allow tunneling
of Secure Shell (SSH) via a client site proxy server CONNECT which is usually
confined to port 443. Of course most Web servers are configured to provide SSL
HTTP on port 443. Sharing of HTTP and SSH on the same port is a little
problematic and involves some protocol detection. The following explanation of
how it is implemented is so that the reader can understand the requirement for
the "timeout quirk".
|^ On configured services; WASD |/peeks| at the incoming TCP byte stream to
see if it's SSH protocol. If it is, the socket is associated with a proxy raw
tunneling service and proxy tunneling initiated to a mapped SSH server. However
(just to make it interesting) some SSH clients do not initiate their own
exchange until after the SSH server, and so |/peeking| only works for a subset
of clients. Of course this is a Catch-22 of sorts! To provide for these
clients; if an input timeout should occur (an SSH client waiting) WASD sets up
the tunnel anyway and begins the proxy. The proxied SSH server should then
initiate the protocol and the client respond. The directive [ServiceShareSSH]
configured to be non-zero both enables this facility for a service and sets the
input timeout period (which perhaps should be shorter than the default 30
seconds because such clients will wait that long for any SSH server response).
|^ This approach seems to work well-enough in practice, although users need to
be aware that some clients will pause (for the duration of the timeout period
|-| the "timeout quirk") during initial connection setup.
|code|
# WASD_CONFIG_SERVICE
[[https://*:443]
[ServiceShareSSH] 10
[[http://*:10022]]
[ServiceProxy] enabled
[ServiceProxyTunnel] raw
|!code|
|code|
# WASD_CONFIG_MAP
[[*:443]
if (request-method:ssh)
pass * raw://ssh.server.host:22 \\
service=the.proxy.host:10022 \\
timeout=none,none,none
endif
[[*:10022]]
pass "403"
|!code|
|^ This example shows an SSL service, the desired SSH service (which can be
local or remote) and the internal proxy service that will provide the
connection.
|3Complex Private Tunneling|
|^ When creating |/raw| tunnels between WASD servers, and possibly in other
circumstances, it is often useful to be able to signal |/tunnel purpose| to the
remote end. In this way a single destination port can support multiple
tunneling purposes simply through mapping rules. An originating end can
|/inject| an HTTP request line, or full request, into the established tunnel
connection, which can then be processed by the usual WASD request mapping, and
from that alternate services provided based on the intent signalled by the
originating end.
|^ This somewhat complex but instructive example illustrates the potential
utility and versatility of WASD tunneling. It involves an originating WASD
server, a destination (service providing) WASD server, and just to make it
interesting an intermediate chained HTTP proxy server (not WASD). The idea is
to provide access to various application services not necessarily supported by
intermediate HTTP proxies and/or gateways. Four services will be supported
by the example; SSH, NNTP IMAP and SMTP.
|draw|
inside firewall outside
+------------+ +-------------+ +------------+
<--raw-->| WASD proxy |<--ENCRYPTED-->| other proxy |<--ENCRYPTED-->| WASD proxy |<--raw-->
+------------+ +-------------+ +------------+
{wasd.internal.net} {proxy.internal.net} {wasd.external.net}
{proxy.external.net}
SSH---8022--+ : : +----22---SSH
SMTP---8025--+----------------------------+....+-----------------------------+----25---SMTP
NNTP---8119--+ :....: +---119---NNTP
IMAP---8143--+ : : +---143---IMAP
|!draw|
|0Internal Services|
|^ These are the services assigned on the WASD server on the inside of the
proxy/gateway. Note that there is one per application to be tunneled. For
simplicity each service port number has been selected to parallel the
well-known application port number. Note that |/proxy| is enabled on each
(allowing them to initiate outgoing connections) and each has |/SSL| enabled
(further allowing them to initiate encrypted connections).
|code|
# client SSH
[[http://*:8022]]
[ServiceProxy] enabled
[ServiceProxyTunnel] RAW
[ServiceClientSSL] enabled
# client SMTP
[[http://*:8025]]
[ServiceProxy] enabled
[ServiceProxyTunnel] RAW
[ServiceClientSSL] enabled
# client IMAP
[[http://*:8143]]
[ServiceProxy] enabled
[ServiceProxyTunnel] RAW
[ServiceClientSSL] enabled
# client NNTP
[[http://*:8119]]
[ServiceProxy] enabled
[ServiceProxyTunnel] RAW
[ServiceClientSSL] enabled
|!code|
|^ Each client application (i.e. IMAP, SSH) must be configured to connect to
its corresponding service port (e.g. IMAP to 8143, SMTP to 8025).
|0Internal Mapping|
|^ These mappings are made on the WASD server on the inside of the
proxy/gateway. The rules essentially initiate an outgoing encrypted (SSL)
connection to the host |/wasd.external.net| supporting the external WASD proxy
server. Each is also configured not to connect directly but to request the
chained proxy server |/proxy.internal.net| to establish the connection on their
behalf.
|code|
!##### SSH #####
[[*:8022]]
pass * https://wasd.external.net:443 notimeout \\
proxy=tunnel=request="CONNECT wasd-ssh" \\
proxy=chain=proxy.internal.net:8080
!##### SMTP #####
[[*:8025]]
pass * https://wasd.external.net:443 \\
proxy=tunnel=request="CONNECT external-smtp" \\
proxy=chain=proxy.internal.net:8080
!##### NNTP #####
[[*:8119]]
pass * https://wasd.external.net:443 \\
proxy=tunnel=request="CONNECT external-nntp" \\
proxy=chain=proxy.internal.net:8080
!##### IMAP #####
[[*:8143]]
pass * https://wasd.external.net:443 \\
proxy=tunnel=request="CONNECT external-imap" \\
proxy=chain=proxy.internal.net:8080
|!code|
|^ If the up-stream proxy server successfully connects to |/wasd.external.net|
port 443 the proxy server allows the byte-stream to be asynchonously and
bidirectionally exchanged with the internal WASD server outgoing connection.
This internal WASD server has initiated an SSL connection and the external
server port 443 expects SSL so they can now both negotiate an SSL-encrypted
channel essentially directly with each other.
|0External Services|
|^ The external WASD service configuration is very simple, a single SSL port.
|code|
# general SSL service
[[https://wasd.external.net:443]]
# outgoing proxy/tunnel service
[[http://wasd.external.net:1234]]
[ServiceProxy] enabled
[ServiceProxyTunnel] raw
[ServiceClientSSL] ENABLED
|!code|
|^ Connections to the 443 port are expected to undertake an SSL negotiation to
establish an encrypted channel. This includes incoming tunnel connections.
The service on port 1234 is required to support the connections outgoing from
the external WASD server to the application server ports.
|0External Mapping|
|^ These mappings are all applied to requests at port 443 on the external WASD
server |/wasd.external.net||. Each rule checks three request characterstics.
First, the request method, "CONNECT". Second, the request URI, varies
according to the request. These are the request data injected by the internal
WASD server |/wasd.internal.net| using the |/set=proxy=tunnel=request=| mapping
rule on the outgoing connection. Third, the originating host
(|/proxy.external.net||) address adds an extra filter on from where this
facility may be used. The respective |/pass| of the matching rule then
initiates an outgoing connection to the respective application server's
well-known port. A timeout is applied to limit connection times.
|code|
!# SSH tunneling
[[*:443]]
if (request-method:CONNECT && \\
request-uri:"wasd-ssh" && \\
remote-addr:205.3.*) \\
pass * raw://wasd.external.net:22 service=*:1234 timeout=noprogress=00:00:50
!# SMTP tunneling
[[*:443]]
if (request-method:CONNECT && \\
request-uri:"external-smtp" && \\
remote-addr:205.3.*) \\
pass * raw://smtp.isp.net:25 service=*:1234 timeout=noprogress=00:00:50
!# NNTP tunneling
[[*:443]]
if (request-method:CONNECT && \\
request-uri:"external-nntp" && \\
remote-addr:205.3.*) \\
pass * raw://news.isp.net:119 service=*:1234 timeout=noprogress=00:00:*
!# IMAP tunneling
[[*:443]]
if (request-method:CONNECT && \\
request-uri:"external-imap" && \\
remote-addr:205.3.*) \\
pass * raw://imap.isp.net:143 service=*:1234 timeout=noprogress=00:00:50
!# disable general 1234 service usage
[[*:1234]]
pass * 403 "Internal use only!"
|!code|
|0Example In Action||
|^ Now let's look at an actual example usage. Consider the internal user's
IMAP application, say Thunderbird, is configured to use an IMAP server at host
|/wasd.internal.net| port 8143. The internal user activates Thunderbird which
then intiates an TCP/IP connection to the configured IMAP server expecting to
commence the IMAP application protocol.
|^ This connection arrives at |/wasd.internal.net| port 8143 which has a WASD
|/raw| tunnel service listening. The connection is accepted and request
processing commences. Mapping rules applied to port 8143 initiate an SSL
connection to host |/wasd.external.net| which is not directly accessable
because of the firewall and must be connected to using the HTTP proxy server
|/proxy.internal.net| as an intermediary. This is specified in the same
mapping rule. The mapping rule also injects an HTTP request header providing
request characteristics that can be identified and acted upon by the external
server.
|^ The internal WASD server initiates a connection to the proxy server
|/proxy.internal.net| acting as part of the firewall. As it is endeavouring
to initiate an SSL connection with the external |/wasd.external.net| host this
proxy connection uses a CONNECT request specifying |/wasd.external.net| port
443. The proxy server establishes a connection with the host
|/wasd.external.net| at port 443. Once the connection is established it
becomes an asynchronous, bidirectional channel between |/wasd.internal.net| and
|/wasd.external.net| with the proxy server as a conduit.
|^ The service connection just established is expecting an SSL negotiation in
an attempt to establish an encrypted channel. When this negotiation concludes
successfully the communications between |/wasd.internal.net| and
|/wasd.external.net| become opaque to all external listeners including
|/proxy.internal.net||.
|^ The encrypted connection now established, the request begins to be processed
by the WASD server at |/wasd.external.net||. A number of mapping rules apply
to port 443. Each rule compares the injected request method and URI until, in
this case, the |/external-imap| rule matches. This rule specifies that a raw
connection be established with the host |/imap.isp.net| at port 143 using the
proxy-capable port 1234 service. A timeout limits the duration this connection
can be held unused.
|^ The IMAP application server at |/imap.isp.external| port 143 accepts the
connection at begins to communicate using the IMAP protocol.
|^ There is now a raw (8 bit, asynchronous, bidirectional) connection from the
Thunderbird client to |/wasd.internal.net||, (encrypted) through to
|/proxy.internal.net||, (encrypted) through to |/wasd.external.net||, and raw
to the IMAP server at |/imap.isp.net||. This raw connection will be used for
communication between Thunderbird and the IMAP server using the IMAP
application protocol.
|3Tunnelling Source|
|^ When a tunnel is established into a system the source of that connection (IP
host-name/address and port) becomes obscured. By setting the path to the
destination port |/proxy=forwarded=for| (host name) or
|/proxy=forwarded=address| (IP address) the external client can be obtained
using data contained in the logical name WASD_TUNNEL.
|^ Consider tunneling external port 22345 to internal port 22 - Secure Shell.
|code|
# WASD_CONFIG_SERVICE
[[http://*:22345]]
[ServiceProxy] enabled
[ServiceProxyTunnel] RAW
# WASD_CONFIG_MAP
[[*:22345]]
pass * raw://localhost:22 notimeout
|!code|
|^ To Secure Shell the source host and port would be |/localhost| and |/some
random port||. It can be useful for the login procedure or other service to
have the actual client host name (or IP address). Adding the path setting.
|code|
# WASD_CONFIG_MAP
[[*:22345]]
pass * raw://localhost:22 notimeout proxy=forwarded=address
|!code|
will result in connection data becoming available in the multivalued
logical name WASD_TUNNEL. Index 0 contains internal data, and then the rest
(1..127) contain one tunneled connection's details each, in the format
|code|
|/||=|/||=|/||
|!code|
For example
|code|
localhost:46851=www.external.net:22345=mydotcom.org:49201
|!code|
|^ Obtaining the SSH source port, say from TT_ACCPORNAM data, the original
client host and port can be searched for with some trivial DCL code. Adapt to
suit local requirements.
|code|
$ if P1 .eqs. "" then P1 = f$element(1,":",f$getdvi("TT:","TT_ACCPORNAM"))
$ value = ""
$ local = ""
$ service = ""
$ client = ""
$ index = 1
$ index_loop:
$ value = f$trnlnm("WASD_TUNNEL","WASD_TABLE",index)
$ if value .eqs. "" then goto end_index_loop
$ local = f$element(0,"=",value)
$ addr = f$element(0,":",local)
$ port = f$element(1,":",local)
$ if port .eqs. P1
$ then
$ service = f$element(1,"=",value)
$ client = f$element(2,"=",value)
$ goto end_index_loop
$ endif
$ index = index + 1
$ goto index_loop
$ end_index_loop:
$ if f$trnlnm("TT_CLIENT","LNM$PROCESS") .nes. "" -
then deassign /process TT_CLIENT
$ if client .nes. "" then define /process TT_CLIENT "''client'"
|!code|
|^ The tunnel data remains current for at least one minute and may become
unavailable at any time after that.
|note|
The source data only reflects the client that connects to that system's services
and so cannot be used across multiple, back-to-back tunnels.
|!note|
|2Browser Proxy Configuration|
|^ The browser needs to be configured to access URLs via the proxy server.
This is done using two basic approaches, manual and automatic.
|3Manual|
|^ Most browsers allow the configuration for access via a proxy server. This
commonly consists of an entry for each of the common Web protocol schemes
("http:", "ftp:", "gopher:", etc.). Supply the configured WASD proxy service
host name and port for the HTTP scheme. This is currently the only one
available. This would be similar to the following example:
|code|
http: www.example.com 8080
|!code|
|^ To exclude local hosts, and other servers that do not require proxy access,
there is usually a field that allows a list of hosts and/or domain names for
which the browser should not use proxy access. This might be something like:
|code|
www.example.com,example.com,example.com
|!code|
|3Automatic|
|^ A proxy auto-config (PAC) file defines how web browsers and other user
agents can automatically choose the appropriate proxy server (access method)
for fetching a given URL.
|^+ |%https://en.wikipedia.org/wiki/Proxy_auto-config|
|^ The following is a very simple proxy configuration JavaScript function.
This specifies that all URL host names that aren't full qualified, or that are
in the "example.com" domain will be connected to directly, with all other being
accessed via the specified proxy server.
|code|
function FindProxyForURL(url,host)
{
if (isPlainHostName(host) \|\|
dnsDomainIs(host, ".example.com"))
return "DIRECT";
else
return "PROXY www.example.com:8080; DIRECT";
}
|!code|
|^ This JavaScript is contained in a file with a specific, associated MIME file
type, "application/x-ns-proxy-autoconfig". For WASD it is recommended the file
be placed in WASD_ROOT:[LOCAL] and have a file extension of
.PAC (which follows Netscape naming convention).
|^ The following WASD_CONFIG_GLOBAL directive would map the file extension to
the required MIME type:
|code|
[AddType]
.PAC application/x-ns-proxy-autoconfig - proxy autoconfig
|!code|
|^ This file is commonly made the default document available from the proxy
service. The following example shows the HTTP$MAP rules required to do this:
|code|
[www.example.com:8080]
pass http://* http://*
pass / /wasd_root/local/proxy.pac
pass *
|!code|
|^ All that remains is to provide the browser with the location from which load
this |/automatic proxy configuration| file. In the case of the above set-up
this would be:
|code|
http://www.example.com:8080/
|!code|
|^ A template for a proxy auto-configuration file may be found at
|link%|/wasd_root/example/proxy_autoconfig.txt|\
WASD_ROOT:[EXAMPLE]PROXY_AUTOCONFIG.TXT|