For version 11.5 release of WASD VMS Web Services.
Published July 2020
Document generated using wasDOC version 2.0.0
This document describes the more significant features and facilities available with the WASD Web Services package.
For installation and update details see WASD Web Services - Installation
For detailed configuration information see WASD Web Services - Configuration
For information on CGI, CGIplus, ISAPI, OSU, etc., scripting, see WASD Web Services - Scripting
And for a description of WASD Web document, SSI and directory listing behaviours and options, WASD Web Services - Environment
WASD VMS Web Services – Copyright © 1996-2020 Mark G. Daniel
Licensed under the GNU Public License, Version 3;
This package is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
https://www.gnu.org/licenses/gpl.txt
You should have received a copy of the GNU General Public License along with this package; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
Mark.Daniel@wasd.vsm.com.au
A pox on the houses of all spamers. Make that two poxes.
All copyright and trademarks within this document belong to their rightful owners. See 15. Attribution and Acknowledgement.
This is a static (file), single document.
Alternative multi-part static
and dynamic documents.
Links followed by ⤤ open in a new page.
| 1.1Troubleshooting? |
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |
With the installation, update and detailed configuration of the WASD Web Services package provided in WASD Web Services - Install and Config why have an introduction in this subsequent document? After getting the basics up and running (often the first thing we want to do) it's time to stop and consider the tool and what we're trying to accomplish with it. So this section provides an overview of the package's design philosophy, history and significant features and capabilities by topic.
The document assumes a basic understanding of Web technologies and uses terms without explaining them (e.g. HTTP, HTML, URL, CGI, SSI, etc.) The reader is refered to documents specifically on these topics.
WASD Web Services originated from a 1993 decision by Wide Area Surveillance Division (WASD) management (then High Frequency Radar Division, HFRD) to make as much information as possible, both administrative and research, available online to a burgeoning personal desktop workstation and PC environment (to use the current term … an intranet) using the then emerging Web technologies.
It then became the objective of this author to make all of our systems' VMS-related resources available via HTTP and HTML, regardless of the underlying data or storage format. An examination of the WASD package will show that this objective is substantially achieved.
Reasons for developing (remember; back in 1994!) a local HTTP server were few but compelling:
When initially installing or configuring WASD, and sometimes later where something breaks spectacularly, it is most useful to be able to gain insight into what the server is up to.
The go-to tool is WATCH (yes, all capitals, and for no other reason than it makes it stand out).
WATCH is described in detail in 10. WATCH Facility of this document.
For most circumstances WATCH can be made available for troubleshooting even if the configuration is significantly broken. This is done by using a skeleton-key to authorise special access into the server.
The skeleton-key is described in detail in 3.12 Skeleton-Key Authentication, also in this document.
TL;DR
Enable at the command-line with the username anything beginning with an underscore and at least 8 characters, same for the password length.
Then using a browser access any available service, entering the above username (including underscore) and password when prompted.
The service administration facilities (of which WATCH is one) are also available and useful.
| 2.1Server Behaviour |
| 2.2VMS Versions |
| 2.3TCP/IP Packages |
| 2.4International Features |
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |
The most fundamental component of the WASD VMS Web Services environment is the HTTP server (HyperText Transport Protocol Daemon, or HTTPd). WASD has a single-process, multi-threaded, asynchronous I/O design.
The following bullet-points summarise the features and facilities, many of which are described in significant detail in following chapters.
The technical aspects of server design and behaviour are described in WASD_ROOT:[SRC.HTTPD]READMORE.TXT
The WASD server is supported on any VMS version from V7.0 upwards, on Alpha, Itanium and VAX architectures. The current version (as of 2019), V8.4 Alpha and Itanium, as is commonly the case on VMS platforms, required nothing more than relinking. Obviously no guarantees can be made for yet-to-be-released versions but at a worst-case these should only require the same.
Up until v10.1 WASD was supported on VMS V6.0 and later. Eventually it had to be dragged kicking and screaming into the mid-1990s!
The WASD distribution and package organisation fully supports mixed-architecture clusters (Alpha, Itanium and/or VAX in the one cluster) as one integrated installation.
The WASD server uses the TCP/IP Services (UCX) BG $QIO interface. The following packages support this interface and may be used.
To deploy IPv6 services this package must support IPv6.
WASD provides a number of features that assist in the support of non-English and multi-language sites. These "international" features only apply to the server, not necessarily to any scripts!
A directory may contain language-specific variants of a basic document. When requesting the basic document name these variants are automatically and transparently provided as the response if one matches preferences expresses in the request's "Accept-Language:" request header field. Both text and non-text documents (e.g. images) may be provided using this mechanism.
Configuration information is provided in section Language Variants of WASD Configuration.
Generally the default character set for documents on the Web is ISO-8859-1 (Latin-1). The server allows the specification of any character set as a default for text document responses (plain and HTML). In addition, text document file types may be modified or additional ones specified that have a different character set associated with that type. Furthermore, specific character sets may be associated with mapping paths. A site can therefore relatively easily support multiple character set document resources.
In addition the server may be configured to dynamically convert one character set to another during request processing. This is supported using the VMS standard NCS character set conversion library.
For further information see [CharsetDefault], [CharsetConvert] and [AddType] in Alphabetic Listing of WASD Configuration.
The server uses an administrator-customizable database of messages that can contain multiple language instances of some or all messages, using the Latin-1 character set (ISO8859-1). Although the base English messages can be completely changed and/or translated to provide any message text required or desired, a more convenient approach is to supplement this base set with a language-specific one.
One language is designated the prefered language. This would most commonly be the language appropriate to the geographical location and/or clientele of the server. Another language is designated the base language. This must have a complete set of messages and is a fall-back for any messages not configured for the additional language. Of course this base language would most commonly be the original English version.
More than just two languages can be supported. If the browser has prefered languages set the server will attempt to match a message with a language in this preference list. If not, then the server-prefered and then the base language message would be issued, in that order. In this way it would be possible to simultaneously provide for English, French, German and Swedish audiences, just for example.
For message configuration information see Message Configuration of WASD Configuration.
Dates appearing in server-generated, non-administrative content (e.g. directory listings, not META-tags, which use Web-standard time formats) will use the natural language specified by any SYS$LANGUAGE environment in use on the system or specifically created for the server.
Virtual-server-associated mapping, authorization and character-sets allow for easy multiple language and environment sites. Further per-request tailoring may be deployed using conditional rule mapping described below. Single server can support multi-homed (host name) and multiple port services.
For virtual services information see Configuration Considerations of WASD Configuration.
Mapping rules map requested URL paths to physical or other paths (see Request Processing Configuration of WASD Configuration). Conditional rules are only applied if the request matches criteria such as prefered language, host address (hence geographical location to a certain extent), etc. This allows requests for generic documents (e.g. home pages) to be mapped to language versions appropriate to the above criteria.
For conditional mapping information see Conditional Configuration of WASD Configuration.
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |
Authentication is the verification of a user's identity, usually through username/password credentials. Authorization is allowing a certain action to be applied to a particular path based on authentication of the originator.
Generally, authorization is a two step process. First authentication, using a username/password database. Second authorization, determining what the username is allowed to do for this transaction.
Basic authorization was discussed in Authorization Configuration (Basics) of WASD Configuration. This section discusses all the aspects of WASD authentication and authorization.
By default, the logical name WASD_CONFIG_AUTH locates a common authorization rule file. Simple editing of the file and reloading into the running server changes the processing rules.
Server authorization is performed using a configuration file, authentication source, and optional full-access and read-only authorization grouping sources, and is based on per-path directives. There is no user-configured authorization necessary, or possible! In the configuration file paths are associated with the authentication and authorization environments, and so become subject to the HTTPd authorization mechanism. Reiterating … WASD HTTPd authorization administration involves those two aspects, setting authorization against paths and administering the authentication and authorization sources.
Authorization is applied to the request path (i.e. the path in the URL used by the client). Sometimes it is possible to access the same resource using different paths. Where this can occur care must be exercised to authorize all possible paths.
Where a request will result in script activation, authorization is performed on both script and path components. First script access is checked for any authorization, then the path component is independently authorized. Either may result in an authorization challenge/failure. This behaviour can be disabled using a path SETting rule, see SET Rule of WASD Configuration.
The authentication source name is refered to as the realm, and refers to a collection of usernames and passwords. It can be the system's SYSUAF database.
The authorization source is refered to as the group, and commonly refers to a collection of usernames and associated permissions.
The configuration file rules are scanned from first towards last, until a matching rule is encountered (or end-of-file). Generally a rule has a trailing wildcard to indicate that all sub-paths are subject to the same authorization requirements.
Rule matching is string pattern matching, comparing the request specified path, and optionally other components of the request when using configuration conditionals Conditional Configuration of WASD Configuration, to a series of patterns, until one of the patterns matches, at which stage the authorization characteristics are applied to the request and authentication processing is undertaken. If a matching pattern (rule) is not found the path is considered not to be subject to authorization. Both wildcard and regular expression based pattern matching is available String Matching of WASD Configuration.
A policy regarding when and how authorization can be used may be established on a per-server basis. This can restrict authentication challenges to "https:" (SSL) requests (4. Transport Layer Security), thereby ensuring that the authorization environment is not compromised by use in non-encrypted transactions. Two server qualifiers provide this.
Note also that individual paths may be restricted to SSL requests using either the mapping conditional rule configuration or the authorization configuration files. See Conditional Mapping of WASD Configuration.
In addition, the following configuration parameters have a direct role in an established authorization policy.
Details of authentication failures are logged to the server process log.
Failures may also be directed to the OPCOM facility OPCOM Logging of WASD Configuration.
Both paths and usernames have permissions associated with them. A path may be specified as read-only, read and write, write-only (yes, I'm sure someone will want this!), or none (permission to do nothing). A username may be specified as read capable, read and write capable, or only write capable. For each transaction these two are combined to determine the maximum level of access allowed. The allowed action is the logical AND of the path and username permissions.
The permissions may be described using the HTTP method names, or using the more concise abbreviations R, W, and R+W.
| Path/User | DELETE | GET | HEAD | POST | PROPFIND | PUT | WebDAV |
|---|---|---|---|---|---|---|---|
| READ or R | no | yes | yes | no | yes | no | no |
| WRITE or W | yes | no | no | yes | no | yes | yes |
| R+W | yes | yes | yes | yes | yes | yes | yes |
| NONE | no | no | no | no | no | no | no |
| DELETE | yes | yes | no | no | no | no | no |
| GET | no | yes | no | no | no | no | no |
| HEAD | no | no | yes | no | no | no | no |
| POST | no | no | no | yes | no | no | no |
| PROPFIND | no | no | no | no | yes | no | no |
| PUT | no | yes | no | no | no | yes | no |
| Other WebDAV | no | no | no | no | no | no | yes |
Requiring a particular path to be authorized in the HTTP transaction is accomplished by applying authorization requirements against that path in a configuration file. This is an activity distinct from setting up and maintaining any authentication/authorization databases required for the environment.
By default, the system-table logical name WASD_CONFIG_AUTH locates a common authorization configuration file, unless an individual rule file is specified using a job-table logical name. Simple editing of the file changes the configuration. Comment lines may be included by prefixing them with the hash "#" character, and lines continued by placing the backslash character "\" as the last character on a line.
The [IncludeFile] is a directive common to all WASD configuration, allowing a separate file to be included as a part of the current configuration. (see Include File Directive of WASD Configuration.
Configuration directives begin either with a "[realm]", "[realm;group]" or "[realm;group-r+w;group-r]" specification, with the forward-slash of a path specification, or with a "[AuthProxy]" or "[AuthProxyFile]" introducing a proxy mapping. Following the path specification are HTTP method keywords controlling group and world permissions to the path, and any access-restricting request scheme ("https:") and/or host address(es) and/or username(s).
Square brackets are used to enclose a [realm;group;group] specification, introducing a new authentication grouping. Within these brackets is specified the realm name (authentication source), and then optional group (authorization source) names separated by semi-colons. All path specifications following this are authenticated against the specified realm database, and permissions obtained from the group "[realm;group]" database (or authentication database if group not specified), until the next [realm;group;group] specification.
The following shows the format of an authentication source (realm) only directive.
This one, the format of a directive using both authentication and authorization sources (both realm and group).
The third variation, using an authentication, full-access (read and write) and read-only authorization sources (realm and two grouping).
The authentication source may also be given a description. This is the text the browser dialog presents during password prompting. See ‘Realm Description’ in 3.5 Authentication Sources.
Paths are usually specified terminated with an asterisk wildcard. This implies that any directory tree below this is included in the access control. Wildcards may be used to match any portion of the specified path, or not at all. Following the path specification are control keywords representing the HTTP methods or permissions that can be applied against the path, and optional access-restricting list of host address(es) and/or username(s), separated using commas. Access control is against either or both the group and the world. The group access is specified first followed by a semi-colon separated world specification. The following show the format of the path directive, see the examples below to further clarify the format.
The [AuthProxy] and [AuthProxyFile] directives introduces one or more SYSUAF proxy mappings (3.10.5 VMS Account Proxying).
The same path cannot be specified against two different realms for the same virtual service. The reason lies in the HTTP authentication schema, which allows for only one realm in an authentication dialog. How would the server decide which realm to use in the authentication challenge? Of course, different parts of a given tree may have different authorizations, however any tree ending in an asterisk results in the entire sub-tree being controlled by the specified authorization environment, unless a separate specification exists for some inferior portion of the tree.
There is a thirty-one character limit on authentication source names.
The following realm names are reserved and have special functionality.
By default accounts with SYSPRV authorized are always rejected to discourage the use of potentially significant usernames (e.g. SYSTEM). Accounts that are disusered, have passwords that have expired, or that are captive or restricted are also automatically rejected.
The authentication source may be disguised by giving it a specific description. This will the text the browser dialog presents during password prompting. See ‘Realm Description’ in 3.5 Authentication Sources.
See 3.10 SYSUAF-Authenticated Users for further information on these topics.
The following username is reserved.
If a host name, protocol identifier or username is included in the path configuration directive it acts to further limit access to matching clients (path and username permissions still apply). If more than one are included a request must match each. If multiple host names and/or usernames are included the client must match at least one of each. Host and username strings may contains the asterisk wildcard, matching one or more consecutive characters. This is most useful when restricting access to all hosts within a given domain, etc. In addition a VMS security profile may be associated with the request.
These examples show the use of a network mask to restrict based on the source network of the client. The first, four octets supplied as a mask. The second a VLSM used to specify the length of the network component of the address.
These examples both specify a 6 bit subnet. With the above examples the host 131.185.250.250 would be accepted, but 131.185.250.50 would be rejected.
Note that it more efficient to place protocol and host restrictions at the front of a list.
Authentication credentials may be validated against one of several sources, each with different characteristics.
An identifier is indicated by appending a "=ID" to the name of the realm or group. Also refer to 3.10.3 Rights Identifiers.
Whether or not any particular username is allowed to authenticate via the SYSUAF may be controlled by that account holding or not holding a particular rights identifier. Placing "=ID" against realm name implies the username must exist in the SYSUAF and hold the specified identifier name.
When (and only when) a username has been authenticated via the SYSUAF, rights identifiers associated with that account may be used to control the level-of-access within that realm. This is in addition to any identifier controlling authentication itself.
In this example a username would need to hold the PROJECT_A identifier to be able to authenticate, PROJECT_A_LIBRARIAN to write the path(s) (via POST, PUT) and PROJECT_A_USER to be able to read the path(s).
The server system SYSUAF may be used to authenticate usernames using the VMS account name and password. The realm being VMS may be indicated by using the name "VMS", by appending "=VMS" to another name making it a VMS synonym, or by giving it a specific description ( in ). Further information on SYSUAF authentication may be found in . These examples illustrate the general idea.
Three Authentication and Credential Management Extension (ACME) agents are currently available (as at VMS V8.3 and WASD v9.3), "VMS" (SYSUAF), "MSV1_0" (Microsoft domain authentication used by Advanced Server) and an LDAP kit. There is also an API that will allow local or third-party agents to be developed. WASD ACME authentication is completely asynchronous and so agents that make network or other relatively latent queries will not add granularity into server processing. By default ACME is used to authenticate requests against the SYSUAF on Alpha and Itanium running VMS V7.3 or later (3.10.1 ACME).
For authorization rules explicitly specifying ACME the Domain Of Interpretation (DOI) becomes the realm name, interposed between the relam description and the ACME authentication source keyword. In this first example the DOI is VMS and so all WASD SYSUAF authentication capabilities are available.
In the second example authentication is performed using the same credentials as Advanced Server running on the local system.
In this final example the DOI is a third-party agent.
A plain-text list may be used to provide usernames for group membership. The format is one username per line, at the start of the line, with optional, white-space delimited text continuing along the line (which could be used as documentation). Blank lines and comment lines are ignored. A line may be continued by ending it with a "\" character. These files may, of course, be created and maintained using any plain text editor. They must exist in the WASD_AUTH: directory, have an extension of ".$HTL", and do not need to be world accessible.
Simple lists are indicated in the configuration by appending a "=LIST" to the name.
It also possible to use a simple list for authentication purposes. The plain-text password is appended to the username with a trailing equate symbol. Although in general this is not recommended as everything is stored as plain-text it may be suitable as an ad hoc solution in some circumstances. The following example shows the format.
These are binary, fixed 512 byte record files, containing authentication and authorization information. HTA databases may be used for authentication and group membership purposes. The content is much the same, the role differs according to the location in the realm directive. These databases may be administered using the online Server Administration facility (9.5 HTTPd Server Revise) or the HTAdmin command-line utility (13.9 HTAdmin). They are located in the WASD_AUTH: directory and have an extension of ".$HTA".
(Essentially for historical reasons) HTA databases are the default sources for authorization information. Therefore, using just a name, with no trailing "=something", will configure an HTA source. Also, and recommended for clearly showing the intention, appending the "=HTA" qualifier specifies an HTA database. The following example show some of the variations.
Uses X.509 v3 certificates (browser client certificates) to establish identity (authentication) and based on that identity control access to server resources (authorization). This is only available for SSL transactions. See 4. Transport Layer Security for further information on SSL, and 4.5.12 Authorization Using X.509 Certification on X509 realm authorization.
From RFC1413 (M. St.Johns, 1993) …
The Identification Protocol is not intended as an authorization or access control protocol. At best, it provides some additional auditing information with respect to TCP connections. At worst, it can provide misleading, incorrect, or maliciously incorrect information.
Nevertheless, RFC1413 may be useful for some purposes in some heterogeneous environments, and so has been made available for authentication purposes.
The RFC1413 realm generates no browser username/password dialog. It relies on the system supporting the client to return a reliable identification of the user accessing the HTTP server by looking-up the user of the server connection's peer port.
An authorization agent is a CGI-compliant CGIplus script that is specially activated during the authorization processing. Using CGI environment variables it gets details of the request, makes an assessment based on its own internal authentication/authorization processing, and using the script callout mechanism returns the results to the server, which then acting on these, allows or denies access.
Such agents allow a site to develop local authentication/authorization mechnisms relatively easily, based on CGI principles. A discussion of such a development is not within the scope of this section, see the WASD Web Services - Scripting document for information on the use of callouts, and the example and working authorization agents provided in the WASD_ROOT:[SRC.AGENT] directory. The description at the beginning of these programs covers these topics in some detail.
An authorization agent would be configured using something like the following, where the "AUTHAGENT" is the actual script name doing the authorization. This has the the path "/cgiauth-bin/" prepended to it.
It is possible to supply additional, per-path information to an agent. This can be any free-form text (up to a maximum length of 63 characters). This might be a configuration file location, as used in the example CEL authenticator. For example
Generally authorization agent scripts use 401/WWW-Authorize: transactions to establish identity and credentials. It is possible for an agent to establish identity outside of this using mechanisms available only to itself. In this case it is necessary suppress the usually automatic generation of username/password dialogs using a realm of agent+opaque
An older mechanism required a leading parameter of "/NO401". It is included here only for reference. The agent+opaque realm should now always be used.
It is necessary to have the following entry in the WASD_CONFIG_MAP configuration file:
This allows authentication scripts to be located outside of the general server tree if desired.
A token is a short-lived, cookie delivered, representation of authentication established in another context. Originally devised to allow controlled access to very large datasets without the overhead of SSL in the transmission but with access credentials supplied in the privacy of an SSL connection. The cookie contains NO CREDENTIAL data at all and the authenticator manages an internal database of these so it can determine whether any supplied token is valid and when that token has expired. By default (and commonly) token authorisation occurs in non-SSL space (http:) and the credential authorisation in SSL space (https:).
Token authorisation is described in 3.11 Token Authentication).
Instead of a list of usernames contained in a database, a group within a realm (either or both full-access-source or read-only-source, see 3.4 Authorization Configuration File) may be specified as a host, group of hosts or network mask. This acts to restrict all requests from clients not matching the IP address specification. Unlike the per-path access restrict list (‘Access Restriction Keywords’ in 3.4 Authorization Configuration File) this construct applies to all paths in the realm. It also offers relative efficiencies over restriction lists and lends itself to some environments based on per-host identification (e.g. the RFC1413 realm). Note that IP addresses can be spoofed (impersonated) so this form of access control should be deployed with some caution.
The examples of realm specifications above all act to restrict read-write access via the RFC1413 realm to hosts within the 131.185.250.nnn subnet.
Generally the WASD model is for the server to perform authorisation processing and so the password never becomes visible at the application level. For scripting environments performing their own authentication the server will decode and parse the request "Authorization:" header for paths under the EXTERNAL realm.
The various authentication data are then provided in the CGI variables
If the script is performing its own authentication and authorisation using the raw request header then the server needs to be advised of this by placing the required paths under the OPAQUE realm.
The server will then provide only the "Authorization:" header data in the cgi variable HTTP_AUTHORIZATION from which the username and password may processed.
A realm directive may contain one or more different types of authorization information source, with the following restrictions.
It is possible to supply text describing the authentication realm to the browser user that differs from the actual source name. This may be used to disguise the actual source or to provide a more informative description than the source name conveys.
Prefixing the actual realm source name with a double-quote delimited string (of up to 31 characters) and an equate symbol will result in the string being sent to a browser as the realm description during an authentication challenge. Here are some examples.
WASD authorization offers a number of combinations of access control. This is a summary. Please note that when refering to the level-of-access a particular username may be allowed (read-only or full, read-write access), that it is always moderated by the level-of-access provided with a path configured within that realm. See 3.3 Permissions, Path and User.
When a path is controlled by a realm that comprises an authentication source only, as in this example
Where a group membership source is provided following the authentication source, as illustrated in this example
When a second group is specified, as in
The second group may be specified as an asterisk wildcard ("*") which is interpreted as everyone else (i.e. everyone else gets read-only access).
As described in Virtual Services of WASD Configuration, virtual service syntax may be used with authorization mapping to selectively apply rules to one specific service. This example provides the essentials of using this syntax. Note that service-specific and service-common rules may be mixed in any order allowing common authorization environments to be shared.
The online Server Administration facility path authorization report (9.4 HTTPd Server Reports) provides a selector allowing the viewing and checking of rules showing all services or only one particular virtual server, making it simpler to see exactly what any particular service is authorizing against.
Mixed case is used in the configuration examples (and should be in configuration files) to assist in readability. Rule interpretation however is completely case-insensitive.
WASD authorization allows for very simple authorization environments and provides the scope for quite complex ones. The path authentication scheme allows for multiple, individually-maintained authentication and authorization databases that can then be administered by autonomous managers, applying to widely diverse paths, all under the ultimate control of the overall Web administrator.
Fortunately great complexity is not generally necessary.
Most sites would be expected to require only an elementary setup allowing a few selected Web information managers the ability to write to selected paths. This can best be provided with the one authentication database containing read and write permissions against each user, with and access-restriction list against individual paths.
For example. Consider a site with three departments, each of which wishes to have three representatives capable of administering the departmental Web information. Authentication is via the SYSUAF. Web administrators hold an approriate VMS rights identifier, "WEBADMIN". Department groupings are provided by three simple lists of names, including the Web administrators (whose rights identifier would not be applied if access control is via a simple list), a fourth lists those with read-only access into the Finance area. The four grouping files would look like:
The authorization configuration file then contains:
Access to authentication sources, SYSUAF, simple lists and HTA databases, are relatively expensive operations. To reduce the impact of this activity on request latency and general server performance, authentication and realm-associated permissions for each authenticated username are stored in a cache. This means that only the initial request needs to be checked from appropriate databases, subsequent ones are resolved more quickly and efficiently from cache.
Such cached entries have a finite lifetime associated with them. This ensures that authorization information associated with that user is regularly refreshed. This period, in minutes, is set using the [AuthCacheMinutes] configuration parameter. Zero disables caching with a consequent impact on performance.
Where-ever a cache is employed there arises the problem of keeping the contents current. The simple lifetime on entries in the authentication cache means they will only be checked for currency whenever it expires. Changes may have occured to the databases in the meantime.
Generally there is are other considerations when adding user access. Previously the user attempt failed (and was evaluated each time), now the user is allowed access and the result is cached.
When removing or modifying access for a user the cached contents must be taken into account. The user will continue to experience the previous level of access until the cache lifetime expires on the entry. When making such changes it is recommended to explicitly purge the authentication cache either from the command line using /DO=AUTH=PURGE (9.7 HTTPd Command Line) or via the Server Administration facility (9. Server Administration). Of course the other solution is just to disable caching, which is a less than optimal solution.
The ability to authenticate using the system's SYSUAF is controlled by the server /SYSUAF[=keyword] qualifier. By default it is disabled.
By default accounts with SYSPRV authorized are always rejected to discourage the use of potentially significant usernames (e.g. SYSTEM). This behaviour can be changed through the use of specific identifiers, see 3.10.3 Rights Identifiers immediately below. Accounts that are disusered, have passwords that have expired or that are captive or restricted are always rejected. Accounts that have access day/time restricting access will have those restrictions honoured (see 3.10.3 Rights Identifiers for a workaround for this).
Also see 3.10.6 Nil-Access VMS Accounts.
By default the Authentication and Credential Management Extension (ACME) is used to authenticate SYSUAF requests on Alpha and Itanium running VMS V7.3 or later (3.5 Authentication Sources). VAX and earlier versions of VMS use WASD's own SYSUAF authentication routines. The advantage of ACME is with the processing of the (rather complex) authentication requirements by a vendor-supplied implementation. It also allows SYSUAF password change to be made subject to the full site policy (password history, dictionary checking, etc.) which WASD does not implement.
By default SYSUAF authentication uses the NETWORK access restriction from the account SYSUAF record. Alternatives LOCAL, DIALUP and REMOTE may be specified using global configuration directive
Whether or not any particular username is allowed to authenticate via the SYSUAF may be controlled by that account holding or not holding a particular VMS rights identifier. When a username has been authenticated via the SYSUAF, rights identifiers associated with that account may be used to control the level-of-access within that realm.
Use of identifiers for these purposes are enabled using the /SYSUAF=ID server startup qualifier.
The first three reserved identifier names are optional. A warning will be reported during startup if these are not found. The fourth must exist if SYSUAF proxy mappings are used in a /SYSUAF=ID environment.
Identifiers may be managed using the following commands. If unsure of the security implications of this action consult the relevant VMS system management security documentation.
They can then be provided to desired accounts using commands similar to the following:
Be aware that, as with all successful authentications, and due to the WASD internal authentication cache, changing database contents does not immediately affect access. Any change in the RIGHTSLIST won't be reflected until the cache entry expires or it is explicitly flushed ().
Any authentication realm can have its usernames mapped into VMS usernames and the VMS username used as if it had been authenticated from the SYSUAF. This is a form of proxy access.
When a proxy mapping occurs request user authorization detail reflects the SYSUAF username characteristics, not the actual original authentication source. This includes username, user details (i.e. becomes that derived from the owner field in the SYSUAF), constraints on the username access (e.g. SSL only), and user capabilities including any profile if enabled. Authorization source detail remains unchanged, reflecting the realm, realm description and group of the original source. For CGI scripting an additional variable, WWW_AUTH_REMOTE_USER, provides the original remote username.
For each realm, and even for each path, a different collection of mappings can be applied. Proxy entries are strings containing no white space. There are three basic variations, each with an optional host or network mask component.
The "SYSUAF" is the VMS username being mapped to. The remote is the remote username (CGI variable WWW_REMOTE_USER). The first variation maps a matching remote username (and optional host/network) onto the specific SYSUAF username. The second maps all remote usernames (and optional host/network) to the one SYSUAF username (useful as a final mapping). The third maps all remote usernames (optionally on the remote host/network) into the same SYSUAF username (again useful as a final mapping if there is a one-to-one equivalence between the systems).
Proxy mappings are processed sequentially from first to last until a matching rule is encountered. If none is found authorization is denied. Match-all and default mappings can be specified.
In this example the username bloggs on system 131.185.250.1 can access as if the request had been authenticated via the SYSUAF using the username and password of FRED, although of course no SYSUAF username or password needs to be supplied. The same applies to the second mapping, doe on the remote system to JOHN on the VMS system. The third mapping disallows a system account ever being mapped to the VMS equivalent. The fourth, wildcard mapping, maps all accounts on all systems in 131.185.250.0 8 bit subnet to the same VMS username on the server system. The fifth mapping provides a default username for all other remote usernames (and used like this would terminate further mapping).
Note that multiple, space-separated proxy entries may be placed on a single line. In this case they are processed from left to right and first to last.
Proxy mapping rules should be placed after a realm specification and before any authorization path rules in that realm. In this way the mappings will apply to all rules in that realm. It is possible to change the mappings between rules. Just insert the new mappings before the (first) rule they apply to. This cancels any previous mappings and starts a new set. This is an example.
An alternative to in-line proxy mapping is to provide the mappings in one or more independent files. In-line and in-file mappings may be combined.
To cancel all mappings for following rules use an [AuthProxy] (with no following mapping detail). Previous mappings are always cancelled with the start of a new realm specification. Where proxy mapping is not enabled at the command line or a proxy file cannot be loaded at startup a proxy entry is inserted preventing all access to the path.
REMEMBER – proxy processing can be observed using the WATCH facility.
It is possible, and may be quite effective for some environments, to have a SYSUAF account or accounts strictly for HTTP authorization, with no actual interactive or other access allowed to the VMS system itself. This would relax the caution on the use of SYSUAF authentication outside of SSL transactions. An obvious use would be for the HTTP server administrator. Additional accounts could be provided for other authorization requirements, all without compromising the system's security.
In setting up such an environment it is vital to ensure the HTTPd server is started using the /SYSUAF=ID qualifier (3.2 Authentication Policy). This will require all SYSUAF-authenticated accounts to possess a specific VMS resource identifier, accounts that do not possess the identifier cannot be used for HTTP authentication. In addition the identifier WASD_NIL_ACCESS will need to be held (3.10.3 Rights Identifiers), allowing the account to authenticate despite being restricted by REMOTE and NETWORK time restrictions.
To provide such an account select a group number that is currently unused for any other purpose. Create the desired account using whatever local utility is used then activate VMS AUTHORIZE and effectively disable access to that account from all sources and grant the appropriate access identifier (see 3.10.3 Rights Identifiers above).
When SSL is in use (4. Transport Layer Security) the username/password authentication information is inherently secured via the encrypted communications of SSL. To enforce access to be via SSL add the following to the WASD_CONFIG_MAP configuration file:
Note that this mechanism is applied after any path and method assessment made by the server's authentication schema.
The qualifier /SYSUAF=SSL provides a powerful mechanism for protecting SYSUAF authentication, restricting SYSUAF authenticated transactions to the SSL environment. The combination /SYSUAF=(SSL,ID) is particularly effective.
Also see 3.2 Authentication Policy.
It is possible to control access to files and directories based on the VMS security profile of a SYSUAF-authenticated remote user. This functionality is implemented using VMS security system services involving SYSUAF and RIGHTSLIST information. The feature must be explicitly allowed using the server /PROFILE qualifier. By default it is disabled.
When a SYSUAF-authenticated user (i.e. the VMS realm) is first authenticated a VMS security-profile is created and stored in the authentication cache (3.9 Authorization Cache). A cached profile is an efficient method of implementing this as it obviously removes the need of creating a user profile each time a resource is assessed. If this profile exists in the cache it is attached to each request authenticated for that user. As it is cached for a period, any change to a user's security profile in the SYSUAF or RIGHTSLIST won't be reflected in the cached profile until the cache entry expires or it is explicitly flushed (9.6 HTTPd Server Action).
When a request has this security profile all accesses to files and directories are assessed against it. When a file or directory access is requested the security-profile is employed by a VMS security system service to assess the access. If allowed, it is provided via the SYSTEM file protection field. Hence it is possible to be eligible for access via the OWNER field but not actually be able to access it because of SYSTEM field protections! If not allowed, a "no privilege" error is generated.
Once enabled using /PROFILE it can be applied to all SYSUAF authenticated paths, but must be enabled on a per-path basis, using the WASD_CONFIG_AUTH profile keyword (‘Access Restriction Keywords’ in 3.4 Authorization Configuration File)
Of course, this functionality only provides access for the server, IT DOES NOT PROPAGATE TO ANY SCRIPT ACCESS. If scripts must have a similar ability they should implement their own scheme (which is not too difficult, see WASD_ROOT:[SRC.MISC]CHKACC.C) based on the CGI variable WWW_AUTH_REALM which would be "VMS" indicating SYSUAF-authentication, and the authenticated name in WWW_REMOTE_USER.
If the /PROFILE qualifier has enabled SYSUAF-authenticated security profiles, whenever a file or directory is assessed for access an explicit VMS security system service call is made. This call builds a security profile of the object being assessed, compares the cached user security profile and returns an indication whether access is permitted or forbidden. This is addition to any such assessments made by the file system as it is accessed.
This extra security assessment is not done for non-SYSUAF-authenticated accesses within the same server.
For file access this extra overhead is negligible but becomes more significant with directory listings ("Index of") where each file in the directory is independently assessed for access.
Much of a site's package directory tree is inaccessible to the server account. One use of the SYSUAF profile functionality is to allow authenticated accesss to all files in that tree. This can accomplished by creating a specific mapping for this purpose, subjecting that to SYSUAF authentication with /PROFILE behaviour enabled (3.10.8 SYSUAF Security Profile), and limiting the access to a SYSTEM group account. As all files in the WASD package are owned by SYSTEM the security profile used allows access to all files.
The following example shows a path with a leading dollar (to differentiate it from general access) being mapped into the package tree. The "set * noprofile" limits the application of this to the /$WASD_ROOT/ path (with the inline "profile").
This path is then subjected to SYSUAF authentication with access limited to an SSL request from a specific IP address (the site administrator's) and the SYSTEM account.
This is a niche authorisation environment for addressing niche requirements.
A token is an HTTP cookie delivered representation of authentication established in another context. Originally devised to allow controlled access to very large datasets without the overhead of SSL in the transmission but with access credentials supplied in the privacy of an SSL connection.
A common scenario is where the client starts off attempting to access a resource in non-SSL space which is controlled by token authentication. In the first instance the authenticator detects there is no access token present and redirects the client (browser) to the SSL equivalent of that space, where credentials can be supplied encrypted. In this example scenario the SSL area is controlled by WASD SYSUAF authentication (can be SSL client certificate, etc.) and the username/password is prompted for. When correctly entered this generates a token. The token is stored (with corresponding detail) as a record in a server-internal database and then returned to the browser as a set-cookie value.
With the token data stored the browser is transparently redirected back to the non-SSL space where the actual access is to be undertaken, this time the browser presenting the cookie containing the token. The authenticator examines the token, looking it up in the database. If found, has originated from the same IP address, represents the same authentication realm, and has not expired, it then allows the non-SSL space access to proceed, and in this example scenario the dataset transfer is initiated (in unencrypted clear-text). If the token is not found in the database or has expired, then the process is repeated with a redirect back into SSL space. If the realms differ a 403 forbidden response is issued (see configuration below).
The token itself is a significant sequence of pseudo-random characters, is short-lived (configurable as anything from a few seconds to a few tens of seconds, or more), and as a consequence is frequently regenerated. The token is just that, containing no actual credential data at all. It might be possible to snoop but as it contains nothing of value in itself, expires relatively quickly, and has an originating IP address check, the fairly remote risk of playback is just that.
The authenticator does all the work, implicitly redirecting the user from non-SSL space to SSL space for the original authentication, and then back again with the token used for access in the non-SSL space. With the expiry of a token it undertakes that cycle again, redirecting back to the SSL-space where the browser-cached credentials will be supplied automatically allowing the fresh token to be issued, and then redirected back into non-SSL space for access. To emphasise - all this is transparent to the user.
As a consequence of this model the resource being controlled can ONLY be accessed from non-SSL space using the controlled path. To access the same resource from SSL space a distinct path to the resource must be provided.
As token authorisation relies on the client agent having HTTP cookies enabled (globally or specifically for the site) it is useful to have this tested for and/or advised about, on some related but other area of the site. There are simple techniques using JavaScript for detecting the availability of cookie processing. Search the Web for a suitable solution.
The automatic authorisation and redirection occurs using a combination of two distinguishable authorisation rules, one for supplying the credentials, the other for using the token for authorisation. In this example (and commonly) the resources are at "/location/" and the configuration accepts user-supplied credentials in SSL space and uses the token in non-SSL space. The asterisk just indicates that in the absence of any other parameter this authorisation rule has a complementary token rule where access will actually occur.
And in this example, the same arrangement but with non-standard ports (specified using an integer with a leading colon).
To prevent potential thrashing, where multiple, distinct realms within a single request are authorised using tokens, corresponding multiple token (cookie) names must be used. It is expected that this would be an uncommon but not impossible scenario. The "thrashing" would be a result of authorisation associated with a single, particular token name. Where a realm differs from a previous token generated another is required. The token authorisation scheme forces the use of distinct token names by 403-forbidding change of realm using the one token. Use explicitly specified, independent token (cookie) names, or an integer preceded by an ampersand (which appends the integer to the base token name), ensuring the complementary rules are using the same name/integer.
For the final example, the token is contained in the non-default cookie named "Wasd_example" and the authentication performed using an X509 client certificate (which can only be supplied via SSL).
Some additional detail is available from the AUTHTOKEN.C code module.
Provides a username and password that is authenticated from data placed into the global common (i.e. in memory) by the site administrator. The username and password expire (become non-effective) after a period, one hour by default or an interval specified when the username and password are registered.
It is a method for allowing ad hoc authenticated access to the server, primarily intended for non-configured access to the online Server Administration facilities (9.1 Access Before Configuration) but is available for other purposes where a permanent username and password in an authentication database is not necessary. A skeleton-key authenticated request is subject to all other authorization processing (i.e. access restrictions, etc.), and can be controlled using the likes of '~_*', etc.
The site administrator uses the command line directive
The username must begin with an underscore (to reduce the chances of clashing with a legitimate username) and have a minimum of 6 other characters. The password is delimited by a colon and must be at least 8 characters. The optional period in minutes can be from 1 to 10080 (one week). If not supplied it defaults to 60 (one hour). After the period expires the skeleton key is no longer accepted until reset.
The authentication process (with skeleton-key) is performed using these basic steps.
Note that the authenticator resumes looking for a username from a configured authentication source unless the request and skeleton-key usernames match. After that the passwords either match allowing access or do not match resulting in an authentication failure.
The server account should have no direct write access to into any directory structure. Files in these areas should be owned by SYSTEM ([1,4]). Write access for the server into VMS directories (using the POST or PUT HTTP methods) should be controlled using VMS ACLs. This is in addition to the path authorization of the server itself of course! The recommendation to have no ownership of files and provide an ACE on required directories prevents inadvertant mapping/authorization of a path resulting in the ability to write somewhere not intended.
Two different ACEs implement two grades of access.
This example shows a suitable ACE that applies only to the original directory:
This example shows a suitable ACE that applies only to the original directory:
To assist with the setting of the required ACEs an example, general-purpose DCL procedure is provided, WASD_ROOT:[EXAMPLE]AUTHACE.COM).
Some sites may be sensitive enough about Web resources that the possibility of providing inadvertant access to some area or another is of major concern. WASD provides a facility that will automatically deny access to any path that does not appear in the authorization configuration file. This does mean that all paths requiring access must have authorization rules associated with them, but if something is missed some resource does not unexpectedly become visible.
At server startup the /AUTHORIZE=ALL qualifier enables this facility.
For paths that require authentication and authorization the standard realms and rules apply. To indicate that a particular path should be allowed access, but that no authorization applies the "NONE" realm may be used. The following example provides some indication of how it should be used.
There is also a per-path equivalent of the /AUTHORIZE=ALL functionality, described in SET Rule of WASD Configuration). This allows a path tree to be require authorization be enabled against it.
The server provides for users to be able to change their own HTA passwords (and SYSUAF if required). This functionality, though desirable from the administrator's viewpoint, is not mandatory if the administrator is content to field any password changes, forgotten passwords, etc. Keep in mind that passwords, though not visible during entry, are passed to the server using clear-text form fields (which is why SSL is recommended).
Password modification is enabled by including a mapping rule to the internal change script. For example:
Any database to be enabled for password modification must have a writable authorization path associated with it. For example:
Use some form of cautionary wrapper if providing this functionality over something other than an Intranet or SSL connection:
When using SYSUAF authentication it is possible for a password to pre-expired, or when a password lifetime is set for a password to expire and require respecification. By default an expired password cannot be used for access. This may be overridden using the following global configuration directive.
Expired passwords may be specially processed by specifying a URL with WASD_CONFIG_GLOBAL [AuthSysUafPwdExpURL] configuration directive Alphabetic Listings of WASD Configuration).
The WASD_CONFIG_MAP set auth=sysuaf=pwdexpurl=<string> rule allows the same URL to be specified on a per-path basis. When this is set a request requiring SYSUAF authentication that specifies a username with an expired password is redirected to the specified URL. This should directly or via an explanatory (wrapper) page redirect to the password change path described above. The password change dialog will have a small note indicating the password has expired and allows it to be changed.
The following WASD_CONFIG_GLOBAL directive
It is also possible to redirect an expired password to a site-specific page for input and change. This allows some customization of the language and content of the expired password change dialog. An example document is provided at WASD_ROOT:[EXAMPLE]EXPIRED.SHTML (what it looks like) ready for relocation and customisation. Due to the complexities of passing realm information and then submitting that information to the server-internal change facility some dynamic processing is required via an SSI document.
This example assumes the site-specific document has been located at WEB:[000000]EXPIRED.SHTML and is accessed using SSL.
The reason authorization information is not required to be reentered on subsequent accesses to controlled paths is cached information the browser maintains. It is sometimes desirable to be able to access the same path using different authentication credentials, and correspondingly it would be useful if a browser had a purge authorization cache button, but this is commonly not the case. To provide this functionality the server must be used to "trick" the browser into cancelling the authorization information for a particular path.
This is achieved by adding a specific query string to the path requiring cancellation. The server detects this and returns an authorization failure status (401) regardless of the contents of request "Authorization:" field. This results in the browser flushing that path from the authorization cache, effectively requiring new authorization information the next time that path is accessed.
There are two variations on this mechanism.
Also when using logout with a revalidation period a redirection URL may be appended to the logout query string. It then redirects to the supplied URL. It is important that the redirection is returned to the browser and not handled internally by WASD. Normal WASD redirection functionality applies.
These examples redirect to
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |
Transport Layer Security (TLS), and its predecessor Secure Sockets Layer (SSL), are cryptographic protocols designed to provide communication privacy over a network, in the case of HTTP between the browser (client) and the server. It also authenticates server and optionally client identity. TLS/SSL operates by establishing an encrypted communication path between the two applications, "wrapping" the entire application protocol inside the secure link, providing complete privacy for the entire transaction. In this way security-related data such as user identification and password, as well as sensitive transaction information can be protected from unauthorized access while in transit. This section is not a tutorial on TLS/SSL. It contains only information relating to WASD's use of it. See 4.9 SSL References for further information on TLS/SSL technology.
|
| WASD implements SSL using a freely available software toolkit supported by the OpenSSL Project. |
OpenSSL licensing allows unrestricted commercial and non-commercial use. This toolkit is in use regardless of whether the WASD OpenSSL package, HP SSL for OpenVMS product, or other stand-alone OpenSSL environment is installed. It is always preferable to move to the latest support release of OpenSSL as known bugs in previous versions are progressively addressed (ignoring the issue of new bugs being introduced ;-)
Be aware that export/import and/or use of cryptography software, or even just providing cryptography hooks, is illegal in some parts of the world. When you re-distribute this package or even email patches/suggestions to the author or other people, please PAY CLOSE ATTENTION TO ANY APPLICABLE EXPORT/IMPORT LAWS. The author of this package is not liable for any violations you make here.
Ralf S. Engelschall (rse@engelschall.com) is the author of the popular Apache mod_ssl package. This section is taken from the mod_ssl read-me and is well-worth some consideration for this and software security issues in general.
Have (or want) a TLS/SSL secured site?
Using self-signed or commercial server certificate(s)?
Let's Encrypt makes it possible to obtain and maintain browser-trusted certificates, simply, automatically and at no cost.
See WASD Certificate Management Environment (wuCME) on the WASD download page at https://wasd.vsm.com.au/wasd/
Secure Sockets Layer functionality is easily integrated into WASD and is available from one (or more) of the following sources. See for the basics of installing WASD SSL and for configuration of various aspects.
This is provided from the directory SYS$COMMON:[SSL111] containing shared libraries, executables and templates for certificate management, etc. If this product is installed and started the WASD installation and update procedures should detect it and provide the option of compiling and/or linking WASD against its shareable libraries.
WASD OpenSSL installation creates an OpenSSL directory in the source WASD_ROOT:[SRC.OPENSSL-n_n_n] (look for it here) containing the OpenSSL copyright notice, object libraries, object modules for building executables, example certificates, and some other support files and documentation.
The OpenSSL v1.1.1 code stream is supported. WASD requires a 32 bit OpenSSL build (the default).
To change linkage use step 2 described in selecting the alternate toolkit build.
OpenSSL v1.1.1 uses the naming schema OSSL$… for logical and file names. It also provides object libraries for a static linked executable, as well as shareable images, for the two main APIs (SSL and crypto). In common with the VSI SSL111 product, the shareable images must be installed to be used with the WASD server privileged executable. The WASD STARTUP.COM procedure will undertake this when directed (see immediately below).
There is one other consideration. For a privileged executable to activate a shareable image, not only must the image be installed but any associated logical names must be defined in executive (or kernel) mode. When executing the OpenSSL v1.1.1 startup procedure P1 must be "SYSTEM/EXECUTIVE" as in the following example:
SSL functionality can be installed with a new package, or with an update, or it can be added to an existing non-SSL enabled site. The following steps give a quick outline for support of SSL.
It is also possible to check the SSL package at any other time using the server demonstration procedure. It is necessary to specify that it is to use the SSL executable. Follow the displayed instructions.
The OPENSSL.EXE application is a command line tool for using the various cryptography functions of OpenSSL's crypto library from the shell. It is described being used several times in this section of the documentation. Refer to the OpenSSL Man page for descriptions of the various commands and their syntax.
It is commonly used as a foreign verb on VMS systems and assigned during SYLOGIN.COM or LOGIN.COM and depends on the distribution and version in use. For example:
A simple addition to SYLOGIN.COM or LOGIN.COM for WASD-specific OpenSSL kits to assign the OPENSSL verb is:
The example server startup procedure already contains support for the SSL executable. If this has been used as the basis for startup then an SSL executable will be started automatically, rather than the standard executable. The SSL executable supports both standard HTTP services (ports) and HTTPS services (ports). These must be configured using the [service] parameter. SSL services are distinguished by specifying "https:" in the parameter. The default port for an SSL service is 443.
WASD can configure services using the WASD_CONFIG_GLOBAL [SSL..] directives, the per-service WASD_CONFIG_SERVICE [ServiceSSL..] directives, or the /SSL= qualifier. Configuration precedence is WASD_CONFIG_SERVICE, /SSL= and finally WASD_CONFIG_GLOBAL.
SSL service configuration using the WASD_CONFIG_SERVICE configuration is slightly simpler, with a specific configuration directive for each aspect. (see Service Configuration of WASD Configuration). This example illustrates configuring the same services as used in the previous section.
As WASD uses the OpenSSL package in one distribution or another it largely supports all of the capability of that underlying package. The obsolete SSLv2, and the deprecated SSLv3 are no longer accepted by default. WASD default comprise the TLS family of protocols, at the time of writing, TLSv1, TLSv1.1, TLSv1.2 and TLSv1.3.
Some older clients employing SSLv3 may fail. Symptoms are dropped connection establishment and WATCH [x]SSL variously showing "SSL routines SSLn_GET_RECORD wrong version number", "SSL routines SSLn_GET_CLIENT_HELLO unknown protocol", possibly others. It is generally considered SSL best-practice not to have SSLv3 enabled but if required may be supported by configuring WASD_CONFIG_GLOBAL [SSLversion] with "SSLv3,TLSvALL", the per-service WASD_CONFIG_SERVICE equivalent, or using the /SSL=(SSLv3,TLSvALL) command line parameter during server startup.
TLSv1.3 perhaps should have been designated TLSv2.0 and not be considered as an incremental improvement over earlier versions of TLS but a significant upgrade!
TLSv1.3 can be tested for as demonstrated at ‘test TLS Version 1.3’ in 4.8 SSL Service Evaluation.
Ciphers are the algorithms, designed and implemented on mathematical computations, that render the readable plaintext into unreadable ciphertext. Ciphers tend to be available in suites (or families) where variants, usually based on key size and therefore resistence to decryption without a known key, that browsers and otheragents negotiate on and accept when setting up a secure (encrypted) network transports with servers.
Cipher selection is important to the overall security of the supported environment as well as the range of clients and servers that can establish communication due to shared cipher suites. Including only more recent (and technically secure) ciphers can preclude older clients from establishing secure connection, and including older (and perhaps more susceptible to modern attack) ciphers increases site vunerability. Some environments, for example HTTP/2, are quite prescriptive regarding the secure connection, to the point of blacklisting protocol versions and cipher suites no longer considered secure enough.
Fortunately a number of sites provide cipher guidelines based on requirements. The Mozilla Developer Network provides these amongst other useful information on security and server side TLS.
https://wiki.mozilla.org/Security/Server_Side_TLS
WASD has a default (built-in) functional cipher list that is general in application and relevant to when it was compiled. This in particular and site cipher lists in general, should be reviewed from time to time as opinions and requirements do change.
Many agents (browsers) require the elliptic curve ciphers provided by Forward Secrecy elements (4.5.5 Forward Secrecy) to negotiate later TLS versions.
The OpenSSL package provides for various options to be flagged against an TLS/SSL service. WASD sets the (OpenSSL) default options and then allows these to be overwitten/set/reset using hexadecimal values representing bit patterns. OpenSSL defaults are suitable for most sites.
The SSL options directives in global and per-service configuration, and the OPTIONS= keyword for the /SSL= qualifier, accept
Alternatively, the following OpenSSL option mnemonics can be used with a leading "+" to enable, or "-" to disable
Forward secrecy, sometimes known as perfect forward secrecy (PFS), is a property of key-agreement protocols ensuring that a session key derived from a set of long-term keys cannot be compromised if one of the long-term keys is compromised in the future.
http://en.wikipedia.org/wiki/Forward_secrecy
OpenSSL supports forward secrecy using Diffie-Hellman key exchange with elliptic curve cryptography and this relies on generating emphemeral keys based on unique, safe prime numbers. These are expensive to generate and so this is done infrequently, often during software build or installation. In the case of WASD, to maximise flexibility, these numbers are stored in external PEM-format files, by default located in the WASD_ROOT:[LOCAL] directory. These files are only briefly accessed during server startup SSL initialisation and the content later used during network connection SSL negotiation to generate the required ephemeral keys.
PFS requires a small number of elements working in concert
The detail is described in these references
Executing the WASD OpenSSL procedure
Alternatively, generated directly at the command-line using the OpenSSL dhparam utility, as in these examples;
Alternatively, files containing emphemeral keys generated freshly with each release, may be copied from the WASD OpenSSL package using
When a TLS/SSL connection is initiated an expensive handshake (in terms of time and compute) is required to establish the cryptographic and other elements of the connection. Mitigation of this expense is undertaken by allowing the resumption of a previous session (abbreviating the handshake exchanges) using connection state stored either at the server or at the client.
This TLS extension provides the connection state to the client, encrypted with keys available only to the server. The client stores the (encrypted) state and when (re-)connecting to the server provides that ticket in the initial part of the handshake. The server decrypts the ticket and if valid expedites the connection by resuming the previously negotiated session. This is the more modern, almost universally supported mechanism and is generally enabled by default.
Session tickets introduce a potential vulnerability to TLS security, in particular to the benefits of Forward Secrecy (PFS). If the ticket can be compromised, through theft of the keys or brute-force decryption attack, the entire session becomes vulnerable to attack. It is therefore advised to periodically rotate (change) the keys used by the server to encrypt the tickets. WASD does this every (RFC recommended) 24 hours, at midnight (local time).
Where a site is provided by multiple servers and connections distributed between these, session resumption using tickets relies on each server using the same keys. The current keys must be distributed to each server (using a secure mechanism) and this performed every time the keys are rotated. WASD uses the DLM to perform this for multiple per-node and cluster-wide instances as applicable.
In a full handshake the server sends a Session ID (unique, non-repeating value) as part of the handshake. On a subsequent connection the client can pass this session ID back to the server when connecting. To support session resumption via session IDs the server must maintain a cache that maps past session IDs to those sessions' states. The cache has limited capacity and is expensive for the server to maintain. If the session ID is still available in the cache the session can be resumed. This is the original session resumption mechanism.
Where a single WASD instance is involved the session cache is implemented in-memory. With multiple instances on a single node it is provided across those instances using a shared global section. The cacpacity of this shared cache is determined by the WASD_CONFIG_GLOBAL directives [SSLinstanceCacheMax] and [SSLinstanceCacheSize] directives. There is no cluster-wide session cache. When multiple instances are in use the shared session cache is enabled by default. Session ID caching may be globally disabled by setting [SSLsessionCacheMax] to -1.
With Session Tickets being the more modern, flexible and efficient solution to session resumption (and being available cluster-wide) it is recommended that WASD sites disable Session ID caching.
The default maximum period for session reuse is five minutes. This may be set globally using the [SSLsessionLifetime] directive or on a per-service basis using [ServiceSSLsessionLifetime].
To some extent, the relatively long-lived connections and lower concurrency with HTTP/2 means the importance of session resumption in improving request latency and connection overhead is reduced.
HTTP Strict Transport Security (HSTS) is a security policy mechanism which helps protect sites against protocol downgrade attack and cookie hijacking. It allows web servers to declare that browsers and other complying agents should only interact using secure (TLS) HTTP connections and never via clear-text HTTP. HSTS is an IETF standard specified in RFC 6797.
When global configuration directive [SSLstrictTransSec] is non-zero, or per-service configuration directive [ServiceSSLstrictTransSec] is non-zero, or a path is SET response=sts=<value>, TLS/SSL HTTP responses include a "Strict-Transport-Security: max-age=seconds" header field. Conforming agents note this period and refuse to communicate with the site via clear-text HTTP for the period represented by the integer number of seconds specified.
The server certificate is used by the browser to authenticate the server against the server certificate Certificate Authority (CA), in making a secure connection, and in establishing a trust relationship between the browser and server. By default this is located using the WASD_CONFIG_GLOBAL [SSLcert] or WASD_CONFIG_SERVICE [ServiceSSLcert] configuration directive, the WASD_CONFIG_SSL_CERT logical name, or using the /SSL= command-line qualifier, however if required. Each SSL service can have an individual certificate configured as in the example above.
The private key is used to validate and enable the server certificate. A private key is enabled using a secret, a password. It is common practice to embed this (encrypted) password within the private key data. This private key can be appended to the server certificate file, or it can be supplied separately. If provided separately it can be located using the WASD_CONFIG_GLOBAL [SSLkey] or WASD_CONFIG_SERVICE [ServiceSSLkey] configuration directive, tor using the WASD_CONFIG_SSL_KEY logical. When the password is embedded in the private key information it becomes vulnerable to being stolen as an enabled key. For this reason it is possible to provide the password separately and manually.
If the password key is not found with the key during startup the server will request that it be entered at the command-line. This request is made via the HTTPDMON "STATUS:" line (see OPCOM Logging of WASD Configuration), and if any OPCOM category is enabled via an operator message. If the private key password is not available with the key it is recommended that OPCOM be configured, enabled and monitored at all times.
When a private key password is requested by the server it is supplied using the /DO=SSL=KEY=PASSWORD directive (9.7 HTTPd Command Line). This must be used at the command line on the same system as the server is executing. The server then prompts for the password.
Multiple virtual SSL services (https:) sharing the same or individual certificates (and other characteristics) can essentially be configured against any host name (unique IP address or host name alias) and/or port in the same way as standard services (http:).
WASD SSL implements Server Name Indication (SNI), an extension to the TLS protocol that indicates what hostname the client is attempting to connect to at the start of the handshaking process. This allows a server to present multiple certificates on the same IP address and port number and hence allows multiple secure (HTTPS) websites (or any other Service over TLS) to be served off the same IP address without requiring all those sites to use the same certificate.
When the client presents an SNI server name during SSL connection establishment, WASD searches the list of services it is offering for an SSL service (the first hit) operating with a name matching the SNI server name. If matched, the SSL context (certificate, etc.) of that service is used to establish the connection. If not matched, the service the TCP/IP connection originally arrived at is used.
When authorization is in place (3. Authentication and Authorization) access to username/password controlled data/functionality benefits enormously from the privacy of an authorization environment inherently secured via the encrypted communications of SSL. In addition there is the possibility of authentication via client X.509 certification (4.5.12 Authorization Using X.509 Certification). SSL may be used as part of the site's access control policy, as whole-of-site, see 3.2 Authentication Policy, or on a per-path basis (see Request Processing Configuration of WASD Configuration).
The server access control functionality (authentication and authorization) allows the use of public key infrastructure (PKI) X.509 v3 client certificates for establishing identity and based on that apply authorization constraints. See 3. Authentication and Authorization for general information on WASD authorization and 3.4 Authorization Configuration File for configuring a X509 realm. 4. Transport Layer Security provides introductory references on public-key cryptography and PKI.
A client certificate is stored by the browser. During an SSL transaction the server can request that such a certificate be provided. For the initial instance of such a request the browser activates a dialog requesting the user select one of any certificates it has installed. If selected it is transmitted securely to the server which will usually (though optionally not) authenticate its Certificate Authority to establish its integrity. If accepted it can then be used as an authenticated identity. This obviates the use of username/password dialogs.
The initial negotiation and verification of a client certificate is a relatively resource intensive process. Once established however, OpenSSL sessions are usually either stored in a cache or stored encrypted withing the client, reducing subsequent request overheads significantly. Each session has a specified expiry period after which the client is forced to negotiate a new session. This period is adjustable using the "[LT:integer]" and "[TO:integer]" directives described below.
An X.509 client certificate is requested at either TLS/SSL connection establishment (WASD_CONFIG_GLOBAL [SSLverifyPeer], WASD_CONFIG_SERVICE [ServiceSSLverifyPeer]) or once the request has been made and assessed against authorisation rules. If an X509 realm controls access to the resources then the TLS/SSL connection is queried for an X.509 client certificate to authenticate the client and authorise the access.
This is performed via a TLS/SSL renegotiation and for this the connection must have been cleared of request data. In the case of a HEAD, GET, OPTIONS, etc. request, this already has implicitly occurred by there being no request body. For POST, PROPFIND, PUT, etc. requests, the client most likely already will be transmitting the request body. This (application data) must be absorbed before the client certificate renegotiation can be performed.
In avoiding disruption to the current request, any request body must be buffered (in full, based on the content length specified in the header) before issuing the renegotiation. This consumes memory and potentially large quantities. The default maximum buffer space is 1MB. The maximum request body size and hence maximum memory accomodated can be configured using the per-service WASD_CONFIG_SERVICE [ServiceSSLverifyDataMax] directive, or the global WASD_CONFIG_GLOBAL configuration directive [SSLverifyDataMax].
Where a request with a body exceeds the maximum allowed buffer space the authorisation fails. This can be observed using WATCH. Where very large files are being sent the only solution is to first authenticate with a request without a body (e.g. using OPTIONS) then using the persistent connection and associated X.509 authentication perform the PUT or POST.
WASD provides a range of capabilities when using X.509 client certificates.
The basic syntax for this field is the full extension name, and the short-hand equivalent.
The Subject Alternative Name (SAN) extension (in common with many others) may contain multiple data elements, each with a leading name, a colon, and a (if multi line) carriage-control terminated value. WASD parses these into unqiue fields using keywords fixed in function SesolaCertKeyword() and the site configurable logical name WASD_X509_EXTENSION_KEYWORDS value. To select one of these fields, for example the common (Microsoft) user principal name (UPN), append the required field name to the extension name as shown in the following example (includes "shorthand" equivalents, along with the underscore and equate variants). Note that the identifying name match is not case sensitive.
Object Identifiers (OIDs) may be used for either record and field name (if an unknown otherName) by prefixing with "OID_". For example, the SAN may be alternatively selected, and the (Microsoft) UPN, as in the following examples.
X509 certificate extensions are in general visible from WATCH and accessible via CGI variables (when enabled using SET SSLCGI=apache_mod_ssl_extens and SSLCGI=apache_mod_ssl_client path mappings). The identifying names derived from X509 extensions are built of the alphanumerics in the element names. Non-alphanumerics (e.g. spaces) have underscores substituted. Multiple underscores are compressed into singles. Where elements have identical names the first multiple has TWO underscores and the digit two appended, the second mutiple, two underscores and three appended, etc.
Of course, the WASD OpenSSL component must be installed and in use to apply client X.509 certificate authorization. There is general server setup, then per-service and per-resource configuration.
Client certificate authorization has reasonable defaults. If some aspect requires site refinement the WASD_CONFIG_GLOBAL [SSL..] directives (see WASD Configuration) or command-line /SSL= qualifier parameters can provide per-server defaults.
The location of the CA verification file can also be determined using the logical name WASD_CONFIG_SSL_CAFILE. The order of precedence for using these specifications is
The WASD_CONFIG_SERVICE directive is provided for per-service CA file specification, if necessary allowing different services to accept a different mix of CAs.
Client certificate authorization is probably most usefully applied on a per-resource (per-request-path) basis using WASD_CONFIG_AUTH configuration file rules. Of course, per-resource control also applies to services that always require a client certificate (the only difference is the certificate has already been negotiated for during the initial connection handshake). The reserved realm name "X509" activates client certificate authentication when a rule belonging to that realm is triggered. The following example shows such a rule providing read access to those possessing any verified certificate.
Optional directives may be supplied to the X.509 authenticator controlling what mode the certificate is accepted in, as well a further access-restriction rules on specifically which certificates may or may not be accepted for authorization. Such directives are passed via the "param=" mechanism. The following real-life example shows a script path requiring a mandatory certificate, but not necessarily having the CA verified. This would allow a certificate display service to be established, the "[to:EXPIRED]" directive forcing the client to explicitly select a certificate with each access.
A number of such directives are available controlling some aspects of the certificate negotiation and verification. The "[LT:integer]" directive causes a verified certificate selection to continue to be valid for the specified period as long as requests continue during that period (lifetime is reset with each access).
Optional "param=" passed conditionals may also be used to provide additional filtering on which certificates may or may not be used against the particular path. This is based on pattern matching against client certificate components.
These functions can be used in a similar fashion to mapping rule conditionals (see Conditional Configuration of WASD Configuration). This includes the logical ORing, ANDing and negating of conditionals. Asterisk wildcards match any zero or more characters, percent characters any single character. Matching is case-insensitive.
Note that the "IS:" and "SU:" conditionals each have a specific-record and an entire-field mode. If the conditional string begins with a slash then it is considered to be a match against a specified record contents within the field. If it begins with a wildcard then it is matched against the entire field contents. Certificate DN records recognised by WASD,
The following (fairly contrived) examples provide an illustration of the basics of X509 conditionals. When matching against Issuer and Subject DNs some knowlege of their contents and structure is required (see 4. Transport Layer Security for some basic resources).
Of course, access control via group membership is also available. The effective username for the list is the 32 digit fingerprint of the client certificate (shown as REMOTE_USER IN the first example of 4.5.18 X.509 Authorization CGI Variables), or the Subject DN record as specified using the [RU:/record=] directive. This may be entered into simple lists as part of a group of which membership then controls access to the resource. The following examples show the contents of simple list files containing the X.509 fingerprints, derived remote-user names, and the required WASD_CONFIG_AUTH realm entries.
In a similar fashion the effective username can be placed in an access restriction list. The following configuration would only allow the user of the certificate access to the specified resources. Other verified certificate holders would be denied access.
For the CA certificate component of the client certificate to be verified as being what it claims to be (and thus establishing the integrity of the client certificate) a list of such certificates must be provided for comparison purposes. For WASD this list is contained in a single, plain-text file variously specified using either the WASD_CONFIG_SSL_CAFILE logical or per-service "[ServiceSSLclientCAfile]" directives, or the global [SSLverifyPeerCAFile] directive.
Copies of CA certificates are available for such purposes. The PEM copies (base-64 encoded versions of the binary certificate) can be placed into this file using any desired text editor. Comments may be inserted by prefixing with the "#" character. For WASD this would be best stored in the WASD_ROOT:[LOCAL] directory, or site equivalent.
An example of how such a file appears is provided below (bulk of the file has been 8< snipped 8< for bevity).
The WASD OpenSSL package provides an example CA verification file. The exact date and source can be found in the opening commentary of the file itself. The contents of this file easily can be pared down to the minimum certificates required for any given site.
The bundle may be refreshed at any time using any reliable source. The cURL project provides such a resource suitable for its own use, Apache mod_ssl and WASD. This is sourced from the root certificates used by the Mozilla Foundation for its Firefox product (and others). Mozilla uses a non-PEM format source which must be converted before use by WASD. The cURL site provides this already converted for use with its own utility and made available as a general resource.
Download the bundle using a command-line tool as in this example
CGI variables specific to client certificate authorization are always generated for use by scripts and SSI documents. These along with the general WASD authorization variables are shown in the example below. Note, that due to length of particular items some in this example are displayed wrapped.
Other CGI variables optionally may be enabled using WASD_CONFIG_MAP mapping rules. See 4.5.18 X.509 Authorization CGI Variables. Specific client certificate variables providing the details of such certificates are available with SSLCGI=apache_mod_ssl. These are of course in addition to the more general apache_mod_ssl variables described in the above section. Note that where some ASN.1 records are duplicated (as in SSL_CLIENT_S_DN) some variables will contain newline characters (0x10) between those elements (e.g. SSL_CLIENT_S_DN_OU). The line breaks in this example do not necesarily reflect those characters.
This is not a tutorial on X.509 certificates and their management. Refer to the listed references, 4. Transport Layer Security, for further information on this aspect. It does provide some basic guidelines.
Certificates identify something or someone, associating a public cryptographic key with the identity of the certificate holder. It includes a distinguished name, identification and signature of the certificate authority (CA, the issuer and guarantor of the certificate), and the period for which the certificate is valid, possibly with other, additional information.
The three types of certificates of interest here should not be confused.
The various OpenSSL tools are available for management of all of these certificate types in each of the three SSL environments.
XCA is a GUI application intended for creating and managing X.509 certificates, certificate requests, RSA, DSA and EC private keys, Smartcards and CRLs. It uses the OpenSSL library for the cryptographic operations. The application is available for Linux, macOS and Windows, as well as source code.
The server uses a certificate to establish its identity during the initial phase of the SSL protocol exchange. Each server should have a unique certificate. An example certificate is provided with the WASD OpenSSL package. If this is not available (for instance when using the VSI SSL111 product) then the server will fallback to an internal, default certificate that allows SSL functionality even when no external certification is available. If a "live" SSL site is required a unique certificate issued by a third-party Certificate Authority is desirable.
See wuCME on the WASD download page at https://wasd.vsm.com.au/wasd/
A less satisfactory alternative to obtaining one of these certificates is provided by the WASD support DCL procedures, which are quick hacks to ease the production of certificates on an ad hoc basis. In all cases it is preferable to directly use the utilities provided with OpenSSL, but the documentation tends to be rather sparse.
The VSI SSL111$COM:SSL111$CERT_TOOL.COM described above can create self-signed certificates.
Also note that the WASD server dynamically generates a self-signed certificate for TLS services that otherwise do not have a configured server certificate. This is largely for testing a server immediately after installation (e.g. using @WASD_ROOT:[INSTALL]DEMO SSL at the command-line). This certificate suffers all the short-comings of self-signed certificates with modern browsers (post-2019) but is better than no certificate all all. Interestingly, Incognito/[In]Private instances of a browser are often more relaxed about accepting certificates with recognised security deficiencies (e.g. unknown Certificate Authority signing). At least at the time of writing.
The first requirement may be a tailored "Certificate Authority" certificate. As the Certificate Authority is non-authoritative (not trying to be too oxymoronic, i.e. not a well-known CA) these certificates have little value except to allow SSL transactions to be established with trusting clients. More commonly "Server Certificates" for specific host names are required.
CA certificates can be loaded into browsers to allow sites using that CA to be accessed by that browser without further dialog. Browsers commonly invoke a server certificate load dialog when encountering a site using a valid but unknown server certificate.
A manual load is accomplished by requesting the certificate in a format appropriate to the particular browser. This triggers a browser dialog with the user to confirm or refuse the loading of that certificate into the browser Certificate Authority database.
To facilitate loading CA certificates into a browser ensure the following entries are contained in the HTTP$CONFIG configuration file:
Then just provide a link to the required certificate file(s), and click.
If a site's server (or CA certificate) is changed and the server restarted any executing browsers will probably complain (Netscape Navigator reports an I/O error). In this case open the browser's certificate database and delete any relevant, permanently stored certificate entry, then close and restart the browser. The next access should initiate the server certificate dialog, or the CA certificate may be explicitly reloaded.
Recognised Certificate Authorities (CAs) such as Thawte and VeriSign publish lists of requirements for obtaining a server certificate. These often include such documents required to prove organisational name and the right to use the domain name being requested. Check the particular vendor for the exact requirements.
In addition, a document containing the site's private key is required. This is known as the Certificate Signing Request (CSR) and must be generated digitally at the originating site.
Using the VSI SSL111 for OpenVMS product "SSL Certificate Tool" described in 4.6 Certificate Management a CSR can easily be generated using its menu-driven interface. The alternative is using a command-line interface tool.
The following instructions provide the basics for generating a CSR at the command-line in the WASD and generally the any OpenSSL environment (including the VSI SSL111 for OpenVMS product).
Keep the SERVER.KEY file secure. You'll need it when you receive the certificate back from the CA.
The SERVER.CSR is what you send to the CA (usually by mail or Web form). It looks something like the following
You can see the details of this file using
Once the signed certificate has been issued by the Certificate Authority it can be placed directly into the server configuration directory, usually WASD_ROOT:[LOCAL], and configured for use from there. Using the certificate direct from the CA requires that the private key password be given to the server each time (4.5.9 SSL Private Key). It is possible to embed the password into the certificate key so that this is not required.
Remember to keep original files secure, only work on copies!
When using the VSI SSL111 product or other OpenSSL toolkit the verb may already be available.
Using an editor, ensure the header/trailer looks this:
CGI variables specific to SSL transactions optionally may be enabled using WASD_CONFIG_MAP mapping rules. (See Request Processing Configuration of WASD Configuration). The may be done on a specific per-path or general CGI basis. In the following examples, due to length of particular items, some in this example are displayed wrapped. Also, where some ASN.1 records are duplicated (as in SSL_CLIENT_S_DN), some variables will contain newline characters (0x10) between those elements (e.g. SSL_CLIENT_S_DN_OU). The line breaks in the examples do not necesarily reflect those characters.
The Apache mod_ssl client certificate details described in 4.5.18 X.509 Authorization CGI Variables above are not shown in the above example but would be included if the request was X.509 authenticated.
X509 certificate extensions are in general visible from WATCH and accessible via CGI variables when enabled using SET SSLCGI=apache_mod_ssl_extens and SSLCGI=apache_mod_ssl_client path mappings.
This section is just the barest introduction to a significant topic.
"How well do you know SSL? If you want to learn more about the technology that protects the Internet, you've come to the right place."
Not necessarily an endorsement by WASD but a useful resource in itself.
Provides a free and unencumbered, comprehensive SSL Server test service
https://www.ssllabs.com/ssltest/
reporting on certificate status, protocol version, cipher suites, handshakes with various simulated clients, and protocol details including known vulnerabilities. It also summarises the report with a colour-coded rating.
So to speak.
The OPENSSL command-line application (4.4 OPENSSL.EXE Application) provides a configurable client for checking and testing various aspects of server configuration and behaviour. The basic operation represented by the command-line
A "bad select 38" is a VMS (C-RTL) limitation of earlier versions of OpenSSL and is not present in later versions or on other platforms, and the default use of -s_client will prompt for an HTTP request line, send that to the server, and report the response.
Checking whether a specific protocol version is enabled on a site:
The following example shows a server test where the protocol version is NOT supported.
Server TLSv1.3 response may be checked using an OPENSSL.EXE v1.1.1 or later.
The following provide a starting-point for investigating SSL and OpenSSL further (verified available at time of publication).
| 5.1WASD HTTP/2 |
| 5.2HTTP/2 and Performance |
| 5.3HTTP/2 Configuration |
| 5.3.1Global Configuration |
| 5.3.2Service Configuration |
| 5.3.3HTTP/2 Set Rules |
| 5.4HTTP/2 Detection |
| 5.5HTTP/2 References |
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |
HTTP/2 is the most recent standard (RFC 7540, 2015) for implementing how HTTP is represented by, and transported between, client and server. It is not a ground-up rewrite of the established standard, HTTP/1.1 (RFC 2616, 1999). Those elements and semantics remain substantially the same. Instead HTTP/2 modifies how the data is encapsulated (framed) and transferred between agents, abstracting the complexity of this within the new protocol layer, leaving the application level largely insulated from change. As a result all existing HTTP/1.1 web-based environments should be able to continue without modification.
The focus of the protocol is on performance, in particular end-user perceived page rendering and web application responsiveness. With the original web use case being a relatively simple, single resource request-response, and early markup involving text with a few illustrative images, the single network connection, back-to-back request-response paradigm was simple to implement and worked well enough. In short time this moved to multiple network connections, each loading elements in parallel as the complexity and density of the individual elements on the pages increased, and to the introduction of HTTP/1.1 pipelining (back-to-back requests over a single connection) in an attempt to avoid request-response-request latency. Modern web documents and applications tend to have dozens of fine-grained elements that dynamically load resources based on the content of the page and/or user interaction. The single, then multiple network connections, each with its round-trip TCP connection establishment overhead and request-response blocking of resources, did not scale effectively. HTTP/2 replaces it with a single TCP connection on which multiple resources concurrently can be requested, pushed, and transferred. A more rigorous and effective implementation of the pipeline concept.
While multiplexing communication over a single network connection is a core performance technology there are other contributing elements. The framing layer uses binary tokens and parameters. The plain-text request and response headers of HTTP/1.n are replaced with tokenised, encoded and dynamically cached equivalents, commonly providing compression in excess of eighty percent. The relationship and priority of resources can be established allowing inferior resources to be delivered after or dependent on superior ones. The HTTP/2 server can send multiple responses to a single request. Known as server push it can be used to pre-load the browser (cache) with resources it has not encountered yet.
HTTP/2 has the potential to place additional load on the client and server in comparison to HTTP/1.n. One particular consideration for WASD sites is the stream concurrency setting of the HTTP/2 connection. The server specifies to the client the maximum number of concurent request-response (and server push) streams it will accept. RFC 7540 contains, "This limit is directional: it applies to the number of streams that the sender permits the receiver to create. Initially, there is no limit to this value. It is recommended that this value be no smaller than 100, so as to not unnecessarily limit parallelism." This translates to a hypothetical ten browsers connected to the site each with up to one hundred concurrent streams, or potentially one thousand active requests! Time to check those server configuration and SYSGEN parameters…
Note that HTTP/1.1 has recently been revisited with RFC 7230 family of specifications (2014) providing some clarifications and refinements on the original.
WASD HTTP/2 implements all of the essential requirements of RFC 7540 (naturally enough). This includes the framing protocol, datagram (message) and stream management, header compression (RFC 7541), connection settings and flow control, along with HTTP/2 connection establishment and termination (TLS ALPN and HTTP upgrade). It does not ((perhaps) currently) provide server-push or stream prioritisation and dependency.
Prior to the introduction of HTTP/2, WASD's fundamental abstraction was the request, with each request interfacing directly with the network stack. With an HTTP/2 protocol connection somewhat supplanting the role of a Transmission Control Protocol (TCP) connection in HTTP/1.n, a new level of communication abstraction was required between the request processing and the network processing. It should be noted that HTTP/2 itself is transported on TCP.
Another new layer of abstraction required interfacing each protocol's request/response header formats with the underlying server processing (avoiding excessive duplication of code). HTTP/1.n has a plain-text, carriage-control separated format, while HTTP/2 has a binary, compressed, lookup-table oriented format (RFC 7541). The layer was implemented using a key-value dictionary.
The accomodations for handling both HTTP/2 and HTTP/1.1, along with related and ancilliary design and code changes, have not measurably impacted overall WASD performance, although as noted below there is a server process CPU impost associated with HTTP/2.
WATCH reports have the network item: [x]HTTP/2. This provides a detailed overview of the underlying framing and connection management exchanges between client and server. WATCH reports are available to HTTP/2 connected clients with one consideration. Due to multiplexed requests over the single network connection, WATCHing the [x]HTTP/2 item of another request in the same browser (using the same HTTP/2 connection - and there can be multiple from a single browser) is not possible (or at least more code than it's worth). The HTTP/2 activity of the WATCHing generates more report items which generate … a descent into reporting oblivion.
WASD detects when a request is initiated on the same HTTP/2 connection as an [x]HTTP/2 WATCHing client and if this sort of reporting cascade is possible (any networking group item) advises
Workarounds?
With HTTP/2 not modifying the fundamentals of HTTP/1.1 semantics the commonly touted payoff for all the additional complexity (in implementation) is performance. While this is often stated in terms of page rendering speeds or web application responsiveness there is another significant measure of performance - efficiency. HTTP/2 much more efficiently utilises each network (TCP) connection, as well as reducing the (time and processing) overhead of setting-up and tearing-down of each of these required for parallelism under HTTP/1.1.
There are a number of sufficiently good analyses of both the factors that affect HTTP/2 performance and the actual performance relative to HTTP/1.1. See the references section and search the Web. This section contains some observations made during WASD HTTP/2 development. All of these seem to correspond with others' observations, as well as what might reasonably be expected considering the strategies employed by the protocol.
As also related in 11. Server Performance, using the same load profile as above** and using HTTP/1.1, WASD v11.0 compared to v10.4 showed ~5% additional CPU and duration. This is (probably) largely due to dictionary processing.
** 100 individual files, size 2kB to 250kB, 50 concurrent, ~30% CPU utilisation (~5% USER mode, mostly INTERRUPT servicing), batched 10,000 at a time over a LAN.
As described in Server and Site Testing in WASD Configuration the OWASP ZAP application is integral to WASD test and exercise. It can generate an intense stream of traffic via cleartext (port 80) or TLS (port 443).
Using the nghttpx proxy utility (see reference below) it is also used to exercise WASD's HTTP/2.
On the development bench Alpha PWS500 formal performance assessment using this is disappointing
See ‘HTTP/2 (encrypted)’ in 11.1 Simple File Request Turn-Around in section 11. Server Performance.
This may just reflect the CPU capacity of the benchmark system and that all requests are being transported through a single encrypted connection.
The simplest tool for getting a feel for, and elementary measurement of HTTP/2 may be found in the WASD_ROOT:[EXERCISE] directory. The document DOTTY.HTML and its companion files provide a page that loads a selectable number of resources (images) in a consistent and reproducible manner. This DOTTY.HTML can be accessed via unencrypted HTTP (http://), encrypted HTTP (https://) and services configured to provide HTTP/2 or HTTP/1.1. Using these combinations with the selectable volume of resources, elementary comparisons may be made in target environments.
The Server Admin, HTTP Report (9. Server Administration) contains comparative duration and bytes-per-second minimum/maximum/average for total server HTTP/2 and HTTP/1.n requests. These cannot simply be taken at face value without some consideration of the respective load profile but under controlled conditions can provide useful metrics.
Other development and load/performance tools were employed from a Linux platform. For someone educated in computing during the (19)70s, the availability of VM technology for such purposes is just brilliant! But you know, we were happy in those days, though we were poor.
Indispensible were
Many thanks to the developer(s) of this package.
While effectively transparent to the end-user, HTTP/2 has some aspects that need to be carefully considered by the server administrator.
Use of elliptic curve ciphers (ECDHE), as an element of Perfect Forward Security (PFS), is mandated for HTTP/2 (RFC 7540 section 9.2.2). The keys for the elliptic curve ciphers are stored in PEM-encoded files ocated in WASD_ROOT:[LOCAL]. These can be copied from the WASD OpenSSL package using
This SSL configuration and minimum cipher list seems to work for all major browsers at the time of writing:
HTTP/2 and its features are globally enabled and configured using directives contained in the WASD_CONFIG_GLOBAL configuration file.
| Directive | Description | Default |
|---|---|---|
| [Http2Protocol] | enabled or disabled on a whole-of-server basis | disabled |
| [Http2FrameSizeMax] | maximum frame size in octets (bytes) the server is prepared to receive | 16384 |
| [Http2HeaderListMax] | maximum number of octets (bytes) permitted in a received header once uncompressed | 65535 |
| [Http2HeaderTableMax] | maximum number of bytes permitted in the server-end header cache | 4096 |
| [Http2PingSeconds] | number of seconds between connection RTT pings | 300 |
| [Http2StreamsMax] | maximum number of concurrent streams (requests) the server permits on the connection | 32 |
| [Http2InitWindowSize] | initial window size (number of octets in transit) for flow-control purposes | 6291456 |
These largely reflect settings and defaults from RFC 7540 6.5.1
Using the WASD_CONFIG_SERVICE directive [ServiceHttp2Protocol] HTTP/2 may be disabled on a per-service basis. The default is enabled if HTTP/2 is enabled globally.
WASD request processing rules may be used on a per-path basis to modify (some) global configuration settings and provide other WevDAV configuation. See Request Processing Configuration of WASD Configuration).
| Rule | Description |
|---|---|
| HTTP2=PROTOCOL=1.1 | send a "HTTP_1_1_REQUIRED" error causing the client to use HTTP/1.1 (RFC 7540 section 7) |
| HTTP2=SEND=GOAWAY | send a "GOAWAY" frame to the client resulting in it dropping the HTTP/2 connection |
| HTTP2=SEND=PING | send a "PING" frame to the client calculating the Round Trip Time (RTT) of the connection |
| HTTP2=SEND=RESET | send a "RST_STREAM" frame to the client causing it to drop the HTTP/2 stream (request in progress) |
| HTTP2=STREAMS=MAX=integer | set the maximum concurrent streams on a per-path basis |
| HTTP2=WRITE=low|normal|high | When request data is written it is queued at the specified priority, where high priority are written before normal (default) and low priority, and normal priority before low. This is only for associated stream (request) and is not a connection or whole-of-server prioritisation. |
Use path SETings to prioritise some resources (e.g. CSS and JavaScript) over others (e.g. images) and potentially improve page rendering speed. Where multiple concurrent requests are being serviced on the one HTTP/2 connection this will deliver the higher priority content before others.
A request using HTTP/2 may be detected during processing with the http2: conditional.
See Conditional Configuration of WASD Configuration).
A script may detect HTTP/2 using the REQUEST_PROTOCOL CGI variable with the value "HTTP/2". Other protocol versions are similarly represented.
A Server-Side Includes (SSI) document can use variations on the following construct (and similar to the script suggestion immediately above) to detect and process the request protocol.
WASD_ROOT:[EXERCISE]SHTML.SHTML
At the time of writing there is no browser-supported mechanism for a dynamic document (i.e. JavaScript) determining the underlying HTTP protocol used to access a resource. To access this information the server must be used. The suggested method, and the one employed by the DOTTY.HTML tool described above, is to provide one JavaScript source for HTTP/2 and another for everything else.
The document would contain
The following provide a starting-point for investigating HTTP/2 (verified available at time of publication).
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |
Web-based Distributed Authoring and (not) Versioning for the WASD package.
Effective WASD WebDAV file-space (without significant naming constraints) relies on being hosted on ODS-5 volumes. Behaviour hosting file-space on ODS-2 volumes is untested (though possible provided file naming is constrained to ODS-2 conventions).
WASD WebDAV methods and request headers, etc., are also propagated to the scripting environment and so functionality may be implemented using CGI, CGIplus or RTE based applications.
WASD proxy-serving supports WebDAV methods, header fields, etc.
Generally WebDAV clients are applications other than browsers and so response bodies with human-readable error explanations are unnecessary and consume bandwidth to no good purpose, and so not provided.
File-systems are notoriously latent components relative to the rest of the system (more so with VMS). Any operation to collections (directories) are not going to be atomic and for large collections requiring many sub-operations the potential for the process to be interrupted or otherwise disturbed are enormous. File-systems are not databases amenable to extensive ACID operations.
In addition each file under WebDAV management has the potential for an associated but independent metadata file. This of course means for every DAV-specific resource file activity there is at least a file-system action to check for a metadata file and for some actions such as COPY the potential for an associated but entirely independent file operation.
Of course WebDAV was not intended or designed as a general file-system protocol but one for distributed management of somewhat restricted collections of Web-related resources and so in context probably works well enough.
See sections below on file-system operation method restrictions.
during server WebDav file-system modifications is a recipe for inconsistency and/or corruption!
A list of WebDAV methods, what WASD does with them, and any limitations or restrictions. Some of these are familiar HTTP/1.n methods and other are RFC 4981 specific. Some of the HTTP/1.n methods are overloaded with additional or variant behaviours when used in a WebDAV context. Issues of atomicity with the manipulation of file-system trees containing numbers of individual files makes strict RFC 4918 compliance difficult. See "…Restrictions" below.
| Method | Description |
|---|---|
| COPY** | Reproduces both single resources (files) and collections (directory trees). Will overwrite files (if specified by the request) but will respond 209 (Conflict) if it would overwrite a tree. |
| DELETE** | deletes files and directory trees |
| GET | just the vanilla HTTP/1.1 behaviour |
| HEAD | ditto |
| LOCK** | see WEBDAV LOCKING below |
| MKCOL** | create a directory |
| MOVE** | Moves (rename or copy) a file or a directory tree. Will 'overwrite' files (if specified by the request) but will respond 209 (Conflict) if it would overwrite a tree. |
| OPTIONS | If WebDAV is enabled and available for the path this reports the WebDAV extension methods |
| PROPFIND** | Retrieves the requested file characteristics, DAV lock status and 'dead' properties for individual files, a directory and its child files, or a directory tree. |
| PROPPATCH** | set and remove 'dead' meta-data properties |
| PUT | Against a WebDAV resource behaves a little differently to historical WASD implementation of PUT. |
| UNLOCK** | see WebDAV locking below |
| **WebDAV RFC 4918 method |
WASD Statistics Reports gather WebDAV related data. Where a method can be used both for vanilla HTTP/1.n and WebDAV purposes it is counted in WebDAV statistics if the request header contains some other indication of a WebDAV activity.
Does not comply with the overwrite:T directive for collections (does so for files). Will not preemptively delete the existing tree. It returns a 209 (Conflict) response instead.
COPY does not maintain collection consistent URL namespace if a member resource cannot be moved as required by RFC 4918. It should maintain the source subtree completely uncopied. Instead it is best-effort and continues copying resources until exhausted. This is consistent with file-system behaviour. The RFC 4918 requirement, while not impossible, is fraught with issues inside a file-system.
Deletion of collections is particularly fraught with issues for a file-system. In userland it is almost impossible to predetermine if an individual file in a directory tree is going to resist deletion (due to locking, protections, etc) and in kernel land it's probably no easier. It leaves the undeleted tree hierachy (resource ancestors) intact. This is RFC 4918 compliant however!
So, in the case of WASD WebDAV it's just best-effort and if something down the tree won't disappear, it just reports the failure in the 207 response and carries merrily on through the tree regardless. This IS acceptable WebDAV server behaviour!
Does not comply with the overwrite:T directive for collections (does so for files). Will not currently pre-emptively delete the existing tree. It returns a 209 (Conflict) response instead.
MOVE first attempts to rename the file or directory. This is reasonably efficient, especially for directory trees but obviously only suitable for a target on the same disk volume. If a rename failure is due to a different device it falls back to using a COPY then DELETE in two separate phases. Needless-to-say this is hardly atomic and can lead to inconsistencies between source and target.
MOVE does not maintain collection consistent URL namespace if a member resource cannot be moved as required by RFC 4918. It should maintain the source subtree unmoved. Instead it is best-effort and continues moving resources until exhausted. This is consistent with file-system behaviour. The RFC 4918 requirement, while not impossible, is fraught with issues inside a file-system.
The conditional "If:" request header field does not have full RFC 4918 support. It implements lock token and etag token processing with parenthetical OR and NOT processing. For unsupported features WATCH reports that the header was not understood and always returns an abort status. WebDAV "If:" processing is an extrodinarily complex kludge for on-the-fly decision making by the server and much of what I have read indicates most clients only ever use extremely simple conditions anyway.
WebDAV and its features are globally enabled and configured using directives contained in the WASD_CONFIG_GLOBAL configuration file.
| Directive | Description |
|---|---|
| [PutMaxKBytes] | maximum size of a file (PUT and POST) |
| [WebDAV] | This directive enables and disables WebDAV. |
| [WebDAVlocking] | Enables and disables WebDAV locking. |
| [WebDAVlockTimeoutDefault] | see ‘Locking Timeout’ in 6.4 WebDAV Locking |
| [WebDAVlockTimeoutMax] | see ‘Locking Timeout’ in 6.4 WebDAV Locking |
| [WebDAVlockCollectionDepth] | See ‘Locking Depth’ in 6.4 WebDAV Locking |
| [WebDAVmetaDir] | see 6.3 WebDAV Metadata |
| [WebDAVquota] | Enables and disables RFC 4331 functionality (disk quota reporting). |
In addition these and other configurations are provided on a per-path basis using mapping rules.
WASD request processing rules (see Request Processing Configuration of WASD Configuration) may be used on a per-path basis to modify (some) global configuration settings and provide other WevDAV configuation.
| Rule | Description |
|---|---|
| ODS=NAME=8BIT|UTF8|DEFAULT | When a file is PUT using WebDAV (or upload), for non-7bit ASCII file names use native ODS-5 8bit syntax (default) or UTF-8 encoded character sequences (see 6.2.2 File Naming) |
| PUT=MAX=<integer> | * | Maximum number of kilobytes file size, if "*" then effectively unlimited (per-path equivalent of the global directive [PutMaxKBytes]). |
| WEBDAV=[NO]HIDDEN | list (default) or hide U*x hidden files (i.e. those with names beginning with period) |
| WEBDAV=[NO]LOCK | allow/apply WebDAV locking to this path |
| WEBDAV=[NO]PROFILE | WebDAV access according to SYSUAF profile |
| WEBDAV=[NO]PROP | allow/apply WebDAV 'dead' property(ies) to this path |
| WEBDAV=[NO]PUT=LOCK | a resource must be locked before a PUT is allowed |
| WEBDAV=[NO]READ | WebDAV methods allowed read this tree |
| WEBDAV=[NO]SERVER | WebDAV access as server account (best effort) |
| WEBDAV=[NO]WINPROP | when NOWINPROP windows properties are ignored and emulated |
| WEBDAV=[NO]WRITE | WebDAV methods allowed write to this path (implied read) |
| WEBDAV=LOCK=TIMEOUT=DEFAULT= | hh:mm:ss |
| WEBDAV=LOCK=TIMEOUT=MAX= | hh:mm:ss |
| WEBDAV=META=DIR= | per-path equivalent of global [WevbDAVmetaDir] (see 6.3 WebDAV Metadata) |
An essential function of the path setting rules is for specifying which paths in server Web-space are allowed to be accessed using the WebDAV protocol and what sort of access (read, write, etc.) that path is allowed.
By default files that are PUT via WebDAV (or upload) support the ISO Latin-1 character set. ASCII and non-7-bit file names use the native ODS-5 syntax. Where character sets other than ISO Latin-1, or where compatibility with other WebDAV implementations is desired (e.g. Apache), a path can be set to allow file names supplied using UTF-8 sequences.
For example, the English language word "naïve", having a diaeresis mark over the "i" character (indicating it is pronounced separately from the preceding vowel) is commonly respresented using the 8 bit character 0xEF, or as the two byte UTF-8 sequence 0xC3AF. This word if used as the file name with a type (extension) of ".TXT" by default would have the sequence of 8-bit characters
"Index of" (directory) listings will honour a path set ods=name=utf8 and make the listing character set UTF-8 resulting in a browser correctly rendering the name (WebDAV listings are by definition UTF-8).
While files and directories created via WebDAV will have a consistent naming schema applied those created by applications or manual operation on the VMS system can result in files that are not accessible with WebDAV.
For example the file name
In addition, the two files
To avoid this situation a potentially ambiguous file name containing an escaped period and no type (extension) is ignored by directory listings and WebDAV property lists. When an ambiguous file name is detected it is reported in WATCH reports.
Is controlled using the mapping rules:
| Rule | Description |
|---|---|
| WEBDAV=PROFILE | access using request SYSUAF-authenticated security profile |
| WEBDAV=WRITE | unconditional permission to read/write |
| WEBDAV=READ | unconditional permission to read |
| WEBDAV=SERVER | access using server account permissions |
All access by WebDAV operations must have at least one set against the path. If access is permitted by one of the above settings SYSPRV is enabled to allow that access using the server account. Therefore files and directories should have a SYSTEM:READ+WRITE+EXECUTE+DELETE protection or equivalent ACL permissions, or the access may fail totally or in some part of a supposedly atomic action.
These file-system access settings are applied in the order listed above. That is, if a path successively has one or more of the above settings applied during rule processing, when it comes to applying those access controls, SYSUAF profile is applied, then if no profile SETing access to read/write, then to read-only, then access via the server account.
In addition WebDAV access requires an authorisation rule against each path.
All access by WebDAV operations must have one set against the path.
All WebDAV access is a combination of WASD_CONFIG_MAP path setting and WASD_CONFIG_AUTH authorisation permissions. The least permissive of the two overrides the more. The combination of an authorisation rule and a path mapping rule mitigates the chance of opening unintended access into the file-system.
These is the test-bench environment used during development:
Note that WebDAV read/write access is a combination of the mapping and the authorisation rule (mapping WEBDAV=READ overrides authorisation read+write). Expect complications with Microsoft environments.
For test-benching you could avoid authorisation issues completely with:
A common requirement is to provide concurrent general access and authorised WebDAV acccess to the same Web-space. This is accomplished by using two paths mapped into the same file-system space, the general access (non-authorised) path, and a WebDAV (authorised) path. The WebDAV client uses the authorised path and can then apply WebDAV methods to maintain the resources.
The following configuration is taken from a site using WebDAV to allow users to manage their Web presence. The user mapping is a fairly standard configuration for VMS accounts (see Mapping User Directories (tilde character ("~")) of WASD Configuration). User Web areas are in the [.WWW] subdirectory of the account home area.
The four WebDAV access rules are located before the three general user access rules. The WebDAV rules are more specific. The first USER rule maps subdirectories - and the parent if a trailing slash is included. The second USER rule maps the parent directory for user agents that do not include trailing slash on their directory specifications (most it seems).
The second pair of rules reverse-maps the VMS file-system specifications represented by the result (right side) of the PASS rule into the path represented by the template (left side) of the PASS rule. Mapping from file-specifications to paths is necessary because of the way the PROPFIND method searches the file-system and then reports its results to the client as URLs.
The use of the notepad rule with a string of "webdav" (the actual string is not significant as long as it is unique within the rules) is used to conditionally process the reverse-mapping rules. They will be applied only to the requests originally mapped by the USER rules. The pass:-1 ensures the rules are only applied during reverse-mapping, not during request mapping.
The fifth rules maps general Web access to the user area. Remember, web access is to a user home subdirectory [.WWW].
The sixth and seventh rules reverse-map the VMS file-system specifications for the general USER rules for similar reasons to those described above. Why two? The user directories occur across two disk volumes and so each must be reverse-mapped.
As noted above, WASD WebDAV requires both mapping and authorization rules (even for "world" - or non-authenticated - access).
In this case authorisation is only required for WebDAV access. There are two rules. The first authorises subdirectories and parent directories for agents that supply a trailing slash. The second for agents that do not provide a trailing slash.
… two rules for each location? Why
For finer control. The first only matches requests with a path of "/~user/dav/subdir/" and "/~user/dav", whereas the latter matches "/~user/dav/subdir/" and "/~user/dav" and "/~user/david/" and "/~user/davros", etc.
Metadata is data (information) about data. WebDAV uses the concept of a resource property. There are "live" properties and "dead" properties. Essentially the live properties are the dynamic characteristics of a file-system object represented by creation and modification date-times, object size, etc. WebDAV dead properties are those supplied by WebDAV clients as XML entities and stored associated with the particular WebDAV object, in WASD's case the file-system object (file or directory). WASD also uses the file metadata to store resource lock data (see 6.4 WebDAV Locking).
WASD manages resource metadata using a separate file associated by name with the data file. This is done for reasons of programmatic simplicity and for the convenience of any command-line owner or sysadmin of the resources. No specialised tools are required. This metadata file can be stored in one of three locations.
For example, with the global directive
For example, with the global directive
The metadata file associated with a directory is stored in the same metadata location as files contained by that directory (not in the metadata location associated with the parent directory that contains the directory file). This metadata file is named ".DIR__wasdav" (i.e. no name, just an extension), with the following example illustrating how this would appear in each of the three metadata locations, for a subdirectory named "New Folder".
All metadata is stored using XML. Multiple XML data can be contained in a single metadata file. Each can be individually manipulated by a WebDAV client. The property elements are stored as-supplied by the client. It is presumed that their XML well-formedness is guaranteed by the original request XML parsing. Metadata files have content similar to the following:
This metadata example contains four properties; an exclusive write lock owned by "MGD" and three set by a client in three different (contrived) namespaces.
An example of such property meta-data generated by a Microsoft Windows (not Internet) Explorer client (example wrapped for presentation):
Every file written or modified by Windows Explorer generates this sort of metadata which is then stored in an associated metadata file and read each time the data file is accessed. Some might consider this unnecessary clutter in most circumstances (I do). WASD allows this metadata to be suppressed and equivalent data generated (fudged) from file live properties when accessed - often sufficient for purpose. To suppress the actual processing of Windows Explorer metadata set a path using the WEBDAV=NOWINPROP in WASD_CONFIG_MAP.
For efficiency and functionality considerations WebDAV locking may be enabled and disabled (default) as global functionality using the WASD_CONFIG_GLOBAL [WebDAVlocking] directive. Additionally the WEBVDAV=[NO]LOCKING path SETing can configure this on a per-path basis.
In common with RFC 4918 WASD WebDAV locking controls only write access. Both exclusive and shared locks are provided. Locking applies to the DELETE, LOCK, MKCOL, MOVE, PROPPATCH, PUT, and UNLOCK methods.
WASD WebDAV locking checks parent collections to a configurable depth. WASD_CONFIG_GLOBAL directive [WebDAVlockCollectionDepth] where the default (0 or 1) checks only WebDAV locking on files, 2 WebDAV locking on the parent directory, 3 on the grandparent, 4 the great-grandparent, etc. Of course each level can add significant latency (and expense) to some operations.
For more information on locking operation and implementation details see the DAVLOCK.C module and for meta-data in general the DAVMETA.C module.
When a client locks a resource it can specify the period for the lock. In the absence of such a specification WASD will apply the [WebDAVlockTimeoutDefault] value (by default 0-01:00:00 - one hour). WASD also applies the [WebDAVlockTimeoutMax] maximum lock period (by default 7-00:00:00 - one week). When the maximum period expires the lock is no longer valid.
WASD uses VMS locking to queue and arbitrate access to WebDAV resources and meta-files.
Two lock modes are employed; 'exclusive', when changes are to be made to the resource or its meta-data, and 'concurrent read', when resource and/or meta-data are only to be read. Concurrent read locks are compatible, but an exclusive queued against a resource currently being read waits, as does a read against a current exclusive.
WASD takes out its own VMS DLM locks on resources (files and directories) before beginning any WebDAV operation, and these prevent conflict with other WASD WebDAV operations on the same system or cluster, but RMS does not use these nor does WASD use RMS locks (except when actually acessing the file-system of course), and so there is potential for interactions between the two domains (in common with general file-system actvities). WASD WebDAV deliberately does not try to block file-system actions from other processing (except where RMS locks/blocks). Its own DLM locking is purely for internal purposes.
Some application/environment-specific considerations when using WASD WebDAV. Please report any you encounter for future inclusion in this section. Also see 6.6 Microsoft Miscellanea immediately below.
OS X Finder requires [WebDAVlocking] enabled for read/write access, otherwise access will be read-only.
Dreamwever 8 (at least, the only version I have access to) insists on using a URI with a trailing "/./" occasionally (I'm guessing to specify the "current" directory - cf. "/../", or "parent" syntax). Just absorb this internally using an appropriate mapping internal redirect.
A cornucopia of of minor and major considerations!
Microsoft approach WebDAV in their own inimitable fashion. Hence Microsoft agents, considering their ubiquity, including their mini-redirector are specifically looked for and functionality modified to accomodate them.
The following is a list topics/issues that were encountered/investigated during WASD WebDAV development. They may or may not be applicable to your site.
Some general references:
DOS/Windows command-line network configuration:
Microsoft agents (at least) seem to request the server OPTIONS of the server root regardless of any path provided with the NET USE or other network drive mapping employed. To selectively map such a request into a path that has WebDAV enabled on it (and will therefore respond with the DAV-related options) use a conditional redirect rule. For example
Subsequent rules will probably be required to map typeless directory requests to the actual directory required.
Requests containing paths /_vti_inf.html and /_vti_bin/* are related to FrontPage protocol discovery probing. They can be adequately handled using a mapping rule lsuch as the following:
See ‘Microsoft Metadata’ in 6.3 WebDAV Metadata.
If the server's response does not contain an MS-Author-Via header, the OLE DB Provider for Internet Publishing loads the WEC and WebDAV protocol drivers one at a time (WEC first, WebDAV second) and asks them, "Do you know how to handle this URL?", specifying the exact URL passed in by the client. The first protocol which responds "yes" is selected. If neither protocol driver responds "yes" then the method which triggered the automatic driver selection (usually IBindResource::Bind) fails with an OLE DB Provider for Internet Publishing specific error code IPP_E_SERVERTYPE_NOT_SUPPORTED.
Some Windows XP machines have a broken Web Folders installation. Microsoft includes a Web Folders repair utility built in to Windows to correct the problem. Use the following steps to fix the problem:
Attach the port-number (80 by default) to the http-address you enter into the field of the "My Network Places"-assistant. As you can see in the following image and the linked screenshot, this will force Windows XP to use the "Microsoft Data Access Internet Publishing Provider DAV 1.1" mechanism instead of "Microsoft-WebDAV-MiniRedir/5.1.2600".
It is also possible to add the number sign # to the http-address you enter into the field of the "My Network Places"-assistant. As you can see in the following image and the linked screenshot, this will also force Windows XP to use the "Microsoft Data Access Internet Publishing Provider DAV 1.1" mechanism instead of "Microsoft-WebDAV-MiniRedir/5.1.2600".
There is a third way to get this working from the client-site. As described in the Microsoft Knowledge Base, Article ID: 841215, Windows XP disables "Basic Auth" in his "Microsoft-WebDAV-MiniRedir/5.1.2600"-mechanism by default for security reasons. See description below.
You can enable BasicAuth by adding the following registry key and setting it to a non-zero value:
If you delete the registry key or set it to 0, the behavior reverts to the default, or disabling the use of BasicAuth.
Disabling Basic Authentication over a clear channel:
Because the DAVRdr is part of the remote file-system stack, a computer is open to attack whenever an attempt is made to remotely access files. Although the threat to other applications that use the Internet APIs is less severe than it is for the DAVRdr, a similar attack is possible whenever an application (or the user) attempts to access a URL. For this reason, WinInet is exposing the mechanism by which the DAVRdr disables BasicAuth to other users of the Internet APIs.
With Windows XP Service Pack 2, there are two ways to block the use of Basic Authentication over clear (or unencrypted) channels:
Create the following registry key and set it to a non-zero value.
This prevents WININET from attempting to use BasicAuth unless the channel is secured (HTTPS or SSL).
The application can disable the use of BasicAuth for its connections by setting the AUTH_FLAG_DISABLE_BASIC_CLEARCHANNEL flag (0x4) in the value supplied in the call to InternetSetOption using INTERNET_OPTION_AUTH_FLAGS.
You can enable BasicAuth by setting the following registry key to the value 3 and restarting the WebClient service:
"In my case I try to copy file over WEBDAV to WEB Client connection e.g. I have mapped drive to web site. file is about 70MB I can copy small files from the same WEBDav folder."
These are the resources used during WASD WebDAV development.
All these have been used during WASD WebDAV development.
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |
A proxy server acts as an intermediary between Web clients and Web servers. It listens for requests from the clients and forwards these to remote servers. The proxy server then receives the responses from the servers and returns them to the clients. Why go to this trouble? There are several reasons, the most common being:
No additional software needs to be installed to provide proxy serving.
Proxy servering is essentially configured using a combination of configuration directives in WASD_CONFIG_GLOBAL and WASD_CONFIG_SERVICE to enable proxy serving both globally and then for allow a specific service to make outgoing connections, along with mapping directives in WASD_CONFIG_MAP to control and direct those outgoing connections.
The following steps provide a brief outline of proxy configuration.
When proxy processing is enabled and WASD_CONFIG_GLOBAL directive [ReportBasicOnly] is disabled it is necessary to make adjustments to the contents of the WASD_CONFIG_MSG message configuration file [status] item beginning "Additional Information". Each of the "/httpd/-/statusnxx.html" links
If this is not provided the links and any error report will be interpreted by the browser as relative to the server the proxy was attempting to request from and the error explanation will not be accessible.
WASD provides a proxy service for the HTTP scheme (prototcol).
Proxy serving generally relies on DNS resolution of the requested host name. DNS lookup can introduce significant latency to transactions. To help ameliorate this WASD incorporates a host name cache. To ensure cache consistency the contents are regularly flushed, after which host names must use DNS lookup again, refreshing the information in the cache. The period of this cache purge is contolled with the [ProxyHostCachePurgeHours] configuration parameter.
When a request is made by a proxy server is is common for it to add a line to the request header stating that it is a forwarded request and the agent doing the forwarding. With WASD proxying this line would look something like this:
An additional, and perhaps more widely used facility, is the Squid extension field to the proxied request header supplying the originating client host name or IP address.
Proxy serving is enabled on a global basis using the WASD_CONFIG_GLOBAL file [ProxyServing] configuration parameter. After that each virtual service must have proxy functionality enabled as a per-service configuration.
WASD can configure services using the WASD_CONFIG_GLOBAL [service] directive, the WASD_CONFIG_SERVICE configuration file, or even the /SERVICE= qualifier.
Using directives listed in Service Configuration of WASD Configuration) this example illustrates configuring a non-proxy server (the disabled is the default and essentially redudant) and a proxy service.
High performance/highly available proxy server configurations require more than one instance configured and running. Whether this is done by running multiple instances on the same host or one instance on multiple hosts, it leads to situations where successive requests will be processed by different instances. As those instances don't share a common name to IP address cache, they will eventually use different IP addresses when trying to connect to an origin server running on multiple hosts.
This may result in the following, user visible, issues:
For these reasons, the proxy server will make every effort to relay successive requests from a given client to the same origin host as long as this one is available (built-in failover capability will ultimately trigger the choice of a new host). This is known as client to origin affinity or proxy affinity capability.
Proxy to origin server affinity is enabled using the following service configuration directive.
Obviously the use of cookies must be enabled in the browser or this facility will not operate for that client. After the first successful connection to an origin host, the proxy server will send a cookie indicating the IP address used to the client browser. Upon subsequent requests, this cookie will be used to select the same host. The cookie is named WasdProxyAffinity_origin.host.name and the value simply the IP address in dotted decimal. This cookie is not propagated beyond the proxy service but may be WATCHed by checking the Proxy Processing item.
It is possible to make the outgoing request appear to originate from a particular source address. The Network Interface must be able to bind to the specified IP address (i.e. it cannot be an arbitrary address).
The same behaviour may be accomplished with an WASD_CONFIG_MAP mapping rule.
Some sites may already be firewalled and have corporate proxy servers providing Internet access. It is quite possible to use WASD proxying in this environment, where the WASD server makes the proxied requests via the next proxy server in the hierarchy. This is known as proxy chaining.
Chaining may also be controlled on a virtual service or path basis using an WASD_CONFIG_MAP mapping rule.
If the upstream proxy server requires authorization this may be supplied using a per-service directive
The basic: keyword allows WASD to appropriately encode the credentials. Basic authentication is the only scheme currently supported.
Requests at a service enabled for proxy processing are directed to proxy processing using a fundamental rule which terminates rule processing and initiates the outgoing connection.
Controlling both access-to and access-via proxy serving is possible.
Access to the proxy service can be directly controlled through the use of WASD authorization. Proxy authorization is distinct from general access authorization. It uses specific proxy authorization fields provided by HTTP, and by this allows a proxied transaction to also supply transaction authorization for the remote server. In the WASD_CONFIG_SERVICE configuration file.
In addition to the service being specified as requiring authorization it is also necessary to configure the source of the authentication. This is done using the WASD_CONFIG_AUTH configuration file. The following example shows all requests for the proxy virtual service must be authorized (GET and well as POST, etc.), although it is possible to restrict access to only read (GET), preventing data being sent out via the server.
An up-stream, chained proxy server (7.1.4 Proxy Chaining) may be permitted to receive proxy authentication from the client via a WASD proxy server using the CHAIN keyword. Unconfigured, WASD does not propagate HTTP proxy authorization fields. Only one proxy server in a chain can be authenticated against.
It is also possible to control proxy access via local authorization, although this is less flexible by removing the ability to then pass authorization information to the remote service. In other repects it is set up in the same way as proxy authorization, but enabled using the LOCAL keyword.
Extensive control of how, by whom and what a proxy service is used for may be exercised using WASD general and conditional mapping Request Processing Configuration of WASD Configuration) and Conditional Mapping of WASD Configuration) possibly in the context of a virtual service specification for the particular connect service host and port (see Virtual Servers of WASD Configuration). The following examples provide a small indication of how mapping could be used in a proxy service context.
Using path mapping rules (see Request Processing Configuration of WASD Configuration). it is possible to remove or specifically set selected proxied request headers. Many headers are critical to server processing but some are informational or otherwise amenable to change. This can be undertaken using the SET mapping rule proxy=header=<parameter>.
For example, to have a proxy service suppress the "Referer:" request header:
To modify the "Referer:" request header to a fixed URL:
To modify the "User-Agent:" request header to a specific string:
Caching involves using the local file-system for storage of responses that can be reused when a request for the same URL is made. The WASD server does not have to be configured for caching, it will provide proxied access without any caching taking place.
When a proxied request is processed, and the characteristics would allow the response to be cached, a unique identifier generated from the URL is used to create a corresponding file name. The response header and any body are stored in this file. This may be the data of an HTML page, a graphic, etc.
When a proxied request is being processed, and the characteristics would allow the request to be cached, the unique identifier generated allows for a previously created cache file to be checked for. If it exists, and is current enough, the response is returned from it, instead of from the remote server. If it exists and is no longer current the request is re-made to the remote server, and the response if still cacheable is re-cached, keeping the contents current. If it does not exist the response is delivered from the remote server.
The main critera are for the response to be successful (200 status), general (i.e. one not in response to a specialized query or action), and not too volatile (i.e. the same page may be expected to be returned more than once, preferably over an extended period).
The [ProxyCacheFileKbytesMax] configuration parameter controls the maximum size of a response before it will not be cached. This can be determined from any "Content-Length:" response header field, in which case it will proactively not be cached, or if during cache load the maximum size of the file increases beyond the specified limit the load is aborted.
As many transactions on today's Web contain query strings, etc., and therefore cannot be meaningfully cached, it should not be assumed the cost/benefit of having a proxy cache enabled is a forgone conclusion. Each site should monitor the proxy traffic reports and decide on a local policy.
The facilities described in 7.2.6 Reporting and Maintenance allow a reasonably informed decision to be made. Items to be considered.
Last, but by no means least, understanding the characteristics of local usage. For example, are there a small number of requests generating lots of non-cacheable traffic? For instance, a few users accessing streaming content.
Selection of a disk device for supporting the proxy cache should not be made without careful consideration, doubly so if significant traffic is experienced. Here are some common-sense suggestions.
Initially the directory will need to be created. This can be done manually as described below, or if using the supplied server startup procedures (see Server Startup of WASD Configuration) it is checked for and if it does not exist is automatically created during startup. The directory must be owned by the HTTP$SERVER account and have full read+write+execute+delete access. It is suggested to name it [WASD_CACHE] and may be created manually using the following command.
It is a relatively simple matter to relocate the cache at any stage. Simply create the required directory in the new location, modify the startup procedures to reflect this, shut the server down completely then restart it using the procedures (not a /DO=RESTART!). The contents of the previous location could be transfered to the new using the BACKUP utility if desired.
It is required to define the logical name WASD_CACHE_ROOT if any proxy services are specified as using cache in the server configuration. The server will not start unless it is correctly defined. The logical should be a concealed device logical specifying the top level directory of the cache tree. The following example shows how to define such a logical name.
If example startup procedure is in use then it is quite straight-forward to have the logical created during server startup (see STARTUP.COM of WASD Configuration).
Caching may enabled on a per-service basis. This means it is possible to have a caching proxy service and a non-caching service active on the one server. Caching is enabled by appending the cache keyword to the particular service specification. The following example shows a non-proxy and a caching proxy service.
Proxy caching may be selectively disabled for a particular site, sites or paths within sites using the SET nocache mapping rule. This rule, used to disable caching for local requests, also disables proxy file caching for that subset of requests. This example shows a couple of variations.
As the proxy cache is implemented using the local file system, management of the cache implies controlling the number of, and exactly which files remain in cache. Essentially then, management means when and which to delete. The [ProxyReportLog] configuration parameter enables the server process log reporting of cache management activities.
Cache file deletion has three variants.
This ensures files that have not been accessed within specified limits are periodically and regularly deleted. The [ProxyCacheRoutineHourOfDay] configuration parameter controls this activity.
The ROUTINE form occurs once per day at the specified hour. The cache files are scanned looking for those that exceed the configuration parameter for maximum period since last access, which are then deleted (the largest number of [ProxyCachePurgeList], as described below).
Setting the [ProxyCacheRoutineHourOfDay] configuration parameter to 24 enables background purging.
In this mode the server continuously scans through the cache files in the same manner as for ROUTINE purging. The difference is it is not all done a single burst once a day, pushing disk activity to the maximum. The background purge regulates the period between each file access, pacing the scan so that the entire cache is passed through once a day. It adjusts this pace according the the size of the cache.
This is a remedial action, when cache device usage is reaching its configuration limit and files need to be deleted to free up space. The following parameters control this behaviour.
The cache device space usage is checked at the specified interval.
If the device reaches the specified percentage used a cache purge is initiated and by deleting files until the specified reduction is attained, the total space in use on the disk is reduced.
The cache files are scanned using the [ProxyCachePurgeList] parameter described below, working from the greatest to least number of hours in the steps provided. At each scan files not accessed within that period are deleted. At each few files deleted the device free space is checked as having reached the lower purge percentage limit, at which point the scan terminates.
This parameter has as its input a series of comma-separated integers representing a series of hours since files were last accessed. In this way the cache can be progressively reduced until percentage usage targets are realized. Such a parameter would be specified as follows,
Once the target reduction percentage is reached the purge stops. During the purge operation further cache files are not created. Even when cache files cannot be created for any reason proxy serving still continues transparently to the clients.
If [ProxyCacheRoutineHourOfDay] is empty or non-numeric the automatic, once-a-day routine purge of the cache by the server is disabled and it is expected to be performed via some other mechanism, such as a periodic batch job. This allows routine purging more or less frequently than is provided-for by server configuration, and/or the purge activity being performed by a process or cluster node other than that of the HTTPd server (reducing server and/or node impact of this highly I/O intensive activity). Progress and other messages are provided via SYS$OUTPUT, and if configured in the [Opcom…] directives to the operator log and designated operator terminal as well. If a process already has the cache locked the initiated activity aborts.
The following example shows a routine purge being performed from the command-line. This form uses the hours from [ProxyCachePurgeList].
A variant on this allows the maximum age to be explicitly specified.
Reactive purging and statistic scans may also be initiated from the command line. For a reactive purge the first number can be the device usage percentage (indicated by the trailing "%"), if not the configuration limit is used.
Any in-progress scan of the cache (i.e. reactive or routine purges, or a statistics scan) can be halted from the command line (and online Server Admininistration facility).
For the purposes of this document, cache invalidation is defined as the determination when a cache file's data is no longer valid and needs to be reloaded.
The method used for cache validation is deliberately quite simple in algorithm and implementation. In this first attempt at a proxy server the overriding criteria have been efficiency, simplicity of implementation, and reliability. Wishing to avoid complicated revalidation using behind-the-scenes HEAD requests the basic approach has been to just invalidate the cache item upon exiry of a period related to the "Last-Modified:" age or upon a no-cache request, both described further below.
The revision count (automatically updated by VMS) tracks the absolute number of accesses since the file was created (actually a maximum of 65535, or an unsigned short, but that should be enough for informational purposes).
The [ProxyCaheReloadList] configuration parameter is used to control when a file being accessed is reloaded from source.
This parameter supplies a series of integers representing the hours after which an access to a cache file causes the file to be invalidated and reloaded from the source during the proxied request. Each number in the series represents the lower boundary of the range between it and the next number of hours. A file with a last-loaded age falling within a range is reloaded at the lower boundary of that particular range. The following example
The HTTPDMON utility allows real-time monitoring of proxy serving activity (13.10 HTTPd Monitor).
Proxy reports and some administrative control may be exercised from the online Server Administration facility (9. Server Administration). The information reported includes:
The following actions can be initiated from this menu. Note that three of these relate to proxy file cache and so may take varying periods to complete, depending on the number of files. If the cache is particularly large the scan/purge may take some considerable time.
Also available from the Server Administration facility is a dialog allowing the proxy characteristics of the running server to be adjusted on an ad hoc basis. This only affects the executing server, to make changes to permanent configuration the WASD_CONFIG_GLOBAL configuration file must be changed.
This dialog can be used to modify the device free space percentages according to recent changes in device usage, alter the reload or purge hour list characteristics, etc. After making these changes a routine or reactive purge will automatically be initiated to reduce the space in use by the proxy cache if implied by the new settings.
It is often useful to be able to list the contents of the proxy cache directory or the characteristics or contents of a particular cache file. Cache files have a specific internal format and so require a tool capable of dealing with this. The WASD_ROOT:[SRC.UTILS]PCACHE.C program provides a versatile command-line utility as well as CGI(plus) script, making cache file information accessible from a browser. It also allows cache files to be selected by wildcard filtering on the basis of the contents of the associated URL or response header. For detailed information on the various command-line options and CGI query-string options see the description at the start of the source code file.
Make the WASD_EXE:PCACHE.EXE executable a foreign verb. It is then possible to
To make the PCACHE script available to the server ensure the following line exists in the HTTP$CONFIG configuration file in the [AddType] section.
The following rule needs to be in the WASD_CONFIG_MAP configuration file.
Once available the following is then possible.
If the configuration changes described above have been made the following link will return such an index.
The connect service provides firewall proxying for any connection-oriented TCP/IP access. Essentially it provides the ability to tunnel any other protocol via a Web proxy server. In the context of Web services it is most commonly used to provide firewall-transparent access for Secure Sockets Layer (SSL) transactions. It is a special case of the more general tunneling provided by WASD, see 7.6 Tunneling Using Proxy.
As with proxy serving in general, CONNECT serving may enabled on a per-service basis using the WASD_CONFIG_GLOBAL [service] directive, the WASD_CONFIG_SERVICE configuration file, or even the /SERVICE= qualifier.
The actual services providing the CONNECT access (i.e. the host and port) are specified on a per-service basis. This means it is possible to have CONNECT and non-CONNECT services deployed on the one server, as part of a general proxy service or standalone. CONNECT proxying is enabled by appending the connect keyword to the particular service specification. The following example shows a non-proxy and proxy services, with and without additional connect processing enabled.
The connect service poses a significant security dilemma when in use in a firewalled environment. Once a CONNECT service connection has been accepted and established it essentially acts as a relay to whatever data is passed through it. Therefore any transaction whatsoever can occur via the connect service, which in many environments may be considered undesirable.
In the context of the Web and the use of the connect service for proxying SSL transactions it may be well considered to restrict possible connections to the well-known SSL port, 443. This may be done using conditional directives, as in the following example:
WASD provides a proxy service for the FTP scheme (prototcol). This provides the facility to list directories on the remote FTP server, download and upload files.
The (probable) file system of the FTP server host is determined by examining the results of an FTP PWD command. If it returns a current working directory specification containing a "/" then it assumes it to be Unix(-like), if ":[" then VMS, if a "\" then DOS. (Some DOS-based FTP servers respond with a Unix-like "/" so a second level of file-system determination is undertaken with the first entry of the actual listing.) Anything else is unknown and reported as such. WASD (for the obvious reason) is particularly careful to perform well with FTP servers responding with VMS file specifications.
Note that the content-type of the transfer is determined by the way the proxy server interprets the FTP request path's "file" extension. This may or may not correspond with what the remote system might consider the file type to be. The default content-type for unknown file types is "application/octet-stream" (binary). When using the alt query string parameters then for any file in a listing the icon provides an alternate content-type. If the file link provides a text document then the icon will provide a binary file. If the link returns a binary file then the icon will return a file with a plain-text content-type.
In addition to content-type the FTP mode in which the file transfer occurs can be determined by either of two conditions. It the content-type is "text/.." then the transfer mode will be ASCII (i.e. record carriage-control adjusted between systems). If not text then the file is transfered in Image mode (i.e. a binary, opaque octet-stream). For any given content-type this default behaviour may be adjusted using the [AddType] directive (see Alphabetic Listing of WASD Configuration) or the "#!+" MIME.TYPES directive (see MIME.TYPES of WASD Configuration).
Rules required in WASD_CONFIG_MAP for mapping FTP proxy. This is preferably made against the virtual service providing the FTP proxy. The service explicitly must make the icon path used available or it must be available to the proxy service in some other part of the mappings. Also the general requirement for error message URLs applies to FTP proxying (‘Proxy Error Messages’ in 7. Proxy Services).
Keywords added to an FTP request query string allow the basic FTP action to be somewhat tailored. These case-insensitive keywords can be in the form of a query keys or query form fields and values. This allows considerable flexibility in how they are supplied, allowing easy use from a browser URL field or for inclusion as form fields.
| Keyword | Description |
|---|---|
| alt | Adds alternate access (complementary content-type at the icon) for directory listings. |
| ascii | Force the file transfer type to be done as ASCII (i.e. with carriage-control conversion between systems with different representations). |
| content | Explicitly specify the content type for the returned file (e.g. "content:text/plain", or "content=image/gif"). |
| dos | When generating a directory listing force the interpretation to be DOS. |
| Explicitly specify the anonymous access email address (e.g. "email:daniel@wasd.vsm.com.au" or "email=daniel@wasd.vsm.com.au"). | |
| image | Force the file transfer type to be done as an opaque binary stream of octets. |
| list | Displays the actual directory plain-text listing returned by the remote FTP server. Can be used for problem analysis. |
| login | Results in the server prompting for a username and password pair that are then used as the login credentials on the remote FTP server. |
| octet | Force the content-type of the file returned to be specified as "application/octet-stream". |
| text | Force the content-type of the file returned to be specified as "text/plain". |
| unix | When generating a directory listing force the interpretation to be Unix. |
| upload | Causes the server to return a simple file transfer form allowing the upload of a file from the local system to the remote FTP server. |
| vms | When generating a directory listing force the interpretation to be VMS. |
The usual mechanism for supplying the username and password for access to a non-anonymous proxied FTP server area is to place it as part of the request line (i.e. "ftp://username:password@the.host.name/path/"). This has the obvious disadvantage that it's there for all and sundry to see.
The "login" query string is provided to work around the more obvious of these issues, having the authentication credentials as part of the request URL. When this string is placed in the request query string the FTP proxy requests the browser to prompt for authentication (i.e. returns a 401 status). When request header authentication data is present it uses this as the remote FTP server username and password. Hence the remote username and password never need to appear in plain-text on screen or in server logs.
WASD is fully capable of mapping non-proxy into proxy requests, with various limitations on effectiveness considering the nature of what is being performed.
Gatewaying between request schemes (protocols)
and also gatewaying between IP versions
All can be useful for various reasons. One example might be where a script is required to obtain a resource from a secure server via SSL. The script can either be made SSL-aware, sometimes a not insignificant undertaking, or it can use standard HTTP to the proxy and have that access the required server via SSL. Another example might be accessing an internal HTTP resource from an external browser securely, with SSL being used from the browser to the proxy server, which the accesses the internal HTTP resource on its behalf.
The basic mechanism allowing this gatewaying is "internal" redirection. The redirect mapping rule (see REDIRECT Rule of WASD Configuration) either returns the new URL to the originating client (requiring it to reinitiate the request) or begins reprocessing the request internally (transparently to the client). It is this latter function that is obviously used for gatewaying.
The use of WASD proxy serving as a firewall component assumes two configured network interfaces on the system, one of which is connected to the internal network, the other to the external network. (Firewalling could also be accomplished using a single network interface with router blocking external access to all but the server system.) Outgoing (internal to external) proxying is the most common configuration, however a proxy server can also be used to provide controlled external access to selected internal resources. This is sometimes known as reverse proxy and is a specific example of WASD's general non-proxy to proxy request redirection capability (7.5 Gatewaying Using Proxy).
In this configuration the proxy server is contacted by an external browser with a standard HTTP request. Proxy server rules map this request onto a proxy-request format result. For example:
Note that the trailing question-mark is required to propagate any query string (see REDIRECT Rule of WASD Configuration).
The server recognises the result format and performs a proxy request to a system on the internal network. Note that the mappings required could become quite complex, but it is possible. See example 7 in 7.1.5 Controlling Proxy Serving.
If a reverse proxied server returns a redirection response (302) containing a "Location: url" field with the host component the same reverse-proxied-to server it can be rewritten to instead contain the proxy server host. If these do not match the rewrite does not occur. Using the redirection example above, the SET mapping rule proxy=reverse=location specifies the path that will be prefixed to the path component in the location field URL. Usually this would be the same path used to map the reverse proxy redirect (in this example "/sales/"), though could be any string (presumably detected and processed by some other part of the mapping).
If the proxy=reverse=location=<string> ends in an asterisk the entire 302 location field URL is appended (rather than just the path) resulting in something along the lines of
WASD can authorize reverse proxy requests locally (perhaps from the SYSUAF) and rewrite that username into the proxied requests "Authorization: …" field. The proxied-to server can then verify that the request originated from the proxy server and extract and use that username as authenticated.
This functionality is described in the WASD_ROOT:[SRC.HTTPD]PROXYVERIFY.C module.
This utility (CGIplus script) can be used to rewrite HTTP response "Location:" fields, "Set-Cookie:" path and domain components and URLs in HTML and CSS content.
This functionality is described in the prologue to the code WASD_ROOT:[SRC.UTILS]PROXYMUNGE.C
This looks a little like reverse proxy, providing access to a non-local resource via a standard (non-proxy) request. The difference allows the client to determine which remote resource is accessed. This works quite effectively for non-HTML resources (e.g. image, binary files, etc.) but non-self-referential links in HTML documents will generally be inaccessible to the client. This can provide provide scripts access to protocols they do not support, as with HTTP to FTP, HTTP to HTTP-over-SSL, etc.
Mappings appropriate to the protocols to be support must be made against the proxy service. Of course mapping rules may also be used to control whom or to what is connected.
The client may the provide the desired URL as the path of the request to the proxy service. Notice that the scheme provided in the desired URL can be any supported by the service and its mappings.
This relies on being able to manipulate host record in the DNS or local name resolution database. If a "*.the.proxy.host" DNS (CNAME) record is resolved it allows any host name ending in ".the.proxy.host" to be resolved to the corresponding IP address. Similarly (at least the Compaq TCP/IP Services) the local host database allows an alias like "another.host.name.proxy.host.name" for the proxy host name. Both of these would allow a browser to access "another.host.name.proxy.host.name" with it resolved to the proxy service. The request "Host:" field would contain "another.host.name.proxy.host.name".
Using this approach a fully functioning proxy may be implemented for the browser without actually configuring it for proxy access, where returned HTML documents contain links that are always correct with reference to the host used to request them. This allows the client an ad hoc proxy for selected requests. For a wildcard (CNAME) record the browser user may enter any host name prepended to the proxy service host name and port and have the request proxied to that host name. Entering the following URL into the browser location field
This wildcard DNS entry approach is a more fully functional analogue to common proxy behaviour but is slightly less flexible in providing gatewaying between protocols and does require more care in configuration. It also relies on the contents of the request "Host:" field to provide mapping information (which generally is not a problem with modern browsers). The mappings must be performed in two parts, the first to handle the wildcard DNS entry, the second is the fairly standard rule(s) providing access for proxy processing.
The obvious difference between this and one-shot proxy is the desired host name is provided as part of the URL host, not part of the request path. This allows the browser to correctly resolve HTML links etc. It is less flexible because a different proxy service needs to be provided for each protocol mapping. Therefore, to allow HTTP to HTTP-over-SSL proxy gatewaying another service and mapping would be required.
This proxy function allows standard HTTP clients to connect to Secure Sockets Layer (4. Transport Layer Security) services. This is very different to the CONNECT service (7.3 CONNECT Serving), allowing scripts and standard character-cell browsers supporting only HTTP to access secure services.
Standard username/password authentication is supported (as are all other standard HTTP request/response interactions). The use of X.509 client certificates (4.5.12 Authorization Using X.509 Certification) to establish outgoing identity is not currently supported.
Unlike HTTP and FTP proxy it requires the service to be specifically configured using the [ServiceClientSSL] directive.
There are a number of Secure Sockets Layer related service parameters that should also be considered (see Service Configuration of WASD Configuration). Although most have workable defaults unless [ServiceProxyClientSSLverifyCA] and [ServiceProxyClientSSLverifyCAfile] are specifically set the outgoing connection will be established without any checking of the remote server's certificate. This means the host's secure service could be considered unworthy of trust as the credentials have not been established.
WASD supports the CONNECT method which effectively allows tunneling of raw octets through the proxy server. This facility is most commonly used to allow secure SSL connections to be established with hosts on the 'other side' of the proxy server. This basic mechanism is also used by WASD to provide an extended range of tunneling services. The term raw is used here to indicate an 8 bit, bidirectional, asynchronous exchange of octets between two entities, as a protocol family, not necessarily as an application (but can be so). Global proxy serving must be enabled (7.1.1 Enabling A Proxy Service) and then each service must be configured and mapped according to the desired mode of tunneling. Disabling or setting timeouts appropriately on the mapped service is important if connections are not to be disrupted by general server timeouts on output and non-progress (quiescent connections).
A service with this configuration is used as a target for CONNECT proxying (usually SSL through a firewall). The client expects an HTTP success (200) response once the remote connection is established, and HTTP error response if there is a problem, and once established just relays RAW octets through the proxy server (classic CONNECT behaviour).
This configuration enables CONNECT processing and limits any connect to SSL tunneling (i.e. port 443 on the remote system).
This allows any raw octet client (e.g. telnet) to connect to the port and by mapping be tunnelled to another host and port to connect to its service (e.g. a telnet service). The usual HTTP responses associated with CONNECT processing are not provided.
Telnet is used in the example above but the principle equally applies to any protocol that uses a raw 8 bit, bidirectional, asynchronous exchange of octets. Another example might be an SMTP service (port 25).
Using a tunnel it is possible to put a TLS/SSL (https://) front-end service to an otherwise plaintext-only service (http://).
It is possible to have a raw tunnel establish itself through a proxy chain (7.1.4 Proxy Chaining) by transparently generating an intermediate CONNECT request to the up-stream proxy server. Note that not all CONNECT proxy will allow connection to just any specified port. For security reasons it it is quite common to restrict CONNECT to port 443.
Any error in connecting to the chained proxy, making the request, connecting to the destination, etc. (i.e. any error at all) is not reported. The network connection is just dropped. Use WATCH to establish the cause if necessary.
With this configuration a service expects that the first line of text from the client contains a host name (or IP address) and optional port (e.g. "the.host.name" or "the.host.name:23"). This allows a variable destination to be mapped. The usual HTTP responses associated with CONNECT processing are not provided.
The pass rules force the supplied domain name (and optional port) to be mapped to the telnet port (23). Of course the mapping rules could allow the supplied port to be mapped into the destination if desired.
As with [ServiceProxyTunnel] RAW it is possible to chain FIREWALL services to an up-stream proxy server. See ‘Chaining RAW’ in 7.6.2 [ServiceProxyTunnel] RAW.
Up to this point the tunnels have merely been through the proxy server. It is possible to establish and maintain ENCRYPTED TUNNELS between WASD servers. SSL is used for this purpose. This is slightly more complex as both ends of the tunnel need to be configured.
This arrangement may be used for any stream-oriented, network protocol between two WASD systems. As it uses standard CONNECT requests (over SSL) it MAY also be possible to be configured between WASD and non-WASD servers.
The following example is going to maintain an encrypted tunnel between WASD servers running on systems KLAATU and GORT. It is designed to allow a user on KLAATU to connect to a specified port using a telnet client, and have a telnet session created on GORT, tunnelled between the two systems via an SSL encrypted connection.
Source of tunnel:
Destination of tunnel:
When a client connects to the service provided by port 10023 on system KLAATU the connection is immediately processed using a pseudo CONNECT request header. The service on this port is a proxy allowed to initiate SSL connections (client SSL). This service is mapped to system GORT port 10443, an SSL service that allows the CONNECT method (tunneling). KLAATU's proxy initiates an SSL connection with GORT. When established and the CONNECT request from KLAATU is received, it is mapped via a raw tunnel (8 bit, etc.) to its own system port 23 (the telnet service). Telnet is in use at both ends while encrypted by SSL inbetween! Note the use of network addresses and general fail rules used to control access to this service, as well as the disabling of timers that might otherwise shutdown the tunnel.
This arrangement is essentially a variation on example 4. It provides a cryptographic authentication of the originator (source) of the tunnel.
Source of tunnel:
Destination of tunnel:
This works by configuring the destination service to insist on proxy authorization. The authorization realm is X509 which causes the destination to demand a certificate from the source (4.5.12 Authorization Using X.509 Certification). The fingerprint of this certificate is checked against the authorization rule before the connection is a allowed to procede.
The objective of this raw tunnel variant (see 7.6.2 [ServiceProxyTunnel] RAW) is to allow tunneling of Secure Shell (SSH) via a client site proxy server CONNECT which is usually confined to port 443. Of course most Web servers are configured to provide SSL HTTP on port 443. Sharing of HTTP and SSH on the same port is a little problematic and involves some protocol detection. The following explanation of how it is implemented is so that the reader can understand the requirement for the "timeout quirk".
On configured services; WASD peeks at the incoming TCP byte stream to see if it's SSH protocol. If it is, the socket is associated with a proxy raw tunneling service and proxy tunneling initiated to a mapped SSH server. However (just to make it interesting) some SSH clients do not initiate their own exchange until after the SSH server, and so peeking only works for a subset of clients. Of course this is a Catch-22 of sorts! To provide for these clients; if an input timeout should occur (an SSH client waiting) WASD sets up the tunnel anyway and begins the proxy. The proxied SSH server should then initiate the protocol and the client respond. The directive [ServiceShareSSH] configured to be non-zero both enables this facility for a service and sets the input timeout period (which perhaps should be shorter than the default 30 seconds because such clients will wait that long for any SSH server response).
This approach seems to work well-enough in practice, although users need to be aware that some clients will pause (for the duration of the timeout period – the "timeout quirk") during initial connection setup.
This example shows an SSL service, the desired SSH service (which can be local or remote) and the internal proxy service that will provide the connection.
When creating raw tunnels between WASD servers, and possibly in other circumstances, it is often useful to be able to signal tunnel purpose to the remote end. In this way a single destination port can support multiple tunneling purposes simply through mapping rules. An originating end can inject an HTTP request line, or full request, into the established tunnel connection, which can then be processed by the usual WASD request mapping, and from that alternate services provided based on the intent signalled by the originating end.
This somewhat complex but instructive example illustrates the potential utility and versatility of WASD tunneling. It involves an originating WASD server, a destination (service providing) WASD server, and just to make it interesting an intermediate chained HTTP proxy server (not WASD). The idea is to provide access to various application services not necessarily supported by intermediate HTTP proxies and/or gateways. Four services will be supported by the example; SSH, NNTP IMAP and SMTP.
These are the services assigned on the WASD server on the inside of the proxy/gateway. Note that there is one per application to be tunneled. For simplicity each service port number has been selected to parallel the well-known application port number. Note that proxy is enabled on each (allowing them to initiate outgoing connections) and each has SSL enabled (further allowing them to initiate encrypted connections).
Each client application (i.e. IMAP, SSH) must be configured to connect to its corresponding service port (e.g. IMAP to 8143, SMTP to 8025).
These mappings are made on the WASD server on the inside of the proxy/gateway. The rules essentially initiate an outgoing encrypted (SSL) connection to the host wasd.external.net supporting the external WASD proxy server. Each is also configured not to connect directly but to request the chained proxy server proxy.internal.net to establish the connection on their behalf.
If the up-stream proxy server successfully connects to wasd.external.net port 443 the proxy server allows the byte-stream to be asynchonously and bidirectionally exchanged with the internal WASD server outgoing connection. This internal WASD server has initiated an SSL connection and the external server port 443 expects SSL so they can now both negotiate an SSL-encrypted channel essentially directly with each other.
The external WASD service configuration is very simple, a single SSL port.
Connections to the 443 port are expected to undertake an SSL negotiation to establish an encrypted channel. This includes incoming tunnel connections. The service on port 1234 is required to support the connections outgoing from the external WASD server to the application server ports.
These mappings are all applied to requests at port 443 on the external WASD server wasd.external.net. Each rule checks three request characterstics. First, the request method, "CONNECT". Second, the request URI, varies according to the request. These are the request data injected by the internal WASD server wasd.internal.net using the set=proxy=tunnel=request= mapping rule on the outgoing connection. Third, the originating host (proxy.external.net) address adds an extra filter on from where this facility may be used. The respective pass of the matching rule then initiates an outgoing connection to the respective application server's well-known port. A timeout is applied to limit connection times.
Now let's look at an actual example usage. Consider the internal user's IMAP application, say Thunderbird, is configured to use an IMAP server at host wasd.internal.net port 8143. The internal user activates Thunderbird which then intiates an TCP/IP connection to the configured IMAP server expecting to commence the IMAP application protocol.
This connection arrives at wasd.internal.net port 8143 which has a WASD raw tunnel service listening. The connection is accepted and request processing commences. Mapping rules applied to port 8143 initiate an SSL connection to host wasd.external.net which is not directly accessable because of the firewall and must be connected to using the HTTP proxy server proxy.internal.net as an intermediary. This is specified in the same mapping rule. The mapping rule also injects an HTTP request header providing request characteristics that can be identified and acted upon by the external server.
The internal WASD server initiates a connection to the proxy server proxy.internal.net acting as part of the firewall. As it is endeavouring to initiate an SSL connection with the external wasd.external.net host this proxy connection uses a CONNECT request specifying wasd.external.net port 443. The proxy server establishes a connection with the host wasd.external.net at port 443. Once the connection is established it becomes an asynchronous, bidirectional channel between wasd.internal.net and wasd.external.net with the proxy server as a conduit.
The service connection just established is expecting an SSL negotiation in an attempt to establish an encrypted channel. When this negotiation concludes successfully the communications between wasd.internal.net and wasd.external.net become opaque to all external listeners including proxy.internal.net.
The encrypted connection now established, the request begins to be processed by the WASD server at wasd.external.net. A number of mapping rules apply to port 443. Each rule compares the injected request method and URI until, in this case, the external-imap rule matches. This rule specifies that a raw connection be established with the host imap.isp.net at port 143 using the proxy-capable port 1234 service. A timeout limits the duration this connection can be held unused.
The IMAP application server at imap.isp.external port 143 accepts the connection at begins to communicate using the IMAP protocol.
There is now a raw (8 bit, asynchronous, bidirectional) connection from the Thunderbird client to wasd.internal.net, (encrypted) through to proxy.internal.net, (encrypted) through to wasd.external.net, and raw to the IMAP server at imap.isp.net. This raw connection will be used for communication between Thunderbird and the IMAP server using the IMAP application protocol.
When a tunnel is established into a system the source of that connection (IP host-name/address and port) becomes obscured. By setting the path to the destination port proxy=forwarded=for (host name) or proxy=forwarded=address (IP address) the external client can be obtained using data contained in the logical name WASD_TUNNEL.
Consider tunneling external port 22345 to internal port 22 - Secure Shell.
To Secure Shell the source host and port would be localhost and some random port. It can be useful for the login procedure or other service to have the actual client host name (or IP address). Adding the path setting.
Obtaining the SSH source port, say from TT_ACCPORNAM data, the original client host and port can be searched for with some trivial DCL code. Adapt to suit local requirements.
The tunnel data remains current for at least one minute and may become unavailable at any time after that.
The browser needs to be configured to access URLs via the proxy server. This is done using two basic approaches, manual and automatic.
Most browsers allow the configuration for access via a proxy server. This commonly consists of an entry for each of the common Web protocol schemes ("http:", "ftp:", "gopher:", etc.). Supply the configured WASD proxy service host name and port for the HTTP scheme. This is currently the only one available. This would be similar to the following example:
To exclude local hosts, and other servers that do not require proxy access, there is usually a field that allows a list of hosts and/or domain names for which the browser should not use proxy access. This might be something like:
A proxy auto-config (PAC) file defines how web browsers and other user agents can automatically choose the appropriate proxy server (access method) for fetching a given URL.
https://en.wikipedia.org/wiki/Proxy_auto-config
The following is a very simple proxy configuration JavaScript function. This specifies that all URL host names that aren't full qualified, or that are in the "example.com" domain will be connected to directly, with all other being accessed via the specified proxy server.
This JavaScript is contained in a file with a specific, associated MIME file type, "application/x-ns-proxy-autoconfig". For WASD it is recommended the file be placed in WASD_ROOT:[LOCAL] and have a file extension of .PAC (which follows Netscape naming convention).
The following WASD_CONFIG_GLOBAL directive would map the file extension to the required MIME type:
This file is commonly made the default document available from the proxy service. The following example shows the HTTP$MAP rules required to do this:
All that remains is to provide the browser with the location from which load this automatic proxy configuration file. In the case of the above set-up this would be:
A template for a proxy auto-configuration file may be found at WASD_ROOT:[EXAMPLE]PROXY_AUTOCONFIG.TXT
| 8.1Server Instances |
| 8.1.1VMS Clustering Comparison |
| 8.1.2Considerations |
| 8.1.3Configuration |
| 8.1.4Status |
| 8.2Server Environments |
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |
WASD instances and environments are two distinct mechanisms for supporting multiple WASD server processes on a single system.
Server instances are multiple, cooperating server processes providing the same set of configured resources.
Server environments are multiple, independent server processes providing differently configured resources.
The term instance is used by WASD to describe an autonomous server process. WASD will support multiple server processes running on a single system, alone or in combination with multiple server processes running across a cluster. This is not the same as supporting multiple virtual servers (see Virtual Services of WASD Configuration). When multiple instances are configured on a single system they cooperate to distribute the request load between themselves and share certain essential resources such as accounting and authorization information.
The approach WASD has used in providing multiple instance serving may be compared in many ways to VMS clustering.
A cluster is often described as a loosely-coupled, distributed operating environment where autonomous processors can join, process and leave (even fail) independently, participating in a single management domain and communicating with one another for the purposes of resource sharing and high availability.
Similarly WASD instances run in autonomous, detached processes (across one or more systems in a cluster) using a common configuration and management interface, aware of the presence and activity of other instances (via the Distributed Lock Manager and shared memory), sharing processing load and providing rolling restart and automatic "fail-through" as required.
On a multi-CPU system there are performance advantages to having processing available for scheduling on each. WASD employs AST (I/O) based processing and was not originally designed to support VMS kernel threading. Benchmarking has shown this to be quite fast and efficient even when compared to a kernel-threaded server (OSU) across 2 CPUs. The advantage of multiple CPUs for a single multi-threaded server also diminishes where a site frequently activates scripts for processing. These of course (potentially) require a CPU each for processing. Where a system has many CPUs (and to a lesser extent with only two and few script activations) WASD's single-process, AST-driven design would scale more poorly. Running multiple WASD instances addresses this.
Of course load sharing is not the only advantage to multiple instances …
When multiple WASD instances are executing on a node and a restart is initiated only one process shuts down at a time. Others remain available for requests until the one restarting is again fully ready to process them itself, at which point the next commences restart. This has been termed a rolling restart. Such behaviour allows server reconfiguration on a busy site without even a small loss of availability.
When multiple instances are executing on a node and one of these exits for some reason (resource exhaustion, bugcheck, etc.) the other(s) will continue to process requests. Of course requests in-progress by the particular instance at the time of instance failure are disconnected (this contrasts with the rolling restart behaviour described above). If the former process has actually exited (in contrast to just the image) a new server process will automatically be created after a few seconds.
The term fail-through is used rather than failover because one server does not commence processing as another ceases. All servers are constantly active with those remaining immediately and automatically taking all requests in the absence any one (or more) of them.
Of course "there is no such thing as a free lunch" and supporting multiple instances is no exception to this rule. To coordinate activity between and access to shared resources, multiple instances use low-level mutexes and the VMS Distributed Lock Manager (DLM). This does add some system overhead and a little latency to request processing, however as the benchmarks indicate increases in overall request throughput on a multi-CPU system easily offset these costs. On single CPU systems the advantages of rolling restart and fail-through need to be assessed against the small cost on a per-site basis. It is to be expected many low activity sites will not require multiple instances to be active at all.
When managing multiple instances on a single node it is important to consider each process will receive a request in round-robin distribution and that this needs to be considered when debugging scripts, using the Server Administration page and the likes of WATCH, etc. (see 8.1 Server Instances).
If not explicitly configured only one instance is created. The configuration directive [InstanceMax] allows multiple instances to be specified Global Configuration of WASD Configuration). When this is set to an integer that many instances are created and maintained. If set to "CPU" then one instance per system CPU is created. If set to "CPU-integer" then one instance for all but one CPU is created, etc. The current limit on instances is eight, although this is somewhat arbitrary. As with all requests, Server Administration page access is automatically shared between instances. There are occasions when consistent access to a single instance is desirable. This is provided via an admin service (see Service Configuration of WASD Configuration).
When executing, the server process name appends the instance number to the "WASD". Associated scripting processes are named accordingly. This example shows such a system:
The instance management infrastructure distributes basic status data to all instances on the node and/or cluster. The intent is to provide an easily comprehended snapshot of multi-instance/multi-node WASD processing status. The data comprises:
The data are constrained to these items due to the need to accomodate it within a 64 byte lock value block for cluster purposes. Single node environments do not utilise the DLM, each instance updating its table entry directly.
Each node has a table with an entry for every other instance in that WASD environment. Instance data are updated once every minute so any instance with data older than one minute is no longer behaving correctly. This could be due to some internal error, or that the instance no longer exists (e.g. been stopped, exited or otherwise no longer executing). An entry for an instance that no longer exists is retained indefinitely, or until a /DO=STATUS=PURGE is performed removing all such expired entries, or a /DO=STATUS=RESET removing all entries (and allowing those currently executing to repopulate the instance data over the next minute.
These status data are accessible via command-line and in-browser reports, intended for larger WASD installations, primarily those operating across multiple nodes in a cluster. With the data being stored in a common, another of those other nodes can provide a per-cluster history even if one or more nodes become completely non-operational.
This is an example report on a 132 column terminal display. Due to screen width constraints the date/time omits the year field of the date.
This provides an example CLI report showing a single node, where a single instance has been started, changed to a three instance configuration, restarted so that the three instances have begun processing. The configuration has been returned a single instance and then the existing three instances restarted the previous day, resulting in the original single instance returning to processing. That instance was last (re)started some 54 minutes ago (a normal exit status showing) and its status was last updated some 41 seconds ago. Note that the three instances showing white-space struck-through with hyphens are stale, having last been updated 1 day ago. Entries older than three minutes are displayed in this format to differentiate them from current entries.
The same report on an 80 column terminal. Note that the overt date/time has been omitted, leaving only the period ago the event happened.
Where multiple instances exist, or have existed, and the terminal page size is greater than 24 lines, HTTPMON displays an equivalent of the 80 column report at the bottom of the display.
Similarly, the Server Admin report (9. Server Administration) shows an HTML equivalent of the 80 column report immediately below the control and time panels.
WASD server environments allow multiple, distinctly configured environments to execute on a single system. Generally, WASD's unlimited virtual servers and multiple account scripting eliminates the need for multiple execution environments to kludge these requirements. However there may be circumstances that make this desirable; regression and forward-compatibility testing comes to mind.
See Server Environments in WASD Installation for deltained information on maintaining multiple installations of WASD.
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |
The online Server Administration facility provides a rich collection of
functionality, including server control, reports and configuration. Some of
these are intended as general administration tools while other provide more
detailed information intended for server debugging and development purposes.
The value of the WATCH facility 10. WATCH Facility as a general configuration and problem-solving tool cannot be overstated.
All server configuration files, with the exception of the authentication databases, are plain text and may be modified with any prefered editor. However the majority of these can also be administered online through a browser. In addition the update facility allows some administration of file system portions of the Web. See 12. HTTPd Web Update.
Access to many portions of the package is constrained by file protections and directory listing access files. See for a method for circumventing these restrictions.
It is often a significant advantage for the inexperienced administrator on a new and largely unconfigured installation to be able to gain access to the facilities offered by Server Administration, particularly the WATCH facility (10. WATCH Facility). This can be done quite simply by using the authentication skeleton-key (3.12 Skeleton-Key Authentication). This allows the site administrator to register a username and password from the command-line that can be used to gain access to the server. In addition, the server ensures that requesting an otherwise non-authorized Server Administration facility generates a challenge which invokes a username/password dialog at the browser allowing the user to enter the previously registered username and password and gain access.
Note that the username must begin with an underscore, be at least 6 characters, is delimited by a colon, and that the password must be at least 8 characters. By default this username and password remains valid for 60 minutes. Choose strings that are less-than-obvious!
Once established the site should make the Server Administration facility a configured facility of the site. The value of its facilities cannot be overstated.
It is also recommended that for production sites the path to these reports be controlled via authentication and authorization, using both host and username restrictions, similar to the following:
If a full authorization environment is not required but administration via browser is still desired restrict access to browsers executing on the server system itself, using an appropriate SYSUAF-authenticated username. Provision of a VMS account for server administration only is quite feasable, see 3.10.6 Nil-Access VMS Accounts.
If SSL is in use (4. Transport Layer Security) then username/password privacy is inherently secured via the encrypted communications. To restrict server administration functions to this secure environment add the following to the WASD_CONFIG_MAP configuration file:
When using the revise capability of the Server Administration facility it is necessary to comply with all the requirements for Web update of files. This is discussed in general terms in 12. HTTPd Web Update. Revision of server configuration files requires path permissions allowing write access for the username(s) doing the administration, as well as the required ACL on the target directory (in the following example WASD_ROOT:[LOCAL]).
It is possible to allow general access to the Server Administration facility and reports while restricting the ability to initiate server actions such as a restart! Using the WORLD realm against the path is necessary, for the obvious security reason, the server administration module will not allow itself to be used without an authenticated username, provided as a pseudo-authenticated "WORLD".
When GZIP compression is configured for the server (see GZIP Encoding of WASD Configuration) it is not by default applied to Server Admin reports or other pages. It can be applied, selectively if desired, using mapping rules. For instance, to apply it to all requests not from the local intranet a rule similar to the following can be added before the Server Admin path mapping itself.
GZIP content-encoding can never be applied to WATCH reports.
With a single instance (see 8.1 Server Instances) access to Server Administration reports, etc. is always serviced by the one server process. If multiple instances are configured then in common with all requests administration requests will be serviced by any one of the associated processes depending on the momentary state of the round-robin distribution.
There are many circumstances where it is preferable to access only the one server. This can be accomplished for two differing objectives.
The latter approach is particularly useful when performing detailed WATCH activities (10. WATCH Facility).
When multiple per-node instances are executing the Server Administration pages and reports all include an indication of which process serviced the request. When accessing no instance in particular the process name is presented in parentheses after the page title
Multi-instance status (see 8.1.4 Status) snapshots are available via HTTPDMON, the Server Admin main page and can be reported from the command line using
The server provides a number of internally generated reports. Some of these are of general interest. Others are more for evaluating WASD behaviour and performance for development purposes. Appropriate reports have a refresh selector allowing the report to be updated at the selected period. The following list is in the approximate order in which they occur top-to-bottom, left-to-right in the menu layout.
It is possible to use this facility standalone, without configuring authorisation (9.1 Access Before Configuration).
DCL module statistics (same information as displayed in the server statistics report). These are cumulative for the entire life of the system (unless zeroed).
Process information shows how many actual processes exist at the time of the report, as indicated by the PID and bolded, non-zero liftime (in minutes). The soft-limit specifies how many CGIplus scripts are allowed to continue existing before the least used is deleted and the hard-limit show how many processes may actually exist at any one time (the margin allows for process deletion latency). A count of how many times the CGIplus processes have been explicitly purged (button available on this report page). The life-time of zombie processes (in minutes, zero implying use of zombies is disabled) and the number that have been purged due to expiry. CGIplus process life-time (in minutes, zero implying indefinite), the number purged due to life-time expiry and the number of CGIplus processes that the server has actually purged (deleted) to maintain the soft-limit margin specified above.
Each of the allocated process data structures is listed. There may be zero up to hard-limit items listed here depending on demand for DCL activities and the life of the server. Items with a PID shown indicate an actual process existing. This can be a zombie process or a CGIplus process. If no process is indicated then the other information represents the state the last time the item's associated process completed. Information includes the script (URL-style path) or DCL command, total count of times the item has been used and the last time it was. The zombie count indicates the number of time the same process finished a request and entered the zombie state. The CGIplus column indicates it is/was a CGIplus script and shows the total number of times that particular script has been/was used. If the process is currently in use the client information show the client host name.
If any processes are associated with any data structure a purge button is provided that forces all processes to be deleted. This can be useful if a new script image is compiled and it is required all scripts now use this. If a script is currently processing a request the process deletion occurs when that processing is complete. The purge button does not force a process to delete, so a second button forces all processes to delete immediately. This can be used to forceably clear errant scripts, etc., but be warned script processing is indiscrimately stopped!
This list will grow, up to the specified configuration maximum, as conconurrent scripting demand occurs. Maintained connections are indicated by the bolded, non-zero lifetime (in minutes). When this reaches zero the task is disconnected. The current/last task for that connection is indicated, along with the number of times the connection was reused and a total number of uses for that list item.
Purge and force buttons allow current links to be broken after request completion or forcibly disconnected.
Two other diagnostic tools are available from the same link. The first, WATCH-peek Report, providing a snapshot of the contents selected internal fields and data structures of the request. This is primarily intended as a problem investiagtion and development tool, and will be of limited value without an understanding of server internals. The second accesses the "peek" internals plus a one-shot WATCH-processing report.
For servers handling a great quantity of concurrent traffic this can generate a very large report. The Supervisor report can also provide a profile of the servers current load.
$ HTTPD /SYSPLUS can provide the same report data at the command-line for circumstances where the server is unresponsive but an interactive session is available. Requires a 132 character width terminal session. The /SYSPLUS report generator may be used with /OUTPUT=<filename> to capture and store report data. See Server Image Command-Line Parameters of WASD Configuration).
The statistics are stored in a permanent global section and so carry-over between server restarts. Where multiple instances are executing the data represents an accumulation of all instances' processing. It is enabled by the configuration parameter [ActivityDays]. The Server Administration facility provides several, represented as a period of hours before the present time. Number of requests and bytes sent to the client are represented by a histogram with respective means for each by a line graph. A bar across the column of the request histogram indicates the peak number of concurrent requests during the period. A greyed area indicates no data available for that time (i.e. before the latest server startup, or in the future).
Server startup and shutdown events are indicated by solid, vertical lines the full height of the graph (see example for a restart event).
Activity data is accumulated on a per-minute basis. This is the maximum granularity of any report. When reports are selected that can display less than this one minute granularity (i.e. with periods greater than four hours) the value shown is the peak of the number of minutes sampled for display. This better represents the load on the server than would a mean of those samples.
The graph is an image map, various regions of which allow the selection of other reports with different periods or durations. This allows previous periods to be examined at various levels of detail using the graph for navigation. Various sections may have no mapping as appropriate to the current report.
For multiple hour reports the upper and lower sections have distinct functions. The middle 50% of the upper section allows the same end time (most commonly the current hour) to be examined over twice the current period, in this case it would be over eight hours. The left 25% allows the previous fours hours to be viewed (if such data exists), and for non-current reports the right 25% allows the next four hours to be viewed. The lower half can be divided into sections representing hours or days depending on the period of the current report. This allows that period to be viewed in greater detail. For single hour reports this section, of course, is not mapped.
Remember that the URL of the mapped section will be displayed in the status
bar of the browser. As the URL contains time components it is not a difficult
task to decipher the URL displayed to see the exact time and period being
selected.
The server provides a comprehensive configuration revision facility.
3. Authentication and Authorization covers authentication detail.
The file name and/or location may be specified using the logical name WASD_SITELOG.
Many of the server activites listed above require server account write access to the directory in which the configuration files are stored. Where an autononmous scripting account is in use this poses minimal threat to server configuration integrity.
If a site is using SYSUAF authentication and security profiles enabled using the /PROFILE startup qualifier (13.13 SECHAN Utility) then a more restrictive set up is possible, retaining the default no-access to the [LOCAL] directory. This relies on the administering account(s) having read and write access to the [LOCAL] directory. It is then not necessary to grant that to the server account. It is possible to limit the application of VMS user profiles. This is an example.
To use this approach perform steps 1, 2 and 4 from above, substituting the following for step 3.
The server allows certain run-time actions to be initiated. Many of these functions can also be initiated from the command line, see 9.7 HTTPd Command Line.
When multiple servers are executing on a single node or within a cluster a JavaScript-driven checkbox appears in the bottom left of the administration menu. Checking that box applies any subsequently selected action to all servers!
Caution! If changing CGIplus script mapping it is advised to restart the server rather than reload. Some conflict is possible when using new rules while existing CGIplus scripts are executing.
A foreign command for the HTTPD control functionality will need to be assigned in the adminstration users' LOGIN.COM, for example:
Some control of the executing server is available from the DCL command line on the system on which it is executing. This functionality, via the /DO= qualifier, is available to the privileged user.
These directives are communicated from the command-line (and Server Administration page analogue - ‘Control Section’ in 9.6 HTTPd Server Action) to the per-node or per-cluster servers using the Distributed Lock Manager. On pre-VMS V8.2 the command buffer is limited to 15 bytes. From VMS V8.2 the buffer space available is 63 bytes. In a cluster all systems must support the larger buffer before WASD enables it. The smaller buffer space limits some of the directives that take free-form parameters (e.g. /DO=DCL=PURGE=USER=DANIEL).
If multiple servers are executing on a host or cluster it is possible to control all of them by adding the /CLUSTER or /ALL qualifiers. Of course, these commands are available from batch jobs as well as interactively. In a clustered WASD environment the same functionality is available via checkboxes from the online Server Administration facility.
Can't quite remember what it can (and by implication can't) do?
Significant server events (e.g. restart, exit, mapping rule change) can often benefit (post-mortem :-) from an annotation in the server process log, especially in a production environment. The command-line /NOTE="<string>" can be used to insert the supplied string as an ad hoc annotation, or in conjunction with a /DO=".." CLI command.
The server process log annotation appear as follows.
Note may also be inserted from the Server Admin main page by using the [/DO=] button and field and prefixing the string with /NOTE= (string delimitting quotation marks are not required). Using the Server Admin page annotation and commands cannot be combined.
Server counters may be zeroed. These counters are those visible from the statistics Server Admininstration item and when using the HTTPDMON utility.
The HTTPDMON utility displays a status line during startup or server exit on error. For example:
On occasion this can status message become constantly displayed (e.g. command-line misoperation) with
On Alpha and Itanium platforms alignment faults can be a significant performance issue and considerable effort has been invested in completely eliminating them. This was done using a internal reporting tool (primarily intended for the WASD developer) available from the Server Admin interface. Defining the logical name WASD_ALIGN_MAP to be a linker map of the build provides additional information.
See 3. Authentication and Authorization.
The authorization rule file (HTTP$AUTH) may be reloaded using either of these variants.
The authentication cache may be purged, resulting in re-authentication for all subsequent authorization-controlled accesses. This may be useful when disabling authorization or if a user has been locked-out due to too many invalid password attempts (3.9 Authorization Cache).
A "skeleton-key" username and password may be entered, amongst things allowing access to the Server Administration facility (9. Server Administration).
Server cache control may also be exercised from the Server Administration page (9. Server Administration). The file cache (see Cache Configuration of WASD Configuration) may be enabled, disabled and have the contents purged (declared invalid and reloaded) using
Changes to configuration files can be validated at the command-line before reload or restart. This detects and reports any syntactical and fatal configuration errors but of course cannot check the intent of the rules.
The config check sequentially processes each of the authorization, global, mapping, message and service configuration files.
If additional server startup qualifiers are required to enable specific configuration features then these must also be provided when checking. For example:
These commands can be useful for flushing any currently executing CGIplus applications from the server, enabling a new version to be loaded with the next access. See "Scripting Environment" document.
All scripting processes, busy with a request or not, can be deleted (this may cause the client to lose data).
A gentler alternative is to delete idle processes and mark busy ones for deletion when completed processing.
For VMS V8.2 and later, a more selective DELETE and PURGE is possible. A user name, script name, or script file name can be supplied and only matching tasks have the specified action peformed.
All DECnet connections, busy with a request or not, can be disconnected (this may cause the client to lose data).
Purging is a better alternative, disconnecting idle tasks and marking busy ones for disconnection when complete.
Disconnect idle HTTP/2 connections.
All HTTP/2 connections can be disconnected (this may cause clients to lose data), or a specific connection number.
The number of server instances (see 8.1 Server Instances) may be set from the command line. This overrides any configuration file directive and applies at the next startup. Any configuration directive value may be used from the command line.
Note that the server must be restarted for this to take effect, that this can be applied to the current node only or to all servers within a cluster, and that it remains in effect until explicitly changed to "MAX" allowing the WASD_CONFIG_GLOBAL configuration directive [InstanceMax] to once again determine the number of instances required. The same functionality is available from the Server Administration page (9.6 HTTPd Server Action).
There are also directives to assist with WATCH activities (8.1 Server Instances).
Multi-instance (see 8.1 Server Instances) status (see 8.1.4 Status) can be reported from the command line using
In addition, stale entries in the status table may be purged using
Server logging control may also be exercised from the server administration menu (9. Server Administration).
Open the access log file(s).
Close the access log file(s).
Close then reopen the access log file(s).
Unwritten log records may be flushed to the file(s).
See Request Processing Configuration of WASD Configuration.
The mapping rule file (WASD_CONFIG_MAP) may be reloaded using either of these variants.
Disconnect idle (persistent) connections.
All network connections can be disconnected (this may cause clients to lose data), a specific connection number and those matching the specified URI.
Additionally, network connection acceptance can be suspended (leaving in-progress requests to complete), suspended and in-progress disconnected, and resumed.
Server shutdown may also be exercised from the Server Administration page (9. Server Administration).
The server may be shut down, without loss of existing client requests. Connection acceptance is stopped and any existing requests continue to be processed until conclusion.
The server may be immediately and unconditionally shut down.
The server may be restarted, without loss of existing client requests. Connection acceptance is stopped and any existing requests continue to be processed until conclusion. This effectively causes the server to exit normally and the DCL wrapper procedure to restart it.
The now variant restarts the server immediately regardless of existing connections.
The when– quiet variant restarts the server whenever request processing drops to zero for more than one second. It allows (perhaps non-urgent) changes to be put into effect through restart when everything has gone "quiet" and no demands are being placed on the server.
Significant server events such as these are prime candidates for server log annotation!
If the optional SSL component is installed and configured these directives become effective.
If X.509 authentication is enabled the Certificate Authority (CA) verification list can be reloaded.
Server certificates, after being updated, may be reloaded into the running services (i.e. without restart). This is a synonym for /DO=SERVICE=LOAD.
If a private key password is not included with the encode key it is requested by the server during startup. The following example shows the directive and the resulting prompt. When entered the password is not echoed.
Unconditionally release all queued requests for immediate processing.
Unconditionally terminate all requests queued waiting for processing. Clients receive a 503 "server too busy" response.
For VMS V8.2 and later, a more selective RELEASE and TERMINATE is possible. A user name or script name can be supplied and only matching requests have the specified action peformed.
Unconditionally disconnects all WebSocket applications.
For VMS V8.2 and later, more selective disconnects are possible. Disconnects WebSocket applications with connection number, with matching script names, and with matching scripting account usernames, respectively.
| 10.1Server Instances |
| 10.2Event Categories |
| 10.3Request Filtering |
| 10.4Report Format |
| 10.5Usage Suggestions |
| 10.6Command-Line Use |
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |
The WATCH facility is a powerful adjunct in server administration. From the Server Administration facility (9. Server Administration) it provides an online, real-time, in-browser-window view of request processing in the running server. The ability to observe live request processing on an ad hoc basis, without changing server configuration or shutting-down/restarting the server process, makes this facility a great configuration and problem resolution tool. It allows (amongst other uses)
A single client per server process can access the WATCH facility at any one time. It can be used in one of two modes.
Options immediately below the duration selector allows the WATCH output to concurrently be included in the server process log. This allows a permanent record (at least as permanent as server logs) to be simply produced.
With a single instance (see 8.1 Server Instances) access to WATCH is always through the one server process. If multiple instances are configured WATCH requests, in common with all others, will be serviced by any one of the associated processes depending on the momentary state of the round-robin distribution.
This is often an issue for request WATCHing. The simplest scenario involves two instances. When the WATCH report is activated it will be serviced by the first process, when the request wishing to be WATCHed is accessed it (in the absence of any other server activity) will be serviced by the other process and will not be reported by WATCH on the first.
The solution is to suspend the round-robin request processing for the period of the WATCH activity. This does not shut any instance down but instead makes all but the supervisor instance quiescent. (Technically, it dequeues all the listening I/Os from non-supervisor instance server sockets, making the TCP/IP network driver send all connection requests to the one instance left with listening I/Os.) It is just a matter of making the non-supervisor instances active again when the WATCH activity is concluded.
This may be done from the command-line using
An event is considered any significant point for which the server code
has a reporting call provided. These have been selected to provide maximum
information with minimum clutter and impact on server performance. Obvious
examples are connection acceptance and closure, request path resolution, error
report generation, network reads and writes, etc. Events are collected together
into groupings to allow clearly defined areas of interest to be selected for
reporting.
The report menu provides for the inclusion of any combination of the following categories.
If the server has been compiled using the WATCH_MOD=1 macro a set of module WATCHing statements is included. These provide far more detailed processing information than available with the generic WATCH, are intended primarily for debugging the server during development and testing. This is considered a specialized tool, with the quantity and level of detail produced most likely proving counter-productive in addressing general site configuration issues. The module items are shown below the usual WATCH items.
By default all requests to all services are WATCHed. Fine control may be exercised over exactly which requests are reported, allowing only a selected portion of all requests being processed to be concentrated on, even on a live and busy server. This is done by filtering requests according the following criteria.
In addition there are in and out selectors against each of the filters which include or exclude the particular request based on it matching the filter.
These filters are controlled using fully-specified, wildcarded strings or using regular expression patterns (see Request processing Configuration of WASD Configuration). In common with all WASD processing, filter matching is case-insensitive. Of course, due to the point of application of a particular filter during request processing, some information may or may not be displayed. When a request is into or out of the report because of a matching filter a FILTER informational item is reported.
These are the equivalent regular expressions but also will stop comparing at the end of the initial request line. The second, in this case, will also only filter against HTTP/1.1 version requests (note the final period matching the <CR> of the <CR><LF> carriage control).
This example uses a regular expression to constrain the match to a single header field (line, or newline-delimited string), matching all requests where the user agent reports using the "Gecko" browser component (Mozilla, Firefox, etc.)
Authenticated requests for user DANIEL.
All authenticated requests.
The following example illustrates the format of the WATCH report. It begins with multi-line heading. The first two record the date, time and official server name, with underline. The third provides the WASD server version. The fourth provides some TCP/IP agent information. Lines following can show OpenSSL version (if deployed), system information, server startup command-line, and then current server process quotas. The last three lines of the header provide a list of the categories being recorded, the filters in use, and the last, column headings described as follows:
Note that some items also include a block of data. The request header category does this, providing the blank-line terminated text comprising the HTTP header. Rule mapping also provides a block of information representing each rule as it is interpreted. Generally WATCH-generated information can be distinguished from other data by the uniform format and delimiting vertical bars. Initiative and imagination is sometimes required to interpret the free-form data but a basic understanding of HTTP serving and a little consideration is generally all that is required to deduce the essentials of any report.
The following provides a brief explanation on the way WATCH operates and any usage implications.
A single client may be connected to the WATCH facility at any given time. When connecting the client is sent an HTTP response header and the WATCH report heading lines. The request then remains connected until the WATCH duration expires or the client overtly aborts the connection. During this period the browser behaves as if receiving a sometimes very slow, sometimes stalled, plain-text document. As the server processes WATCHable events the text generated is sent to the WATCH-connected client.
If the connection is aborted by the user some browsers will consider document retrieval to be incomplete and attempt to reconnect to the service if an attempt is made to print or save the resulting document. As the printing of WATCH information is often quite valuable during problem resolution this behaviour can result in loss of information and generally be quite annoying. Appropriate use of the duration selector when requesting a report can work around this, as at expiry the server disconnects, browsers generally interpreting this as legitimate end-of-document (when no content-length has been specified).
During report processing some browsers may not immediately update the on-screen information to reflect received data without some application activity. If scroll-bars are present on the document window manipulating either the horizonal or vertical slider will often accomplish this. Failing that minimizing then restoring the application will usually result in the most recent information being visible.
Browser reload/refresh may be used to restart the report. A browser will quite commonly attempt to remain at the current position in the document, which with a WATCH report's sustained but largely indeterminate data stream may take some time to reach. It is suggested the user ensure that any vertical scroll-bar is at the beginning of the current report, then refresh the report.
Selecting a large number of categories, those that generate copious output for a single event (e.g. response body) or collecting for extended periods can all result in the receipt of massive reports. Some browsers do not cope well with documents megabytes in size.
When supplying WATCH output as part of a problem report please ZIP the file and include it an an e-mail attachment. Mailers often mangle the report format making it difficult to interpret.
Although intended primarily as a tool for online use WATCH can be deployed at server startup with a command-line qualifier and provide report output to the server process log. This is slightly more cumbersome than the Web interface but may still be useful in some circumstances. Full control over event categories and filters is possible.
The following examples illustrate the command-line WATCH specification.
| 11.1Simple File Request Turn-Around |
| 11.2Scripting |
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |
The server has a single-process, multi-threaded, asynchronous I/O design. On a single-processor system this is the most efficient approach. On a multi-processor system it is limited by the single process context (with scripts executing within their own context). For I/O constrained processing (the most common in general Web environments) the AST-driven approach is quite efficient.
The test-bench system was an DEC PWS 500 with 1 CPU and 1.5GB memory, running VSI OpenVMS V8.4-2L1 and VSI TCP/IP TCPIP V5.7-13ECO5F.
While by today's standards it is a very resource constrained system, especially by the EV56 (21164A) CPU, it has pretty-much done everything asked of it for all that time. Importantly, it has recent releases of system software, courtesy of VSI's ISV support programme. For performance purposes, this allows comparison with recent releases of CSWS (VMS Apache).
The requirements for a test-bench system effectively excludes production systems, especially external ones, hence working with what is at hand.
This performance data (WASD v11.5) has been collected very differently to the next most recent from over a decade ago (WASD v10.0). Apart from the move from an HP rx2600 to the vintage PWS 500, the previous benchmarking tools were WASD-in-house, ApacheBench (AB) and WASDbench (WB), executing on the same system as the server, eliminating network traffic on-the-wire. The current absolute benchmarks cannot meaningfully be compared to previous data. The relativities seem to be comparable.
These data have been collected using the h2load utility (https://nghttp2.org/documentation/h2load.1.html) from the HTTP/2 C Library (https://nghttp2.org). This utility can be used to configurably load HTTP, HTTPS and HTTP/2 servers. Note that the number of client threads (-t) is explicitly set to the connection concurrency (-c) to maximise h2load processing.
The h2load utility is running on a an 8CPU 32GB Mac Pro, across a 500 Mbps LAN to the 100 Mbps interface of the PWS. The obvious resource constraints are the single PWS CPU and network interface. Every effort has been made to ensure these do not unreasonably constrain the comparison.
Clear text HTTP (port 80) data is collected to measure internal server processing without the CPU-intensive overhead of encryption. Encrypted HTTP (port 443) data provides more real-world scenarios (especially now clear-text is largely deprecated). Both WASD and Apache were using OpenSSL 1.1.1 and negotiated TLS v1.2.
Output from h2load benchmarking runs are included in the WASD_ROOT:[EXERCISE]*V115*.TXT directory and is summarised below.
Every endeavour has been made to ensure the comparison is as equitable as possible. Both servers execute at the same process priority, access logging and host name lookup disabled, and runs on the same machine in the same relatively quiescent environment. Each test run was interleaved between each server to try and distribute any environment variations. Those runs that are very high throughput use a larger number of requests to improve sample period validity. Both servers were configured pretty-much "out-of-the-box", minimal changes (generally just enough to get the test environment going). Multiple data collections have yielded essentially equivalent relative results.
For the test-bench WASD v11.5 is present on ports 80 and 443.
The Apache comparison used the latest VSI AXPVMS CSWS V2.4-38C (based on Apache v2.4.38) kit. Apache is present on ports 7780 and 7443.
Previous benchmarking included OSU data. These are no longer collected.
A series of tests using batches of accesses. The first test returned an empty file measuring response and file access time, without any actual transfer. The second requested a file of 64k characters, testing performance with a more realistic load. All were done using one and ten concurrent requests.
| Requests/Second | Data Rate MBps | |||
|---|---|---|---|---|
| Response | WASD | Apache | WASD | Apache |
| 0k | 352 | 71 | 0.104 | 0.018 |
| 64k | 61 | 36 | 3.740 | 2.230 |
| Requests/Second | Data Rate MBps | |||
|---|---|---|---|---|
| Response | WASD | Apache | WASD | Apache |
| 0k | 1146 | 67 | 0.338 | 0.017 |
| 64k | 124 | 48 | 7.590 | 2.940 |
| Requests/Second | Data Rate MBps | |||
|---|---|---|---|---|
| Response | WASD | Apache | WASD | Apache |
| 0k | 276 | 51 | 0.092 | 0.013 |
| 64k | 21 | 25 | 1.300 | 1.550 |
| Requests/Second | Data Rate MBps | |||
|---|---|---|---|---|
| Response | WASD | Apache | WASD | Apache |
| 0k | 175 | 46 | 0.580 | 0.112 |
| 64k | 39 | 24 | 2.360 | 1.440 |
(VMS Apache currently does not support HTTP/2)
| Requests/Second | Data Rate MBps | |||
|---|---|---|---|---|
| Response | WASD | Apache | WASD | Apache |
| 0k | 191 | - | 0.286 | - |
| 64k | 20 | - | 1.210 | - |
| Requests/Second | Data Rate MBps | |||
|---|---|---|---|---|
| Response | WASD | Apache | WASD | Apache |
| 0k | 156 | - | 0.240 | - |
| 64k | 37 | - | 2.250 | - |
Data file (non-relevant output snipped):
Requests for a large binary file (3.92MB - 8039 blocks) indicate a potential transfer rate of multiple Mbytes per second.
(VMS Apache currently does not support HTTP/2)
| Concurrent | WASD | Apache | |
|---|---|---|---|
| HTTP/1.1 (clear) | 1 | 6.07 | 4.40 |
| 10 | 8.85 | 8.70 | |
| HTTP/1.1 (encrypted) | 1 | 2.91 | 3.23 |
| 10 | 2.77 | 2.92 | |
| HTTP/2 (encrypted) | 1 | 2.77 | - |
| 10 | 2.80 | - |
Data file (non-relevant output snipped):
The WASD server can handle STREAM, STREAM_LF, STREAM_CR, FIXED and UNDEFINED record formats very much more efficiently than VARIABLE or VFC files. With STREAM, FIXED and UNDEFINED files the assumption is that HTTP carriage-control is within the file itself (i.e. at least the newline (LF), all that is required required by browsers), and does not require additional processing. With VARIABLE record files the carriage-control is implied and therefore each record requires additional processing by the server to supply it. Even with variable record files having multiple records buffered by the HTTPd before writing them collectively to the network improving efficiency, stream and binary file reads are by Virtual Block and are written to the network immediately making the transfer of these very efficient indeed!
A simple performance evaluation shows the relative merits of WASD scripting and Apache in CGI and persistent environments, using WASD_ROOT:[SRC.CGIPLUS]CGIPLUSTEST.C which executes in standard CGI, CGIplus and Apache loadable module environments. CGIplus and Apache modules are somewhat analagous. A series of accesses were made. The first test returned only the HTTP header, evaluating raw request turn-around time. The second test requested a body of 64k characters, again testing performance with a more realistic load.
| Response | WASD CGI | WASD CGIplus | Apache CGI | Apache module |
|---|---|---|---|---|
| 0kB | 27 | 193 | 5 | 52 |
| 64kB | 14 | 25 | 5 | 31 |
| Response | WASD CGI | WASD CGIplus | Apache CGI | Apache module |
|---|---|---|---|---|
| 0kB | 28 | 337 | 4 | 51 |
| 64kB | 16 | 65 | 4 | 37 |
Data file (non-relevant output snipped):
CGI scripting is notoriously slow (as above), hence the effort expended by designers in creating persistent scripting environments - those where the scripting engine (and perhaps other state) is maintained between requests. Both WASD and Apache implement these as integrated features, the former as CGIplus/RTE, and in the latter as loadable modules.
The CGIplus and Apache module data from the above CGIPLUSTEST.EXE table show the benefits of having scripts persist, reducing activation latency, thereby increasing throughput, and potentially retaining state, including the scripts themselves in local caches. Both WASD and VMS Apache use their respective persistence technologies to provide common scripting environments, including Perl, PHP and Python.
The WASD CGIplus/RTE technology used to implement its persistent scripting environments are available for general use and based on CGI principles offer a ready adaptation of well-known principles. Most site-specific scripts can also be built using the libraries, code fragments, and example scripts provided with the WASD package, and obtain similar efficiencies and low latencies. See WASD Scripting Environment document.
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |
The Update facility allows Web documents and file environments to be administered from a standard browser. This capability is available to Web administrator and user alike. Availability and capability depends on the authorization environment within the server.
It should be stressed that this is not designed as a full hypertext administration or authoring tool, and for document preparation relies on the editing capabilities of the <TEXTAREA> widget of the user's browser. It does however, allow ad-hoc changes to be made to documents fairly easily, as well as allowing documents to be deleted, and directories to be created and deleted.
Consult the Current UPDate documentation for usage detail.
Here is an example of the interface (access may be
denied).
If SSL is in use (see 4. Transport Layer Security) then username/password privacy of the authorization environment is inherently secured via the encrypted communications. To restrict web update functionality to this secure environment add the following to the WASD_CONFIG_MAP configuration file:
Of course, the user must have write (POST/PUT) access to the document or area on the server (i.e. the path) and the server account have file system permission to write into the parent directory.
The server will report "Insufficient privilege or object protection violation ... /path/document" if it does not have file system permission to write into a directory.
Also see 3.13 Controlling Server Write Access for information on write access control for the server account.
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |
Foreign commands for external utilities (and the HTTPD control functionality) will need to be assigned from the adminstration users' LOGIN.COM either explicitly or by calling the WASD_ROOT:[EXAMPLE]WASDVERBS.COM procedure.
Ever had to go to extraordinary lengths to find out exactly what your browser is sending to the server? The server provides a request echo facility. This merely returns the complete request as a plain-text document. This can be used for for checking the request header lines being provided by the browser, and can be valuable in the diagnosis of POSTed forms, etc.
This facility must be enabled through a mapping rule entry.
It may then be used with any request merely by inserting "/echo" at the start of the path, as in the following example.
The hiss facility provides a response stream made up of random alpha-numeric characters (a sort of alpha-numeric white-noise). No response header is generated and the stream will continue (by default) up to one megabyte of output, or until the client closes the connection. This maximum may be controlled my appending an integer representing the number of kilobytes maximum to the mapping. This facility must be enabled through a mapping rule entry and may then be used for specific requests.
Usage details are described in Security Considerations of WASD Configuration).
The stream facility provides a quantified or unlimited response stream of printable or binary octets. It is intended as a light-weight data source delivering content at the maximum throughput capable by the server and platform. This can be used as a test source or for end-to-end metrics. This facility must be enabled through a mapping rule.
It may then be used to generate streams of data with various characteristics and sizes by including parameters in the URL.
Need to locate where VMS has the HTTPd files? This simple facility maps the supplied path then parses it to obtain a resulting VMS file specification. This does not demonstrate whether the path actually exists!
This facility must be enabled through a mapping rule entry.
It may then be used with any request merely by inserting "/where" at the start of the path, as in the following example.
The Xray facility returns a request's complete response, both header and body, as a plain text document. Being able to see the internals of the response header as well as the contents of the body rendered in plain text can often be valuable when developing scripts, etc.
This facility must be enabled through a mapping rule entry.
It may then be used with any request merely by inserting "/xray" at the start of the path, as in the following example.
This server stress-test and benchmarking tool, as used in the Apache Distribution, is included with the WASD package (sourced from http://webperf.zeus.co.uk/ab.c), within license conditions.
ApacheBench will only compile and run for Alpha, Itanium or VAX systems with VMS 7.n or greater available. Also see the WASD analogue, ‘WASDbench Utility’ in 13.14 StreamLF Utility. ApacheBench is a simple but effective tool, allowing a single resource to be requested from a server a specified number of times and with a specified concurrency. This can be used to benchmark a server or servers, or be used to stress-test a server configuration's handling of variable loads of specific resquests (before exhausting process quotas, etc.) This utility has remained at the 1.3 release due to subsequent versions (e.g. 2.0) having Apache API dependencies.
A small addition to functionality has been made. The WASD ApacheBench displays a count of the HTTP response categories received (i.e. the number of 2nns, 4nns, etc.) This allows easier assessment of the relevance of results (i.e. measuring performance of some aspect only to find the results showed the performance of 404 message generation - and yes, an annoying experience of the author's prompted the changes!)
The following examples illustrate its use.
The Consolidate Access LOGS utility (pronounced similar to the breakfast cereal brand :-) merges multiple HTTP server common and combined format access logs into a single log file with records in time-order. Due to the granularity of HTTP server entry timestamps (one second) the records are sorted to the one second but not within the one second.
It uses RMS and the VMS sort-merge routines to provide the basic consolidation functionality. An RMS search uses the supplied wildcard log file specification. Matching files are opened and each record read. The date/time field is parsed and a binary timestamp generated. Records with formats or date/time fields that do not make sense to the utility are discarded. When all files have been processed the sort-merge is performed using the timestamp as the key. The sorted records are then written to the specified output file.
$ calogs <log-file-spec> [<output-file-name>] [<qualifiers>]
| Parameter | Description |
|---|---|
| /HELP | basic usage information |
| /NOPROXY | discard proxy service records |
| /NOWASD | discard WASD server status/timestamp entries |
| /OUTPUT= | alternate method of specifying merged file name |
| /PROXY | discard non-proxy service records |
| /QUIET | no messages apart from errors |
| /VERBOSE | per-file progress messages |
| /VERSION | display the utility version and copyright message |
Content Security Policy (CSP) is an added layer of security that helps to detect and mitigate certain types of attacks, including Cross Site Scripting (XSS) and data injection attacks.
https://en.wikipedia.org/wiki/Content_Security_Policy
https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
WASD provides CSP support using mapping rules. See Content Security Policy (CSP) of WASD Configuration.
When POSTed to, this utility appends a timestamp and CSP report JSON to the file specified by the CSPREPORT_FILE logical name. This file must be located somewhere the scripting account has read+write access to. When accessed using a GET the utility accesses the stored CSP reports and returns a formatted HTML report listing each. GET requests (reporting) must be subject to authentication and authorisation.
For further infomation check the descriptive prologue in the WASD_ROOT:[SRC.UTILS]CSPREPORT.C source code.
The HTAdmin utility assists in with the command-line maintenance of $HTA authorization databases. See Authorization Configuration (Basics) of WASD Configuration and 3. Authentication and Authorization.
htadmin <database> [<username>] [<qualifiers>]
| Parameter | Description |
|---|---|
| /ADD | add a new record |
| /CONFIRM | confirm deletion of database |
| /CONTACT="<string>" | contact information for record |
| /CREATE | create a new database |
| /CSV[=TAB|char] | comma-separated listing (optional character) |
| /DATABASE= | database name (or as command-line parameter) |
| /DELETE | delete a database or username record from a database |
| /DISABLED | username record is disabled (cannot be used) |
| /EMAIL="<string>" | email address for record |
| /ENABLED | username record is enabled (can be used) |
| /FULL | listing showing full details |
| /GENERATE | generate a six character password |
| /HELP | basic usage information |
| /[NO]HTTPS | synonym for /SSL |
| /LIST | listing (brief by default, see /FULL and /CSV) |
| /MODIFY | synonym for /UPDATE |
| /NAME="<string>" | full name for username record |
| /OUTPUT= | alternate output for database listing |
| /PASSWORD[=<string>] | username record password (prompts if not supplied) |
| /PIN | generate four-digit "PIN number" for password |
| /[NO]READ | username can/can't read |
| /SORT[=<parameters>] | sort the records into a new/another database |
| /[NO]SSL | user can only authenticate via SSL ("https:") |
| /[NO]WRITE | username can/can't write |
| /UPDATE | update an existing username record |
| /USER=<string> | username |
| /VERSION | display version of HTADMIN |
The /SORT qualifier sorts the current database records according to the /SORT= parameters. It can be used with the /LIST qualifier to produce ordered reports or will output the records into another authentication file. By default it sorts ascending by username. Qualifier parameters allow a sort by DATE or COUNT. Each of these allows the further specification of which date or count; ACCESS, CHANGE or FAILURE.
The HTTP server may be monitored in real-time using the HTTPDMON utility.
This utility continuously displays a screen of information comprising three or four of the following sections:
The following shows example output:
The "/HELP" qualifier provides a brief usage summary.
The server counter values are carried over when a server (re)starts (provided the system has stayed up). To reset the counters use the online Server Administration facility (9. Server Administration).
If [DNSlookup] is disabled for the HTTP server the HTTPDMON utility attempts to resolve the literal address into a host name. This may be disabled using the /NORESOLVE qualifier.
From RFC1321 …
" The [MD5] algorithm takes as input a message of arbitrary length and produces as output a 128-bit "fingerprint" or "message digest" of the input. It is conjectured that it is computationally infeasible to produce two messages having the same message digest, or to produce any message having a given prespecified target message digest. "
The MD5DIGEST utility is primarily provided with WASD for verifying kits as unchanged from the originals released. With the proliferation of mirror sites and other distribution resources it has become good practice to ensure kits remain unchanged from release, to distribution, to installation site (changes due to to data corruption or malicious intent - as remote a possibility as that may seem). Of course it may also be used for any other purpose where the MD5 hash is useful.
For verifying the contents of a WASD release connect to the original WASD distribution site, refer to the download page, and make a comparison between the release MD5 hash found against the list of all archive hashes and the MD5 hash of your archive. That can be done as follows
Of course, if you have not yet installed your first WASD distribution using the MD5DIGEST utility that is part of it is not feasable. The original site can provide kits and pre-built executables for this purpose.
Quick-and-Dirty LOG STATisticS is a utility to extract
very elementary statistics from Web server common/combined format log files.
It is intended for those moments when we think "I wonder how many times
that new archive has been downloaded?", "How much data was transfered
during November?", "How often is such-and-such a client
using the authenticated so-and-so service?", "How much has
the mail service been used?" … and want the results in a matter of
seconds (or at least a few tens of seconds ;-) It is available at the
command-line and as a CGI script.
For QDLOGSTATS to be available as a CGI script it must have authorization enabled against it (to prevent potential ad hoc browsing of a site's logs). The following provides some indication of this configuration, although of course it requires tailoring for any given site.
It could then be accessed using
The initial access provides a form allowing the various filters and other behaviours to be selected. The CGI form basically parallels the command-line behaviour described below.
A number of filters allow subsets of the log contents to be selected. These filters support the same string matching expressions as the server (see String Matching of WASD Configuration).
A knowlege of the format and contents of the common and combined log formats will assist in deciding which and to what purpose filters should be used. Record filtering is done in the same order as is finally displayed, so method would be processed before user-agent for instance. Normally a record match terminates on the first non-matched filter (to expedite processing). To compare and report each filter for every record apply the /ALL qualifier. To view records as they are processed use the /VIEW qualifier. This by default displays all matched records, but the optional =ALL or =NOMATCH parameters will display all records, or all those but the matches.
QDLOGSTATS log-file-spec [pattern qualifiers] [other qualifiers]
| Parameter | Description |
|---|---|
| /ALL | compare and report on all supplied filters |
| /AUTHUSER= | pattern (any authenticated username) |
| /BEFORE= | log files before this VMS date/time |
| /CLIENT= | pattern (client host name or IP address) |
| /DATETIME= | pattern ("11/Jun/1999:14:08:49 +0930") |
| /DECODE[=keyword] | URL-decode PATH, QUERY, REFERER before match |
| /METHOD= | pattern (HTTP "GET", "POST", etc.) |
| /OUTPUT= | file specification |
| /PATH= | pattern (URL path component only) |
| /PROGRESS | show progress during processing; a "+" for each file started, a "." for each 1000 records processed |
| /QUERY= | pattern (URL query component only) |
| /REFERER= | pattern (HTTP "Referer:" field, COMBINED only) |
| /REMOTEID= | pattern (RFC819 file) |
| /RESPONSE= | pattern (HTTP response code) |
| /SINCE= | log files after this VMS date/time |
| /SIZE[=keyword] | response size (in bytes) MIN=integer MAX=integer |
| /USERAGENT= | pattern (HTTP "User-Agent:" field, COMBINED only) |
| /VIEW[=type] | display matching log records (ALL, NOMATCH, MATCH) |
The SECHAN utility (pronounced "session") is used by [INSTALL]SECURE.COM and associated procedures to make file system security settings. It is also available for direct use by the site administrator. See Security Considerations of WASD Configuration).
This simple procedure used the FDL facility to convert files to STREAM_LF format. The WASD HTTPd server access STREAM_LF files in block/IO-mode, far more efficiently that the record-mode required by variable-record format files.
NOTE: The server can also be configured to automatically convert any VARIABLE record format files it encounters to STREAM_LF.
WASDbench - an analogue to ApacheBench (13.6 ApacheBench). Why have it? ApacheBench only compiles and runs on VMS 7.n and later. This version should compile and run for all supported WASD configurations. It also has the significant performance advantage (looks like ~25%) of using the underlying $QIO services and not the socket API, and is AST event driven rather than using the likes of select(). It is not a full implementation of AB (for instance, it currently does not do POSTs). The CLI attempts to allow the same syntax as used by AB (within the constraint that not all options are supported) so that it is relatively easy to switch between the two (perhaps for comparison purposes) if desired.
The following examples illustrate its use.
WASDbench also has an exercise option, functionality is not found in ApacheBench. It is basically to supercede similar functionality provided by the retired WWWRKOUT. The exercise functionality allows WASDbench to be used to stress-test a server. This behaviour includes mixing HEAD (~5%) with GET requests, and breaking requests during both request and response transfers (~5%). These are designed to shake up the server with indeterminate request types and client error behaviours. The best way to utilize this stress-testing is wrap WASDbench with a DCL procedure providing a variety of different requests types, quantities and concurrencies.
The "WASD Over-The-Shoulder Uptime Picket" is designed to monitor WASD in a production environment for the purpose of alerting operations staff to conditions which might cause that production to be adversely impacted.
Alert triggers include:
Alert reports can be delivered via any combination of:
The utility runs in a detached process and monitors the server environment by periodically polling various server data at a default interval is 15 seconds. As the utility requires access to global memory accounting a per-system WOTSUP is required for each node to be monitored.
The following (somewhat contrived) example illustrates the format and content of a WOTSUP report delivered via OPCOM. Reports delivered via other mechanisms have the same content and similar format.
For further infomation check the descriptive prologue in the WASD_ROOT:[SRC.UTILS]WOTSUP.C source code.
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |
Licensed under the GNU Public License, Version 3;
This product includes software developed by the Apache Group for use in the Apache HTTP server project (http://www.apache.org/).
This package uses the Expat XML parsing toolkit.
This package uses essential algorithm and code from Flexible and Economical UTF-8 Decoder.
This package contains software made available by the Free Software Foundation under the GNU General Public License.
This package contains software provided with the OSU (DECthreads) HTTP server package, authored by David Jones:
This product can include software developed by the OpenSSL Project for use in the OpenSSL Toolkit (https://www.openssl.org/).
This package uses SHA-1 hash code.
This software contains code derived in part from RSA Data Security, Inc:
SortTable version 2
Stuart Langridge, http://www.kryogenix.org/code/browser/sorttable/
nghttp2 - HTTP/2 C Library
Tatsuhiro Tsujikawa, https://github.com/tatsuhiro-t
VSI OpenVMS is a registered trademark of VMS Software Inc.
OpenVMS,
HP TCP/IP Services for OpenVMS,
HP C,
Alpha,
Itanium and
VAX
are registered trademarks of Hewlett Packard Enterprise
MultiNet and TCPware are registered trademarks of Process Software Corporation
| ↩︎ | ↖︎ | ↑︎ | ↘︎ | ↪︎ |