and generating
inappropriate record boundaries the application could be wrapped as follows (a
real-world example).
|code|
$ set noon
$ define/user/nolog sys$input http$input
$ define/user DECC$STDIO_CTX_EOL ENABLE
$ calcserver
$ exit(1)
|!code|
|0IPC Tickler|
|^ The interactions between VMS' record-oriented I/O, various run-time
libraries (in particular the C-RTL), the streaming character-oriented Web, and
of course WASD, can be quite complex and result in unintended output or
formatting. The CGI script Inter-Process Communication (IPC) tickler
|link%|/wasd_root/src/misc/IPCtickler.c|WASD_ROOT:[SRC.MISC]IPCTICKLER.C| is
designed to allow a script programmer to gain an appreciation of how these
elements interact, how WASD attempts to accomodate them, what mechanisms a
script can use to explicitly convey exact requirements to WASD ... and finally,
how these affect output (in particular the carriage-control) delivered to the
client. If installed use |link%|/cgi-bin/IPCtickler| to obtain an HTML form
allowing control of several parameters into the script.
|0Script-Control:|
|^ The |/Apache Group| has proposed a CGI/1.2 that includes a
|/Script-Control:| CGI response header field. WASD implements the
one proposed directive, along with a number of WASD extensions (those
beginning with the "X-"). Note that by convention extensions unknown by
an agent should be ignored, meaning that they can be freely included, only
being meaningful to WASD and not significant to other implementations.
|bullet|
|item| |*no-abort |-| |
The server must not terminate the script during processing for either
no output or no-progress timeouts. The script is to be left completely alone
to control its own termination. Caution, such scripts if problematic could
easily accumulate and "clog up" a server or system.
|item| |*X-buffer-records[=0\|1] |-| |
Buffer records written by the script until there is [BufferSizeDclOutput] bytes
available then write it as a single block to the client. The optional zero
returns the script to non-buffered records, a one enables record buffering.
|item| |*X-content-encoding-gzip[=0\|1] |-| |
A script by specifying a zero can request that the server, by default encoding
the response using GZIP compression, leaves the response unencoded. By
specifying a one a script can request the server to encode the response using
GZIP compression.
|item| |*X-content-handler=|/keyword| |-| |
The output of the script is stripped of the CGI reponse header and the body is
given to the specified content handler. Currently only the SSI engine is
supported using "X-content-handler=SSI".
|item| |*X-crlf-mode[=0\|1] |-| |
The server should always ensure each record has trailing carriage-return then
newline characters (0x0d, 0x0a). This is generally what VMS requires for
carriage control on terminals, printers, etc. The optional zero returns the
script to record mode, a one enables CR-LF mode.
|item| |*X-error-line=|/integer| |-| |
Source code module line number META in server generated error message
(optional).
|item| |*X-error-module=|/string| |-| |
Source code module name META in server generated error message (optional).
|item| |*X-error-text="|/string||" |-| |
Text of error message (mandatory, to trigger server message generation). If
without a VMS status value this is the entire error explanation. If with a VMS
status it becomes the context of the status.
|item| |*X-error-vms-status=|/integer| |-| |
VMS status value in decimal or VMS hexadecimal (e.g. %X0000002C, optional).
Changes format of error message to detailed explanation of status.
|item| |*X-error-vms-text="|/string||" |-| |
Context of a VMS status value (optional). Usually a VMS file specification of
other text related to the status value (in commented content of error message).
|item| |*X-http-status=|/integer| |-| |
Forces the HTTP response line status to this value regardless of the status in
the CGI |/Status:| response or the server status if being used as an
error reporting script
|item| |*X-lifetime=|/value| |-| |
The number of minutes before the idle scripting process is deleted by the
server. Zero sets it back to the default, "none" disables this
functionality.
|item| |*X-record-mode[=0\|1] |-| |
The server should always ensure each record has a trailing newline
character (0x0a), regardless of whether the response is a |/text/...||
content-type or not. This is what is usually required by browsers for
carriage-control in text documents. The optional zero changes the script to
stream mode, a one enables record mode.
|item| |*X-record0-mode[=0\|1] |-| |
Some (C-RTL) carriage-control modes result in empty records being sent in lieu
of carriage control (newlines). This (rather esoteric) mode only adds a
newline to empty records.
|item| |*X-stream-mode[=0\|1] |-| |
The server is not to adjust the carriage-control of records regardless of
whether the response is a |/text/...| content-type or not. What the
script writes is exactly what the client is sent. The optional zero returns
the script to record mode, a one enables stream mode.
|item| |*X-timeout-noprogress=|/value| |-| |
The number of minutes allowed where the script does not transfer any data to
the server before the server deletes the process. Zero sets it back to the
default, "none" disables this functionality.
|item| |*X-timeout-output=|/value| |-| |
The number of minutes allowed before an active script is deleted by the server,
regardless of it still processing the request. Zero sets it back to the
default, "none" disables this functionality.
|!bullet|
|^ The following is a simple example response where the server is instructed
not to delete the script process under any circumstances, and that the body
does not require any carriage-control changes.
|code|
Content-Type: text/plain
Script-Control: no-abort; X-stream-mode
|/long, slowww script-output |...|||
|!code|
|0Example DCL Scripts|
|^ A simple script to provide the system time might be:
|code|
$ say = "write sys$output"
$! the next two lines make it CGI-compliant
$ say "Content-Type: text/plain"
$ say ""
$! start of plain-text body
$ show time
|!code|
|^ A script to provide the system time more elaborately (using HTML):
|code|
$ say = "write sys$output"
$! the next two lines make it CGI-compliant
$ say "Content-Type: text/html"
$ say ""
$! start of HTML script output
$ say ""
$ say "Hello ''WWW_REMOTE_HOST'" !(CGI variable)
$ say ""
$ say "System time on node ''f$getsyi("nodename")' is:"
$ say "
''f$cvtime()'
"
$ say ""
|!code|
|3Non-Parsed-Header Output|
|^ A script does not have to output a CGI-compliant data stream. If it begins
with a HTTP header status line WASD assumes it will supply a |*raw| HTTP data
stream, containing all the HTTP requirements. This is the same as or
equivalent to the |/non-parsed-header||, or "nph|...|" scripts of many
environments. This is an example of such a script response.
|code|
HTTP/1.0 200 Success
Content-Type: text/html
Content-Length: 35
Hello world!
|!code|
|^ Any such script must observe the HyperText Transfer Protocol, supplying a
|*full response header and body, including correct carriage-control||. Once
the server detects the HTTP status header line it pays no more attention to any
response header fields or body records, just transfering everything directly to
the client. This can be very efficient, the server just a conduit between
script and client, but does transfer the responsibility for a correct HTTP
response onto the script.
|0NPH DCL Script||
|^ The following example shows a DCL script. Note the full HTTP header and
each line explicitly terminated with a carriage-return and line-feed pair.
|code|
$ lf[0,8] = %x0a
$ crlf[0,16] = %x0d0a
$ say = "write sys$output"
$! the next line determines that it is raw HTTP stream
$ say "HTTP/1.0 200 Success" + crlf
$ say "Content-Type: text/html" + crlf
$! response header separating blank line
$ say crlf
$! start of HTML script output
$ say "" + lf
$ say "Hello ''WWW_REMOTE_HOST'" + lf
$ say "" + lf
$ say "Local time is ''WWW_REQUEST_TIME_LOCAL'" + lf
$ say "" + lf
|!code|
|0NPH C Script|
|^ When scripting using the C programming language there can be considerable
efficiencies to be gained by providing a binary output stream from the script.
This results in the C Run-Time Library (C-RTL) buffering output up to the
maximum supported by the IPC mailbox. This may be enabled using a code
construct similar to following to reopen |/stdout| in binary mode.
|code|
if ((stdout = freopen ("SYS$OUTPUT", "w", stdout, "ctx=bin")) == NULL)
exit (vaxc$errno);
|!code|
|^ This is used consistently in WASD scripts. Carriage-control must be
supplied as part of the C standard output (no differently to any other C
program). Output can be be explicitly sent to the client at any stage using
the |/fflush()| standard library function. Note that if the
|/fwrite()| function is used the current contents of the C-RTL buffer
are automatically flushed along the the content of the fwrite().
|code|
fprintf (stdout,
"HTTP/1.0 200 Success\\r\\n\\
Content-Type: text/html\\r\\n\\
\\r\\n\\
\\n\\
Hello %s\\n\\
\\n\\
System time is %s\\n\\
\\n",
getenv("WWW_REMOTE_HOST"),
getenv("WWW_REQUEST_TIME_LOCAL"));
|!code|
|3Bulk Content Output|
|^ As described above, |link|Script Output||, the default script|=.<->||server
IPC uses a mailbox. While versatile and sufficiently efficient for general
use, when megabytes, tens of megabytes, and hundreds of megabytes need to be
transferred, using a memory buffer shared between script and server can yield
transfer |*improvements of up to 500%||.
|note><|
|0YMMV|
Of course, your mileage may vary with platform, O/S version and TCP/IP stack
(i.e. as the relative bottlenecks shuffle about).
|!note|
|^ The script requests a memory-buffer using a CGI callout (|link|CGI
Callouts||). Buffer size is constrained by the usual VMS 32bit memory
considerations, along with available process and system resources. The server
creates and maps a non-permanent global section. If this is successful the
script is advised of the global section name using the callout response. The
script uses this to map the section name and can then populate the buffer.
When the buffer is full or otherwise ready, the script issues a callout with
the number of bytes to write, and then stalls. The complete memory buffer may
be written at once or any subsection of that buffer. The write is accomplished
asynchronously and may comprise multiple network $QIOs or TLS/SSL blocks. When
complete, a callout response to the script is issued and the script can
continue processing. Standard script mailbox I/O (SYS$OUTPUT, ) and
memory-buffer I/O may be interleaved as required.
|^ The callouts are as follows:
|bullet|
|item| |*BUFFER-BEGIN:| |/.[k\|M]||
|^ Create a temporary global section to act as a memory buffer shared between a
script process and the server. The default is |*M||egabytes.
|item| |*BUFFER-END:||
|^ Dispose of the shared memory buffer created by callout BUFFER-BEGIN.
|item| |*BUFFER-WRITE:| |/.||
|^ Instruct the server to write bytes from the shared
memory buffer to the client.
|!bullet|
|^ See working examples in
|link%|/wasd_root/src/misc/*membuf*.*|WASD_ROOT:[SRC.MISC]||.
|^ Actual data comparing standard mailbox IPC with memory-buffer generated
using [SRC.MISC]MEMBUFDEMO.C on a HP rx2660 (1.40GHz/6.0MB) with 4 CPUs and
16383MB running VSI VMS V8.4-2L1 with Multinet UCX$IPC_SHR V55A-B147, OpenSSL
1.0.2k and WASD v11.2.0, with [BufferSizeDclOutput] 16384. In each case 250MB
("?250") is transfered via a either a 16.4kB mailbox (default) or 16.4kB memory
buffer ("+b"). Significantly larger memory buffer may well improve throughput
further.
|code|
$ wget "-O" nl: http://127.0.0.1/cgi-bin/membufdemo?250
--2017-10-14 03:19:05-- http://127.0.0.1/cgi-bin/membufdemo?250
Connecting to 127.0.0.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 262144000 (250M) [application/octet-stream]
Saving to: 'nl:'
nl: 100%[=====================>] 250.00M 25.6MB/s in 12s
2017-10-14 03:19:17 (20.5 MB/s) - 'nl:' saved [262144000/262144000]
$ wget "-O" nl: http://127.0.0.1/cgi-bin/membufdemo?250+b
--2017-10-14 03:19:23-- http://127.0.0.1/cgi-bin/membufdemo?250+b
Connecting to 127.0.0.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 262144000 (250M) [application/octet-stream]
Saving to: 'nl:'
nl: 100%[=====================>] 250.00M 105MB/s in 2.4s
2017-10-14 03:19:26 (105 MB/s) - 'nl:' saved [262144000/262144000]
$ wget "-O" nl: https://127.0.0.1/cgi-bin/membufdemo?250
--2017-10-14 03:19:50-- https://127.0.0.1/cgi-bin/membufdemo?250
Connecting to 127.0.0.1:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 262144000 (250M) [application/octet-stream]
Saving to: 'nl:'
nl: 100%[=====================>] 250.00M 14.5MB/s in 17s
2017-10-14 03:20:07 (14.5 MB/s) - 'nl:' saved [262144000/262144000]
$ wget "-O" nl: https://127.0.0.1/cgi-bin/membufdemo?250+b
--2017-10-14 03:20:12-- https://127.0.0.1/cgi-bin/membufdemo?250+b
HTTP request sent, awaiting response... 200 OK
Length: 262144000 (250M) [application/octet-stream]
Saving to: 'nl:'
nl: 100%[=====================>] 250.00M 16.6MB/s in 15s
2017-10-14 03:20:27 (16.6 MB/s) - 'nl:' saved [262144000/262144000]
|!code|
|^ It is obvious that memory-buffer provides significantly greater throughput
than mailbox (from the http:// test) and that with TLS/SSL network transport
the encryption becomes a significant overhead and choke-point. Nevertheless,
there is still an approximate 15% dividend, plus the more efficient interface
the script->memory-buffer->server provides. The VMS TLS/SSL implementation may
improve with time, especially if TLS/SSL hardware engines become available with
the port to x86_64.
|note|
|0Postscript|
The comparison also illustrates that the WASD environment can deliver
significant bandwidth through its script|=.->||server|=.->||network pathways. On
the demonstration class of system; ~200Mbps unencrypted and ~120Mbps encrypted
using the standard mailbox IPC; with ~850Mbps unencrypted and ~130Mbps
encrypted using the memory-buffer IPC.
|!note|
|2Raw HTTP Input (POST Processing)|
|^ For POST and PUT HTTP methods (e.g. a POSTed HTML form) the body of the
request may be read from the HTTP$INPUT stream. For executable image scripts
requiring the body to be present on SYS$INPUT (the C language |/stdin| stream)
a user-mode logical may be defined immediately before invoking the image, as in
the example.
|code|
$ EGSCRIPT = "$WASD_EXE:EGSCRIPT.EXE"
$ DEFINE /USER SYS$INPUT HTTP$INPUT
$ EGSCRIPT
|!code|
|^ The HTTP$INPUT stream may be explicitly opened and read. Note that this is
a raw stream, and HTTP |/lines| (carriage-return/line-feed terminated sequences
of characters) may have been blocked together for network transport. These
would need to be explicity parsed by the program.
|code|
if ((HttpInput = fopen ("HTTP$INPUT", "r", "ctx=bin")) == NULL)
exit (vaxc$errno);
|!code|
|^ When scripting using the C programming language there is a tendency for the
C-RTL to check for and/or add newline (0x10, ) carriage-control on receipt
of record (single write). While this can be useful in converting from VMS to C
conventions it can also be counter-productive if the stream being received is
already using C carriage-control. To prevent the C-RTL reinterpreting data
passed to it it often, perhaps invariably, necessary to reopen the input stream
as binary using a construct similar to following.
|^ This, and its equivalent (below), are used consistently in WASD
scripts.
|code|
if ((stdin = freopen ("HTTP$INPUT", "r", stdin, "ctx=bin")) == NULL)
exit (vaxc$errno);
if ((stdin = freopen ("SYS$INPUT", "r", stdin, "ctx=bin")) == NULL)
exit (vaxc$errno);
|!code|
|^ |*The input stream should be read before generating any output.| If an
error occurs during the body processing it should be reported via a CGI
response header indicating an error (i.e. non-200). With HTTP/1.1 request
processing there is also a requirement (that CGILIB fulfills) to return a "100
Continue" interim response after receiving the client request header and before
the client sends the request body. Output of anything before this "100
Continue" is delivered will cause it to be interleaved with the script response
body.
|2CGI Function Library|
|^ A source code collection of C language functions useful for processing the
more vexing aspects of CGI/CGIplus programming
(|link|Scripting Function Library||).
|2CGIUTL Utility|
|^ This assists with the generation of HTTP responses, including the transfer
of binary content from files (copying a file back to the client as part of the
request), and the processing of the contents of POSTed requests from DCL
(|link|CGIUTL Utility||).