CGIplus-enabled Run-time Environment Example -------------------------------------------- ***** FIRST, EVIDENCE OF PERSISTANCE ***** Usage Count: 1 ***** SECOND, THE CGI ENVIRONMENT AVAILABLE ***** WWW_AUTH_TYPE= WWW_CONTENT_LENGTH=0 WWW_CONTENT_TYPE= WWW_DOCUMENT_ROOT= WWW_GATEWAY_BG=BG47732: WWW_GATEWAY_INTERFACE=CGI/1.1 WWW_GATEWAY_EOF=$Z-46AA6E7BAC91B7804E817181- WWW_GATEWAY_EOT=$D-E1EC714E5D65B177A5E9274C- WWW_GATEWAY_ESC=$E-D9FF53DDDC9560FA9A855A33- WWW_GATEWAY_MRS=4492 WWW_HTTP_CF_RAY=8e99133d8ef62c40-ORD WWW_HTTP_USER_AGENT=Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) WWW_HTTP_HOST=675502.arinterhk.tech WWW_HTTP_CF_IPCOUNTRY=US WWW_HTTP_ACCEPT_ENCODING=gzip, br WWW_HTTP_X_FORWARDED_FOR=3.145.95.233 WWW_HTTP_X_FORWARDED_PROTO=https WWW_HTTP_ACCEPT=*/* WWW_HTTP_CF_CONNECTING_IP=3.145.95.233 WWW_HTTP_CF_VISITOR={"scheme":"https"} WWW_HTTP_CONNECTION=Keep-Alive WWW_HTTP_CDN_LOOP=cloudflare; loops=1 WWW_PATH_INFO= WWW_PATH_TRANSLATED= WWW_QUERY_STRING= WWW_REMOTE_ADDR=172.70.131.58 WWW_REMOTE_HOST=172.70.131.58 WWW_REMOTE_PORT=63924 WWW_REMOTE_USER= WWW_REQUEST_METHOD=GET WWW_REQUEST_PROTOCOL=HTTP/1.1 WWW_REQUEST_SCHEME=http: WWW_REQUEST_TIME_GMT=Thu, 28 Nov 2024 08:50:27 GMT WWW_REQUEST_TIME_LOCAL=Thu, 28 Nov 2024 09:50:27 WWW_REQUEST_URI=/rtbin/dict.c WWW_SCRIPT_FILENAME=WASD_ROOT:[src.httpd]dict.c WWW_SCRIPT_NAME=/rtbin/dict.c WWW_SCRIPT_RTE=cgi-bin:[000000]rte_example.exe WWW_SERVER_ADDR=146.48.108.2 WWW_SERVER_CHARSET=ISO-8859-1 WWW_SERVER_GMT=+01:00 WWW_SERVER_NAME=ns1.gposta.it WWW_SERVER_PROTOCOL=HTTP/1.1 WWW_SERVER_PORT=80 WWW_SERVER_SIGNATURE=
WASD/11.5.1 Server at ns1.gposta.it Port 80
WWW_SERVER_SOFTWARE=HTTPd-WASD/11.5.1 OpenVMS/IA64 SSL WWW_UNIQUE_ID=Z0g84wAAAAQkwAEpACE WWW_KEY_COUNT=0 ***** THIRD, AN "INTERPRETED" FILE (WWW_SCRIPT_NAME/WWW_SCRIPT_FILENAME) ***** [0001] /*****************************************************************************/ [0002] /* [0003] dict.c [0004] [0005] A hash dictionary (associative array) of key-value pairs. Both key and value [0006] are commonly strings but actually can be any sized blob. Keys that begin with [0007] a digit have special meaning to response header processing and so should be [0008] avoided in that type (especially). Inserting a key that already exists [0009] effectively replaces the previous value - unless - the uniquifier bit :-} is [0010] set, in which case a dictionary-unique integer (followed by a colon) is [0011] prefixed to the key allowing multiple entries having (essentially) the same [0012] key. As memory allocation can change due to value length increase any [0013] reference to the entry or its value must be re-set. [0014] [0015] All keys are pushed to lower-case on insertion, though key lookup is case [0016] insensitive. Each key has a "type", represented by a specific, [0017] non-alphanumeric character. [0018] [0019] o $ request processing (server internal) [0020] o ~ meta configuration (DICT and SET DICT=..) [0021] o > request header field [0022] o < response header field [0023] [0024] (There are others, some currently unused.) These allow entries that can be [0025] discriminated between quite simply. For example, iterating through all request [0026] header fields. Some configuration uses of the dictionary (i.e. the DICT [0027] directive and the mapping SET DICT=.. rule) will accept a key with a leading [0028] non-alpha-numeric that allows a specific type key to be looked up and also [0029] inserted. This allows the perhaps dangerous changing of server internal, [0030] request processing dictionary entries. But hey, sysadmins know what they're [0031] doing :-} [0032] [0033] A load factor of 10 collision list entries to 1 hash table element [0034] (empirically) seems to be a performance sweet spot so a table size of 32 means [0035] up to some 300 entries will be efficiently handled (and theoretically, a table [0036] of 512 elements would efficiently support 50k entries - though that's not the [0037] design objective for this purpose). For server request processing purposes the [0038] table size is set at 32 although expected entries would number less than a [0039] couple of dozen (request and response header fields predominating). [0040] [0041] A dictionary built using request memory does not need to be explicitly [0042] destroyed when RequestFreeHeap() is deployed. If the request is just zeroized [0043] then it does need destruction ready for any subsequent request. [0044] [0045] With HTTP/1.n and HTTP/2 having fundamentally different header representations [0046] there was a distinct need for an intermediate abstraction from which these both [0047] could be interfaced with the server-internal requirements. Although the [0048] implementation is quite general purpose, the role of this WASD dictionary is to [0049] act as a repository for request and response header field names and values, [0050] internally used strings (and potentially binary objects), and a new [0051] configuration and request processing (mapping/authorization) abstraction based [0052] on matching key-value pairs. (And it gave me a chance to play with an [0053] associative array implementation :-) [0054] [0055] The implementation is a bog-standard hash array with collision linked-list. An [0056] associated list allows iteration in order of insertion. Each associative array [0057] (dictionary) is dynamically allocated, as are entries in those arrays. As the [0058] entry behaviour seems mostly static after insertion (i.e. values do not change [0059] often if at all) each entry is allocated with space enough for the entry data [0060] structure, key and value. One allocation per insertion. Makes value length [0061] increases a little more complex but that's the tradeoff. Memory is generally [0062] allocated from request heap zones and so disposes of itself when the request [0063] structure is itself disposed of. [0064] [0065] [0066] VERSION HISTORY [0067] --------------- [0068] 13-JAN-2020 MGD bugfix; "tmptr && tmptr->clink.." [0069] 25-FEB-2018 MGD 'uniquifier' delimiter changed from period to colon [0070] 22-DEC-2015 MGD initial [0071] */ [0072] /*****************************************************************************/ [0073] [0074] #ifdef WASD_VMS_V7 [0075] #undef _VMS__V6__SOURCE [0076] #define _VMS__V6__SOURCE [0077] #undef __VMS_VER [0078] #define __VMS_VER 70000000 [0079] #undef __CRTL_VER [0080] #define __CRTL_VER 70000000 [0081] #endif [0082] [0083] #include [0084] #include [0085] [0086] #include "wasd.h" [0087] [0088] #define WASD_MODULE "DICT" [0089] [0090] /******************/ [0091] /* global storage */ [0092] /******************/ [0093] [0094] static uint DictLookupHash, [0095] DictLookupKlen; [0096] [0097] /********************/ [0098] /* external storage */ [0099] /********************/ [0100] [0101] extern int ToLowerCase[], [0102] ToUpperCase[]; [0103] [0104] extern char ErrorSanityCheck []; [0105] [0106] extern CONFIG_STRUCT Config; [0107] extern WATCH_STRUCT Watch; [0108] [0109] /*****************************************************************************/ [0110] /* [0111] Create a new dictionary. [0112] */ [0113] [0114] DICT_STRUCT* DictCreate [0115] ( [0116] REQUEST_STRUCT *rqptr, [0117] int size [0118] ) [0119] { [0120] int idx; [0121] DICT_STRUCT *dicptr; [0122] [0123] /*********/ [0124] /* begin */ [0125] /*********/ [0126] [0127] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0128] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "DictCreate()"); [0129] [0130] if (rqptr) [0131] dicptr = VmGetHeap (rqptr, sizeof(DICT_STRUCT)); [0132] else [0133] dicptr = VmGet (sizeof(DICT_STRUCT)); [0134] dicptr->RequestPtr = rqptr; [0135] [0136] if ((int)(dicptr->size = size) <= 0) dicptr->size = DICT_DEFAULT_SIZE; [0137] if (dicptr->size > 4096) [0138] ErrorExitVmsStatus (SS$_BUGCHECK, ErrorSanityCheck, FI_LI); [0139] dicptr->bytes = sizeof(DICT_ENTRY_STRUCT*) * dicptr->size; [0140] [0141] if (rqptr) [0142] dicptr->table = VmGetHeap (rqptr, dicptr->bytes); [0143] else [0144] dicptr->table = VmGet (dicptr->bytes); [0145] [0146] return (dicptr); [0147] } [0148] [0149] /*****************************************************************************/ [0150] /* [0151] Delete all dictionary items and then the dictionary itself. [0152] Returns a NULL pointer. [0153] */ [0154] [0155] DICT_STRUCT* DictDestroy (DICT_STRUCT *dicptr) [0156] [0157] { [0158] DICT_ENTRY_STRUCT *denptr, *flink; [0159] REQUEST_STRUCT *rqptr; [0160] [0161] /*********/ [0162] /* begin */ [0163] /*********/ [0164] [0165] rqptr = dicptr->RequestPtr; [0166] [0167] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0168] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "DictDestroy()"); [0169] [0170] for (denptr = dicptr->head; denptr != NULL; denptr = flink) [0171] { [0172] flink = denptr->flink; [0173] if (rqptr) [0174] VmFreeFromHeap (rqptr, denptr, FI_LI); [0175] else [0176] VmFree (denptr, FI_LI); [0177] } [0178] [0179] if (rqptr) [0180] VmFreeFromHeap (rqptr, dicptr, FI_LI); [0181] else [0182] VmFree (dicptr, FI_LI); [0183] [0184] return (NULL); [0185] } [0186] [0187] /*****************************************************************************/ [0188] /* [0189] Return the value of the dirty flag and reset. [0190] */ [0191] [0192] uint DictDirty (DICT_STRUCT *dicptr) [0193] [0194] { [0195] uint dirty; [0196] [0197] /*********/ [0198] /* begin */ [0199] /*********/ [0200] [0201] if (dicptr == NULL) return (NULL); [0202] [0203] dirty = dicptr->dirty; [0204] dicptr->dirty = 0; [0205] return (dirty); [0206] } [0207] [0208] /*****************************************************************************/ [0209] /* [0210] Return a pointer to an exact duplicate of the supplied dictionary. If the [0211] request pointer is supplied it is duplicated into that request. This can be [0212] the same request as the dictionary being duplicated or another. If the [0213] request pointer is NULL then it is duplicated into global memory, and of [0214] course that may be where the original dictionary also resides. It can be [0215] duplicated in any combination. [0216] */ [0217] [0218] DICT_STRUCT* DictDuplicate [0219] ( [0220] REQUEST_STRUCT *rqptr, [0221] DICT_STRUCT *dicptr [0222] ) [0223] { [0224] DICT_STRUCT *duptr; [0225] DICT_ENTRY_STRUCT *denptr, *nxtptr; [0226] [0227] /*********/ [0228] /* begin */ [0229] /*********/ [0230] [0231] if (dicptr == NULL) return (NULL); [0232] [0233] duptr = DictCreate (rqptr, dicptr->size); [0234] nxtptr = dicptr->head; [0235] while ((denptr = nxtptr) != NULL) [0236] { [0237] nxtptr = denptr->flink; [0238] DictInsert (duptr, denptr->type, denptr->key, denptr->klen, [0239] denptr->value, denptr->vlen); [0240] } [0241] return (duptr); [0242] } [0243] [0244] /*****************************************************************************/ [0245] /* [0246] Expand the specified entry's available value space by the specified quantity, [0247] copying any existing content to the new memory. As the memory allocation is [0248] likely to change any reference to the entry or its value must be re-set. [0249] */ [0250] [0251] DICT_ENTRY_STRUCT* DictExpandEntry [0252] ( [0253] DICT_ENTRY_STRUCT *denptr, [0254] uint bytes [0255] ) [0256] { [0257] ulong hash; [0258] uchar *cptr, *sptr, *zptr; [0259] DICT_STRUCT *dicptr; [0260] DICT_ENTRY_STRUCT *newptr, *tmptr; [0261] REQUEST_STRUCT *rqptr; [0262] [0263] /*********/ [0264] /* begin */ [0265] /*********/ [0266] [0267] dicptr = denptr->DictPtr; [0268] rqptr = dicptr->RequestPtr; [0269] [0270] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0271] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "DictExpandEntry()"); [0272] [0273] /* this is *additional* value space */ [0274] bytes += denptr->bytes; [0275] [0276] /* plus two terminating nulls and some "elbow-room" */ [0277] if (rqptr) [0278] newptr = VmGetHeap (rqptr, sizeof(DICT_ENTRY_STRUCT) + bytes + 8); [0279] else [0280] newptr = VmGet (sizeof(DICT_ENTRY_STRUCT) + bytes + 8); [0281] [0282] newptr->bytes = bytes; [0283] newptr->hash = hash = denptr->hash; [0284] newptr->type[0] = denptr->type[0]; [0285] newptr->DictPtr = denptr->DictPtr; [0286] [0287] /* reproduce the key */ [0288] newptr->klen = denptr->klen; [0289] zptr = (sptr = newptr->key = newptr->buffer) + newptr->klen; [0290] for (cptr = denptr->key; sptr < zptr; *sptr++ = *cptr++); [0291] [0292] /* reproduce any value (see DictInsert() "just a little too clever") */ [0293] newptr->value = newptr->buffer + newptr->klen + 1; [0294] /* if just reserving space (still valid with an expansion) */ [0295] if (denptr->value == NULL) [0296] newptr->vlen = 0; [0297] else [0298] { [0299] zptr = (sptr = newptr->value) + (newptr->vlen = denptr->vlen); [0300] for (cptr = denptr->value; sptr < zptr; *sptr++ = *cptr++); [0301] } [0302] [0303] /* adjust the hash table / collision list */ [0304] if (dicptr->table[hash] == denptr) [0305] dicptr->table[hash] = newptr; [0306] else [0307] { [0308] for (tmptr = dicptr->table[hash]; [0309] tmptr && tmptr->clink != denptr; [0310] tmptr = tmptr->clink); [0311] tmptr->clink = newptr; [0312] } [0313] newptr->clink = denptr->clink; [0314] [0315] /* adjust the ordered list */ [0316] if (dicptr->head == denptr) dicptr->head = newptr; [0317] if (dicptr->tail == denptr) dicptr->tail = newptr; [0318] if (denptr->flink != NULL) denptr->flink->blink = newptr; [0319] if (denptr->blink != NULL) denptr->blink->flink = newptr; [0320] newptr->flink = denptr->flink; [0321] newptr->blink = denptr->blink; [0322] [0323] /* if necessary adjust the iterator */ [0324] if (dicptr->next == denptr) dicptr->next = newptr; [0325] [0326] if (rqptr) [0327] VmFreeFromHeap (rqptr, denptr, FI_LI); [0328] else [0329] VmFree (denptr, FI_LI); [0330] [0331] return (newptr); [0332] } [0333] [0334] /*****************************************************************************/ [0335] /* [0336] Extract the specified entry from its dictionary. [0337] */ [0338] [0339] DICT_ENTRY_STRUCT* DictExtractEntry (DICT_ENTRY_STRUCT *denptr) [0340] [0341] { [0342] uint hash; [0343] DICT_STRUCT *dicptr; [0344] DICT_ENTRY_STRUCT *tmptr; [0345] REQUEST_STRUCT *rqptr; [0346] [0347] /*********/ [0348] /* begin */ [0349] /*********/ [0350] [0351] dicptr = denptr->DictPtr; [0352] rqptr = dicptr->RequestPtr; [0353] [0354] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0355] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "DictExtractEntry()"); [0356] [0357] /* extract from the hash table / collision list */ [0358] hash = denptr->hash; [0359] if (dicptr->table[hash] == denptr) [0360] dicptr->table[hash] = denptr->clink; [0361] else [0362] { [0363] for (tmptr = dicptr->table[hash]; [0364] tmptr && tmptr->clink != denptr; [0365] tmptr = tmptr->clink); [0366] tmptr->clink = denptr->clink; [0367] } [0368] [0369] /* remove from the ordered list */ [0370] if (dicptr->head == denptr) dicptr->head = denptr->flink; [0371] if (dicptr->tail == denptr) dicptr->tail = denptr->blink; [0372] if (denptr->flink != NULL) denptr->flink->blink = denptr->blink; [0373] if (denptr->blink != NULL) denptr->blink->flink = denptr->flink; [0374] [0375] /* if necessary adjust the iterator */ [0376] if (dicptr->next == denptr) dicptr->next = denptr->flink; [0377] [0378] dicptr->count--; [0379] dicptr->bytes -= denptr->bytes; [0380] dicptr->dirty++; [0381] [0382] /* for detecting any subsequent (re)insertion */ [0383] denptr->extract = dicptr->size; [0384] [0385] return (denptr); [0386] } [0387] [0388] /*****************************************************************************/ [0389] /* [0390] Insert the specified key and value into the specified dictionary. If the key [0391] already exists then that entry will be removed and replaced with this one. If [0392] the type has the DICT_TYPE_UNIQUE bit set then unique entries are created by [0393] prefixing the key with a number 1..n and period (e.g. "10:example"). Generally [0394] the digits and colon leading the supplied key need to be excised before use. [0395] */ [0396] [0397] DICT_ENTRY_STRUCT* DictInsert [0398] ( [0399] DICT_STRUCT *dicptr, [0400] uchar *type, [0401] uchar *key, [0402] int klen, [0403] uchar *value, [0404] int vlen [0405] ) [0406] { [0407] static char type2 [2]; [0408] static char KeyBuf [256]; [0409] static $DESCRIPTOR (KeyBufDsc, KeyBuf); [0410] static $DESCRIPTOR (KeyFaoDsc, "!UL:!#AZ"); [0411] [0412] BOOL reuse; [0413] int cnt, len; [0414] ushort slen; [0415] ulong bytes, hash; [0416] uchar *cptr, *sptr, *zptr; [0417] DICT_ENTRY_STRUCT *denptr, *tmptr; [0418] REQUEST_STRUCT *rqptr; [0419] [0420] /*********/ [0421] /* begin */ [0422] /*********/ [0423] [0424] rqptr = dicptr->RequestPtr; [0425] [0426] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0427] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, [0428] "DictInsert() !&C !SL !#AZ = !SL !#AZ", [0429] type[0], klen, klen, key, vlen, vlen, value); [0430] [0431] /* cannot use an empty key */ [0432] if (klen == 0 || !key[0]) return (NULL); [0433] [0434] if (vlen < 0) [0435] { [0436] for (cptr = value; *cptr; cptr++); [0437] vlen = cptr - value; [0438] } [0439] [0440] /* buffer a type that's free of any potential "uniquifier bit" */ [0441] type2[0] = type[0] & ~DICT_TYPE_UNIQUE; [0442] [0443] denptr = DictLookup (dicptr, type2, key, klen); [0444] [0445] /* cannot use a silly (buggy?) sized key (allowing for enumeration) */ [0446] if ((klen = DictLookupKlen) > sizeof(KeyBuf)-8) return (NULL); [0447] [0448] /* generated by DictLookup() no need to do it again */ [0449] hash = DictLookupHash; [0450] [0451] if (denptr != NULL) [0452] { [0453] /* entry already exists */ [0454] if ((type[0] & DICT_TYPE_UNIQUE) && [0455] type2[0] != DICT_TYPE_OPAQUE_KEY) [0456] { [0457] /* enumerated (unique) entries */ [0458] sys$fao (&KeyFaoDsc, &slen, &KeyBufDsc, ++dicptr->unique, klen, key); [0459] KeyBuf[slen] = '\0'; [0460] key = KeyBuf; [0461] klen = slen; [0462] /* compute the hash */ [0463] hash = type2[0]; [0464] for (cptr = key; slen--; cptr++) hash = hash * DICT_HASH_MULT + *cptr; [0465] hash = hash % dicptr->size; [0466] /* need a fresh one */ [0467] denptr = NULL; [0468] reuse = false; [0469] } [0470] else [0471] { [0472] /* can the entry's value buffer be reused */ [0473] if (klen + vlen > denptr->bytes) [0474] { [0475] /* won't fit, expand using new value (just a little too clever) */ [0476] denptr->value = value; [0477] denptr->vlen = vlen; [0478] denptr = DictExpandEntry (denptr, (klen + vlen) - denptr->bytes); [0479] return (denptr); [0480] } [0481] /* existing entry can contain the new value */ [0482] reuse = true; [0483] } [0484] } [0485] [0486] /* if entry did not already exist allocate and initialise a fresh one */ [0487] if (denptr == NULL) [0488] { [0489] bytes = klen + vlen; [0490] /* to (perhaps) improve memory management impose a minimum sized chunk */ [0491] if (bytes < DICT_MIN_BYTES) bytes = DICT_MIN_BYTES; [0492] [0493] /* plus two terminating nulls and some "elbow-room" */ [0494] if (rqptr) [0495] denptr = VmGetHeap (rqptr, sizeof(DICT_ENTRY_STRUCT) + bytes + 8); [0496] else [0497] denptr = VmGet (sizeof(DICT_ENTRY_STRUCT) + bytes + 8); [0498] [0499] denptr->bytes = bytes; [0500] denptr->hash = hash; [0501] denptr->type[0] = type2[0]; [0502] denptr->DictPtr = dicptr; [0503] [0504] denptr->klen = klen; [0505] zptr = (sptr = denptr->key = denptr->buffer) + klen; [0506] /* push key to lower-case! (no null required in zeroed memory) */ [0507] for (cptr = key; sptr < zptr; *sptr++ = TOLO(*cptr++)); [0508] reuse = false; [0509] } [0510] [0511] denptr->value = denptr->buffer + denptr->klen + 1; [0512] /* if the value is NULL then just reserving space */ [0513] if (value == NULL) [0514] denptr->vlen = 0; [0515] else [0516] { [0517] zptr = (sptr = denptr->value) + (denptr->vlen = vlen); [0518] for (cptr = value; sptr < zptr; *sptr++ = *cptr++); [0519] /* in case a shorter value is being overwritten */ [0520] *sptr = '\0'; [0521] } [0522] [0523] if (reuse) [0524] dicptr->dirty++; [0525] else [0526] DictInsertEntry (dicptr, denptr); [0527] [0528] return (denptr); [0529] } [0530] [0531] /*****************************************************************************/ [0532] /* [0533] Insert the specified entry into the specified dictionary. [0534] */ [0535] [0536] DICT_ENTRY_STRUCT* DictInsertEntry [0537] ( [0538] DICT_STRUCT *dicptr, [0539] DICT_ENTRY_STRUCT *denptr [0540] ) [0541] { [0542] int len; [0543] ulong hash; [0544] uchar *cptr; [0545] DICT_ENTRY_STRUCT *tmptr; [0546] REQUEST_STRUCT *rqptr; [0547] [0548] /*********/ [0549] /* begin */ [0550] /*********/ [0551] [0552] rqptr = dicptr->RequestPtr; [0553] [0554] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0555] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "DictInsertEntry()"); [0556] [0557] /* if extracted from a(nother) dictionary */ [0558] if (denptr->extract) [0559] { [0560] /* check that the memory belongs to the same environment */ [0561] if (dicptr->RequestPtr != denptr->DictPtr->RequestPtr) [0562] ErrorExitVmsStatus (SS$_BUGCHECK, ErrorSanityCheck, FI_LI); [0563] [0564] /* check if there's an existing entry and remove as required */ [0565] if (tmptr = DictLookup (dicptr, denptr->type, denptr->key, denptr->klen)) [0566] DictRemoveEntry (tmptr); [0567] [0568] /* we've certainly done this! */ [0569] denptr->DictPtr = dicptr; [0570] [0571] if (denptr->extract != dicptr->size); [0572] { [0573] /* need to recalculate the hash */ [0574] hash = denptr->type[0]; [0575] len = denptr->klen; [0576] for (cptr = denptr->key; len--; cptr++) [0577] hash = hash * DICT_HASH_MULT + *cptr; [0578] denptr->hash = hash % dicptr->size; [0579] } [0580] [0581] /* reset the extraction indicator (size of original dictionary) */ [0582] denptr->extract = 0; [0583] } [0584] [0585] /* insert into the hash table / collision list */ [0586] hash = denptr->hash; [0587] if (dicptr->table[hash] == NULL) [0588] dicptr->table[hash] = denptr; [0589] else [0590] { [0591] for (tmptr = dicptr->table[hash]; [0592] tmptr && tmptr->clink != NULL; [0593] tmptr = tmptr->clink); [0594] tmptr->clink = denptr; [0595] } [0596] /* in case it's been extracted from another dictionary */ [0597] denptr->clink = NULL; [0598] [0599] /* append to the tail of the ordered list */ [0600] if (dicptr->head == NULL) dicptr->head = denptr; [0601] if (dicptr->tail != NULL) [0602] { [0603] dicptr->tail->flink = denptr; [0604] denptr->blink = dicptr->tail; [0605] } [0606] dicptr->tail = denptr; [0607] /* in case it's been extracted from another dictionary */ [0608] denptr->flink = NULL; [0609] [0610] dicptr->dirty++; [0611] dicptr->count++; [0612] dicptr->bytes += sizeof(DICT_ENTRY_STRUCT) + denptr->bytes; [0613] [0614] return (denptr); [0615] } [0616] [0617] /*****************************************************************************/ [0618] /* [0619] Iterate through dictionary entries. List is ordered first to last insertion. [0620] Second parameter as NULL resets to head, "*" selects all, "\0" those of [0621] that type, and "\0" a string match. Return a pointer to the entry [0622] or NULL on exhaustion or empty dictionary. On reset the returned pointer [0623] should only be considered informational as to empty or not. [0624] */ [0625] [0626] DICT_ENTRY_STRUCT* DictIterate [0627] ( [0628] DICT_STRUCT *dicptr, [0629] uchar *which [0630] ) [0631] { [0632] DICT_ENTRY_STRUCT *denptr; [0633] [0634] /*********/ [0635] /* begin */ [0636] /*********/ [0637] [0638] if (which != NULL) [0639] { [0640] while ((denptr = dicptr->next) != NULL) [0641] { [0642] dicptr->next = denptr->flink; [0643] if (!which[0] || MATCH2 (which, "*")) break; [0644] if (which[0] && !which[1]) [0645] if (denptr->type[0] == which[0]) [0646] break; [0647] else [0648] continue; [0649] if (StringMatch (NULL, denptr->key, which)) break; [0650] } [0651] return (denptr); [0652] } [0653] [0654] /* reset to start */ [0655] return (dicptr->next = dicptr->head); [0656] } [0657] [0658] /*****************************************************************************/ [0659] /* [0660] Look in the specified dictionary for the specified key. If key length is -1 [0661] then assume key is null-terminated string, otherwise some other opaque blob. [0662] String key matching is case-insensitive. Lookup includes the entry type. [0663] Return a pointer to the dictionary item found or NULL if not. [0664] */ [0665] [0666] DICT_ENTRY_STRUCT* DictLookup [0667] ( [0668] DICT_STRUCT *dicptr, [0669] uchar *type, [0670] uchar *key, [0671] int klen [0672] ) [0673] { [0674] uint len; [0675] ulong hash; [0676] uchar *cptr, *sptr, *zptr; [0677] DICT_ENTRY_STRUCT *denptr; [0678] REQUEST_STRUCT *rqptr; [0679] [0680] /*********/ [0681] /* begin */ [0682] /*********/ [0683] [0684] rqptr = dicptr->RequestPtr; [0685] [0686] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0687] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, [0688] "DictLookup() !&C !SL !#AZ", type[0], klen, klen, key); [0689] [0690] /* cannot use an empty key */ [0691] if (klen == 0 || (klen < 0 && !key[0])) return (NULL); [0692] [0693] /* make the type character part of the hash */ [0694] hash = type[0]; [0695] if (klen < 0) [0696] { [0697] for (cptr = key; *cptr; cptr++) hash = hash * DICT_HASH_MULT + *cptr; [0698] klen = cptr - key; [0699] } [0700] else [0701] { [0702] len = klen; [0703] for (cptr = key; len--; cptr++) hash = hash * DICT_HASH_MULT + *cptr; [0704] } [0705] hash = hash % dicptr->size; [0706] [0707] /* save these so that they don't need to be generated again */ [0708] DictLookupHash = hash; [0709] DictLookupKlen = klen; [0710] [0711] zptr = key + klen; [0712] for (denptr = dicptr->table[hash]; denptr != NULL; denptr = denptr->clink) [0713] { [0714] if (denptr->type[0] != type[0]) continue; [0715] if (denptr->klen != klen) continue; [0716] cptr = key; [0717] sptr = denptr->key; [0718] if (denptr->type[0] == DICT_TYPE_OPAQUE_KEY) [0719] while (cptr < zptr && *cptr == *sptr) { cptr++; sptr++; } [0720] else [0721] /* all keys are pushed to lower-case on insertion */ [0722] while (cptr < zptr && TOLO(*cptr) == *sptr) { cptr++; sptr++; } [0723] if (cptr == zptr && sptr == denptr->key + denptr->klen) break; [0724] } [0725] [0726] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0727] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "denptr:!&X", denptr); [0728] [0729] return (denptr); [0730] } [0731] [0732] /*****************************************************************************/ [0733] /* [0734] Delete the specified key from the dictionary. Returns NULL if the key was not [0735] found or a pointer to the entry if the key was found. The pointer is [0736] informational and must not subsequently be accessed. [0737] */ [0738] [0739] DICT_ENTRY_STRUCT* DictRemove [0740] ( [0741] DICT_STRUCT *dicptr, [0742] uchar *type, [0743] uchar *key, [0744] int klen [0745] ) [0746] { [0747] uint hash; [0748] DICT_ENTRY_STRUCT *denptr; [0749] REQUEST_STRUCT *rqptr; [0750] [0751] /*********/ [0752] /* begin */ [0753] /*********/ [0754] [0755] rqptr = dicptr->RequestPtr; [0756] [0757] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0758] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, [0759] "DictRemove() !AZ !&Z", type, key); [0760] [0761] if ((denptr = DictLookup (dicptr, type, key, klen)) == NULL) return (NULL); [0762] [0763] DictExtractEntry (denptr); [0764] [0765] if (rqptr) [0766] VmFreeFromHeap (rqptr, denptr, FI_LI); [0767] else [0768] VmFree (denptr, FI_LI); [0769] [0770] return (denptr); [0771] } [0772] [0773] /*****************************************************************************/ [0774] /* [0775] Delete the specified entry from the dictionary. Returns the freed pointer, [0776] which is informational and must not subsequently be accessed. [0777] */ [0778] [0779] DICT_ENTRY_STRUCT* DictRemoveEntry (DICT_ENTRY_STRUCT *denptr) [0780] [0781] { [0782] DICT_STRUCT *dicptr; [0783] REQUEST_STRUCT *rqptr; [0784] [0785] /*********/ [0786] /* begin */ [0787] /*********/ [0788] [0789] dicptr = denptr->DictPtr; [0790] rqptr = dicptr->RequestPtr; [0791] [0792] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0793] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "DictRemoveEntry()"); [0794] [0795] DictExtractEntry (denptr); [0796] [0797] if (rqptr) [0798] VmFreeFromHeap (rqptr, denptr, FI_LI); [0799] else [0800] VmFree (denptr, FI_LI); [0801] [0802] return (denptr); [0803] } [0804] [0805] /*****************************************************************************/ [0806] /* [0807] Set the value length of the specified entry. Check the size is within the [0808] storage allocated. Adjustments to the value string can be made (within [0809] constraints) because it's stored at the trailing end of the buffer space [0810] following the key. [0811] */ [0812] [0813] void DictValueLength [0814] ( [0815] DICT_ENTRY_STRUCT *denptr, [0816] uint length [0817] ) [0818] { [0819] DICT_STRUCT *dicptr; [0820] REQUEST_STRUCT *rqptr; [0821] [0822] /*********/ [0823] /* begin */ [0824] /*********/ [0825] [0826] dicptr = denptr->DictPtr; [0827] rqptr = dicptr->RequestPtr; [0828] [0829] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0830] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "DictValueLength()"); [0831] [0832] if (denptr->klen + length > denptr->bytes) [0833] ErrorExitVmsStatus (SS$_BUGCHECK, ErrorSanityCheck, FI_LI); [0834] [0835] denptr->vlen = length; [0836] } [0837] [0838] /*****************************************************************************/ [0839] /* [0840] WATCH the contents of the dictionary. If |which| is NULL all entries are [0841] reported. If |which| is a single character string then it should be an entry [0842] type character and only matching entries are reported. If |which| is a string [0843] beginning and entry type character and followed by a key string then report [0844] only that entry. [0845] */ [0846] [0847] void DictWatch [0848] ( [0849] DICT_STRUCT *dicptr, [0850] uchar *type, [0851] uchar *which [0852] ) [0853] { [0854] char *cptr; [0855] DICT_ENTRY_STRUCT *denptr; [0856] REQUEST_STRUCT *rqptr; [0857] [0858] /*********/ [0859] /* begin */ [0860] /*********/ [0861] [0862] rqptr = dicptr->RequestPtr; [0863] [0864] if (which == NULL || (type == NULL && which[0] == '*')) [0865] { [0866] WatchThis (WATCHITM(rqptr), WATCH_INTERNAL, [0867] "DICTIONARY size:!UL count:!UL bytes:!UL", [0868] dicptr->size, dicptr->count, dicptr->bytes); [0869] DictWatchEntry (NULL); [0870] } [0871] [0872] if (which && which[1]) [0873] { [0874] if (type == NULL) type = DICT_TYPE_INTERNAL; [0875] denptr = DictLookup (rqptr->rqDictPtr, type, which, -1); [0876] if (denptr != NULL) DictWatchEntry (denptr); [0877] return; [0878] } [0879] [0880] for (denptr = dicptr->head; denptr != NULL; denptr = denptr->flink) [0881] if (which == NULL || type == NULL || type[0] == denptr->type[0]) [0882] DictWatchEntry (denptr); [0883] } [0884] [0885] /*****************************************************************************/ [0886] /* [0887] WATCH the specified entry. Entry pointer as NULL resets the counter. [0888] */ [0889] [0890] void DictWatchEntry (DICT_ENTRY_STRUCT *denptr) [0891] [0892] { [0893] static int count = 0; [0894] [0895] char *cptr, *sptr; [0896] DICT_STRUCT *dicptr; [0897] [0898] /*********/ [0899] /* begin */ [0900] /*********/ [0901] [0902] if (denptr == NULL) [0903] { [0904] count = 0; [0905] return; [0906] } [0907] [0908] dicptr = denptr->DictPtr; [0909] [0910] if (denptr == dicptr->table[denptr->hash]) [0911] cptr = "[]"; [0912] else [0913] cptr = ".."; [0914] [0915] if (denptr->type[0] == DICT_TYPE_OPAQUE || [0916] denptr->type[0] == DICT_TYPE_OPAQUE_KEY) [0917] WatchDataFormatted ("ENTRY !3ZL !&C!3ZL!&C !&C {!UL}!-!#&H={!UL}!-!#&H\n", [0918] ++count, cptr[0], denptr->hash, cptr[1], [0919] denptr->type[0], denptr->klen, denptr->key, [0920] denptr->vlen, denptr->value); [0921] else [0922] { [0923] if (denptr->vlen && denptr->value[denptr->vlen-1] == '\n') [0924] sptr = ""; [0925] else [0926] sptr = "\n"; [0927] WatchDataFormatted ("ENTRY !3ZL !&C!3ZL!&C !&C {!UL}!-!#AZ={!UL}!-!#AZ!AZ", [0928] ++count, cptr[0], denptr->hash, cptr[1], [0929] denptr->type[0], denptr->klen, denptr->key, [0930] denptr->vlen, denptr->value, sptr); [0931] } [0932] } [0933] [0934] /*****************************************************************************/ [0935] /* [0936] Exercise the dictionary functions. Provide some indicative timings. [0937] fname= should be a list of words such as found at: [0938] https://raw.githubusercontent.com/first20hours/google-10000-english/master/20k.txt [0939] [0940] Some results on an Alpha PWS500 OpenVMS V8.3. [0941] [0942] $! 8 hash, 1,000 additions results in a rate of 85k insertions/sec [0943] $ httpd /tests dictionary=size=8,count=10000,fname=20kwords.txt [0944] 8< snip 8< [0945] size:8 count:1000 max:0 min:0 [0946] total:1000 klen0:62 failures:62 duplicates:53 expands:32 deletions:0 duration:0.011718 85338.797/sec 0.000012/each [0947] table:[0]100 [1]99 [2]123 [3]119 [4]108 [5]109 [6]114 [7]103 [0948] [0949] $! 16 hash, 1,000 additions results in 102k insertions/sec [0950] $ httpd /tests dictionary=size=16,count=10000,fname=20kwords.txt [0951] 8< snip 8< [0952] size:16 count:1000 max:0 min:0 [0953] total:1000 klen0:57 failures:57 duplicates:52 expands:49 deletions:0 duration:0.009765 102406.555/sec 0.000010/each [0954] table:[0]57 [1]59 [2]61 [3]55 [4]48 [5]67 [6]51 [7]57 [8]56 [9]52 [10]44 [11]67 [12]51 [13]63 [14]39 [15]51 [0955] [0956] $! 32 hash, 100 additions, results in 114k insertions/sec [0957] $ httpd /tests dictionary=size=32,count=1000,fname=20kwords.txt [0958] 8< snip 8< [0959] size:32 count:1000 max:0 min:0 [0960] total:1000 klen0:58 failures:58 duplicates:46 expands:46 deletions:0 duration:0.008788 113785.062/sec 0.000009/each [0961] table:[0]28 [1]37 [2]25 [3]23 [4]30 [5]32 [6]31 [7]23 [8]30 [9]26 [10]28 [11]23 [12]25 [13]29 [14]27 [15]27 [16]28 [17]25 [18]27 [0962] [19]34 [20]30 [21]33 [22]19 [23]29 [24]25 [25]29 [26]27 [27]30 [28]22 [29]22 [30]35 [31]28 [0963] [0964] $! 64 hash, 100 additions, results in 114k insertions/sec [0965] $ httpd /tests dictionary=size=32,count=1000,fname=20kwords.txt [0966] 8< snip 8< [0967] size:64 count:1000 max:0 min:0 [0968] total:1000 klen0:60 failures:60 duplicates:51 expands:39 deletions:0 duration:0.008788 113785.062/sec 0.000009/each [0969] table:[0]14 [1]12 [2]13 [3]20 [4]17 [5]12 [6]14 [7]12 [8]13 [9]13 [10]16 [11]10 [12]11 [13]20 [14]13 [15]16 [16]14 [17]12 [18]10 [0970] [19]14 [20]16 [21]15 [22]16 [23]8 [24]15 [25]9 [26]15 [27]10 [28]11 [29]14 [30]9 [31]11 [32]10 [33]16 [34]9 [35]13 [36]10 [37]15 [0971] [38]12 [39]15 [40]5 [41]17 [42]14 [43]13 [44]21 [45]20 [46]18 [47]20 [48]17 [49]12 [50]16 [51]12 [52]10 [53]14 [54]12 [55]14 [0972] [56]12 [57]17 [58]15 [59]18 [60]13 [61]17 [62]16 [63]13 [0973] */ [0974] [0975] #if WATCH_MOD [0976] [0977] void DictTest (char *cliptr) [0978] [0979] { [0980] #define KEY_SIZE 31 [0981] #define KEY_VARY(ran) (ran & 0x1f) [0982] #define VALUE_SIZE 127 [0983] #define VALUE_VARY(ran) (ran & 0x7f) [0984] [0985] #include [0986] #include [0987] [0988] int count, deletions, duplicates, expands, failures, [0989] klen, klen0, max, min, size, status, total, vlen; [0990] #ifdef __VAX [0991] ulong random; [0992] #else [0993] __int64 random; [0994] #endif [0995] ulong DeltaTime [2], [0996] NowTime [2], [0997] ThenTime [2]; [0998] uchar ch; [0999] uchar *bptr, *cptr, *fname, *sptr, *wptr, *zptr; [1000] uchar key [KEY_SIZE+1], [1001] value [VALUE_SIZE+1]; [1002] float fsecs; [1003] stat_t statbuf; [1004] FILE *fptr; [1005] DICT_STRUCT *dicptr; [1006] DICT_ENTRY_STRUCT *denptr; [1007] REQUEST_STRUCT *rqptr; [1008] [1009] /*********/ [1010] /* begin */ [1011] /*********/ [1012] [1013] for (cptr = cliptr; *cptr; cptr++) *cptr = tolower(*cptr); [1014] [1015] size = 0; [1016] cptr = strstr (cliptr, "size="); [1017] if (cptr) size = atoi(cptr+5); [1018] if (size <= 0) size = DICT_DEFAULT_SIZE; [1019] [1020] count = 0; [1021] cptr = strstr (cliptr, "count="); [1022] if (cptr) count = atoi(cptr+6); [1023] if (count <= 0) count = size * 10; [1024] [1025] /* maximum number of entries before a deletion cycle */ [1026] max = 0; [1027] cptr = strstr (cliptr, "max="); [1028] if (cptr) max = atoi(cptr+4); [1029] [1030] /* minimum number of entries at end of delete cycle */ [1031] min = 0; [1032] cptr = strstr (cliptr, "min="); [1033] if (cptr) min = atoi(cptr+4); [1034] if (min < 0) min = 32; [1035] if (max < min) max = min * 2; [1036] [1037] cptr = strstr (cliptr, "fname="); [1038] if (cptr) fname = cptr+6; [1039] [1040] /* read in the file of strings */ [1041] if (stat (fname, &statbuf) < 0) exit (vaxc$errno); [1042] if ((fptr = fopen (fname, "r")) == NULL) exit (vaxc$errno); [1043] bptr = calloc (1, statbuf.st_size); [1044] if ((wptr = bptr) == NULL) exit (vaxc$errno); [1045] while (fgets (wptr, 256, fptr) != NULL) [1046] { [1047] while (*wptr && *wptr != '\n') wptr++; [1048] *wptr++ = ' '; [1049] *wptr = '\0'; [1050] } [1051] fclose (fptr); [1052] if (!*bptr) exit (SS$_ABORT); [1053] [1054] VmRequestInit (); [1055] rqptr = VmGetRequest (999999999); [1056] if (Watch.CliEnabled) [1057] { [1058] Watch.Category = Watch.Module = -1; [1059] WatchSetWatch (rqptr, WATCH_NEW_ITEM); [1060] } [1061] [1062] dicptr = DictCreate (rqptr, size); [1063] [1064] sys$gettim (&ThenTime); [1065] random = ThenTime[0] & ThenTime[1]; [1066] [1067] deletions = duplicates = expands = failures = klen0 = 0; [1068] for (total = 0; total < count; total++) [1069] { [1070] random = random * 69069 + 1; [1071] /* ensure 5% (or so) are duplicate keys to exercise that aspect */ [1072] if (random % 21) [1073] { [1074] zptr = sptr = key; [1075] zptr += KEY_VARY(random) - 1; /* vary size */ [1076] while (sptr < zptr) [1077] { [1078] if (!*wptr) wptr = bptr; [1079] *sptr++ = *wptr++; [1080] } [1081] *sptr = '\0'; [1082] klen = sptr - key; [1083] } [1084] else [1085] duplicates++; [1086] [1087] /* generate random value */ [1088] random = random * 69069 + 1; [1089] zptr = sptr = value; [1090] zptr += VALUE_VARY(random) - 1; /* vary size */ [1091] while (sptr < zptr) [1092] { [1093] if (!*wptr) wptr = bptr; [1094] *sptr++ = *wptr++; [1095] } [1096] *sptr = '\0'; [1097] vlen = sptr - value; [1098] [1099] /* insert the key into the dictionary either of the two calls */ [1100] if (random % 2) [1101] denptr = DictInsert (dicptr, DICT_TYPE_INTERNAL, key, -1, value, -1); [1102] else [1103] denptr = DictInsert (dicptr, DICT_TYPE_INTERNAL, key, klen, value, vlen); [1104] [1105] /* a zero length key should fail */ [1106] if (denptr == NULL) failures++; [1107] if (klen == 0) klen0++; [1108] [1109] /* ensure another small percentage are expanded by 100% */ [1110] if ((random % 21) == 0) [1111] if (denptr != NULL) [1112] { [1113] DictExpandEntry (denptr, DICT_GET_BYTES(denptr) * 2); [1114] expands++; [1115] } [1116] [1117] /* if no deletion cycle */ [1118] if (!(max && min)) [1119] if (total < count) [1120] continue; [1121] else [1122] break; [1123] [1124] if (dicptr->count > max || total >= count) [1125] { [1126] DictIterate (dicptr, NULL); [1127] while (dicptr->count > min) [1128] { [1129] /* trim back down to min by choosing entries at random */ [1130] random = random * 69069 + 1; [1131] for (int cnt = (random & 0xf) + 1; cnt; cnt--) [1132] { [1133] while (denptr = DictIterate (dicptr, "*")); [1134] if (denptr) break; [1135] if ((denptr = DictIterate (dicptr, NULL)) == NULL) break; [1136] } [1137] if (denptr == NULL) break; [1138] if (dicptr->count % 2) [1139] DictRemove (dicptr, DICT_TYPE_INTERNAL, denptr->key, -1); [1140] else [1141] DictRemove (dicptr, DICT_TYPE_INTERNAL, [1142] denptr->key, denptr->klen); [1143] deletions++; [1144] } [1145] if (denptr == NULL) break; [1146] } [1147] if (denptr == NULL) break; [1148] } [1149] [1150] sys$gettim (&NowTime); [1151] status = lib$sub_times (&NowTime, &ThenTime, &DeltaTime); [1152] fsecs = FloatDeltaTime (&DeltaTime); [1153] if (VMSnok (status)) ErrorExitVmsStatus (status, NULL, FI_LI); [1154] [1155] Watch.StdoutOnly = true; [1156] Watch.Category = WATCH_INTERNAL; [1157] DictWatch (dicptr, NULL, NULL); [1158] [1159] fprintf (stdout, [1160] "size:%d count:%d max:%d min:%d\n\ [1161] total:%d klen0:%d failures:%d duplicates:%d expands:%d deletions:%d \ [1162] duration:%s %3.3f/sec %f/each\n", [1163] size, count, max, min, [1164] total, klen0, failures, duplicates, expands, deletions, [1165] DurationString(rqptr,&DeltaTime), [1166] (float)total/fsecs, fsecs/(float)total); [1167] [1168] fputs ("table:", stdout); [1169] for (int idx = 0; idx < dicptr->size; idx++) [1170] { [1171] int cnt = 0; [1172] for (denptr = dicptr->table[idx]; denptr != NULL; denptr = denptr->clink) [1173] cnt++; [1174] fprintf (stdout, "[%d]%d ", idx, cnt); [1175] } [1176] fputs ("\n", stdout); [1177] [1178] DictDestroy (dicptr); [1179] } [1180] [1181] #endif /* WATCH_MOD */ [1182] [1183] /*****************************************************************************/ [1184]