/*****************************************************************************/ /* Cache.c This module implements both a file data and revision time cache, and a general non-file (e.g. script) output cache. With file content caching the file revision time is stored (along with the directory and file IDs for efficiency) allowing the entry validity to expire and to be periodically revalidated against the on-disk version. If the on-disk version has changed the cache is purged and the content reloaded, otherwise the expiry period just recommences. For non-file content the revalidation is not possible so when the validity of the cache entry expires the entry is purged from the cache forcing it to be reloaded through full request processing. The cache has limits imposed on it. A maximum number of files that can be cached at any one time, and a maximum amount of memory that can be allocated in total by the cache. Each of these two operates to limit the other. In addition the maximum size of a file that can be loaded into the cache can be specified. Fine control on cache behaviour is exercised using mapping rules. See the MAPURL.C module for a list of those applicable. FILE CONTENT ------------ File content cache data is loaded by the file module while concurrently transfering that data to the request client, using buffer space supplied by cache module, space that can then be retained for reuse as cache. Hence the cache load adds no significant overhead to the actual reading and initial transfer of the file. An entirely accurate estimate of the quantity of cache memory required for file content can be derived from the file header. Cache loads that exceed the configuration or rule mapping maxima do not proceed. NON-FILE CONTENT ---------------- Non-file content is a little different from file content in that the size of the cache memory required is generally not known when the cache load begins (content-length is often not supplied with script output for example). When this is not known the cache module just allocates an ambit quantity of memory, determined by configuration or rule mapping indicated maxima. When the load successfully completes the actual memory required is compared to that originally allocated and reduced as necessary. If non-file output exceeds the original ambit memory allocation (and hence configuration or rule maxima) the data is just discarded at end-of-request. Non-file content entries are sourced in two ways. First, from the CGI module. This can discriminate and load either (or both) CGI-compliant responses and Non-Parse-Header (NPH) responses. The former allows the server to generate the response header, the latter contains the response header as the leading output from the script. Cached CGI responses generate a new header with each subsequent use, while cached NPH response reuse the original header. The second source of cached response is direct from network-to-client output via the NET module. Like scripts, network loaded content can contain only the body of the response, or the full header and body (viz. NPH). Some care should be excercised with the caching of responses using the NET source. By default only GET requests, without query strings, generating responses with a success HTTP status (200) will be cached. Other response status content will be discarded. Requests containing query strings may be cached by using the 'cache=query' mapping rule. This sort of caching should be done carefully and selectively. The multitude of differing query string usually accepted by such resources would soon fill the cache with generally non-repeated data and thus render the cache completely ineffective. BYTE RANGES ----------- See comments in FILE.C module. TERMINOLOGY ----------- "hit" refers to a request path being found in cache. If the data is still valid the request can be supplied from cache. "load"ing the cache refers to reading the contents of a file into cache memory. "valid" means that the file from which the cached data was originally read has not had it's revision date changed (the implication being is the file contents have not changed. WHY IMPLEMENT CACHING? ---------------------- Caching, be definition, attempts to improve performance by keeping data in storage that is faster to access than it's usual location. The performance improvement can be assessed in three basic ways. Reduction in latency when accessing the data, of processing involved, and in impact on the usual storage location. This cache is provided to address all three. Where networks are particularly responsive a reduction in request latency can often be noticable. Where servers are particularly busy or where disk systems particularly loaded a reduction in the need to access the file system can significantly improve performance. My suggestion is though, that for most VMS sites high levels of hits are not a great concern, and for these caching can easily be left disabled. CACHE SUITABILITY CONSIDERATIONS -------------------------------- A cache is not always of benefit! It's cost *may* outweigh it's return. Any cache's efficiencies can only occur where subsets of data are consistently being demanded. Although these subsets may change slowly over time a consistent and rapidly changing aggregate of requests lose the benefit of more readily accessable data to the overhead of cache management due to the constant and continuous flushing and reloading of cache data. This server's cache is no different, it will only improve performance if the site experiences some consistency in the files requested. For sites that have only a small percentage of files being repeatedly requested it is probably better that the cache be disabled. The other major consideration is available system memory. On a system where memory demand is high there is little value in having cache memory sitting in page space, trading disk I/O and latency for paging I/O and latency. On memory-challenged systems cache is probably best disabled. With "loads not hit", the count represents the cumulative number of files loaded but never subsequently hit. If this percentage is high it means most files loaded are never hit, indicating the site's request profile is possibly unsuitable for caching. The item "hits" respresents the cumulative, total number of hits against the cumulative, total number of loads. The percentage here can range from zero to many thousands of percent :^) with less than 100% indicating poor cache performance, from 200% upwards better and good performance. The items "1-9", "10-99" and "100+" show the count and percentage of total hits that occured when a given entry had experienced hits within that range (e.g. if an entry has had 8 previous hits, the ninth increments the "1-9" item whereas the tenth and eleventh increments the "10-99" item, etc.) Other considerations also apply when assessing the benefit of having a cache. For example, a high number and percentage of hits can be generated while the percentage of "loads not hit" could be in the also be very high. The explanation for this would be one or two frequently requested files being hit while most others are loaded, never hit, and flushed as other files request cache space. In situations such as this it is difficult to judge whether cache processing is improving performance or just adding overhead. Again, my suggestion is, that for most VMS sites, high levels of access are not a great concern, and for these caching can easily be left disabled. DESCRIPTION ----------- An MD5 digest (16 byte hash) is used to uniquely identify cached files. The hash is generated either from a mapped path or from the file name before calling any of the cache search, load or completion functions. This MD5 hash is then stored along with the file details and contents so that identical paths/files can be matched during subsequent searches. The MD5 algorithm "guarantees" a unique identifier for any resource name, and the 16 byte size makes matching using just 4 longword comparisons very efficient. If using a path it MUST be the mapped path, not the client-supplied, request path. With conditional mapping, identical request paths may be mapped to completely different virtual paths. Space for a file's data is dynamically allocated and reallocated if necessary as cache entries are reused. It is allocated in user-specifiable chunks. It is expected this mechanism provides some efficiencies when reusing cache entries. A simple hash table is used to try and initially hit an entry. A collision list allows rapid subsequent searching. The hash table is a fixed 4096 entries with the hash value generated by directly using a fixed three bytes of the MD5 hash for a range from 0..4095. Cache entries are also maintained in a global linked list with the most recent and most frequently hit entries towards the head of the list. The linked-list organisation allows a simple implementation of a least-recently-used (LRU) algorithm for selecting an entry when a new request demands an entry and space for cache loading. The linked list is naturally ordered from most recently and most frequently accessed at the head, to the least recently and least frequently accessed at the tail. Hence an infrequently accessed entry is selected from the tail end of the list, it's data invalidated and given to the new request for cache load. Invalidated data cache entries are also immediately placed at the tail of the list for reuse/reloading. When a new entry is initially loaded it is placed at the top of the list. Hits on other entries result in a check being made against the number of hits of head entry in the list. If the entry being hit has a higher hit count it is placed at the head of the list, pushing the previously head entry "down". If not then it is again checked against the entry immediately before it in the list. If higher then the two are swapped. This results in the most recently loaded entries and the more frequently hit being nearest and migrating towards the start of the search. To help prevent the cache thrashing with floods of requests for not currently loaded files, any entry that has a suitably high number of hits over the recent past (suitably high ... how many is that, and recent past ... how long is that?) are not reused until no hits have occured within that period. Hopefully this prevents lots of unnecessary loads of one-offs at the expense of genuinely frequently accessed files. To prevent multiple loads of the same path/file, for instance if a subsequent request demands the same file as a previous request is still currently loading, any subsequent request will merely transfer the file, not concurrently load it into the cache. CACHE CONTENT VALIDATION ------------------------ The cache will automatically revalidate the data after a specified number of seconds. With file content this is done by comparing the original file revision time to the current revision time. If different the file contents have changed and the cache contents declared invalid. If found invalid the file transfer then continues outside of the cache with the new contents being concurrently reloaded into the cache. With non-file content the entry is just purged and the full request processing is used to reload the content. Cache validation is also always performed if the request uses "Pragma: no-cache" (i.e. as with the Netscape Navigator reload function). Hence there is no need for any explicit flushing of the cache under normal operation. If a document does not immediately reflect and changes made to it (i.e. validation time has not been reached) validation (and consequent reload) can be "forced" with a browser reload. There is a discretional "guard" period that can be set which prevent forced reloading of cached data within the specified number of seconds since last loaded or revalidated. This period can be used to prevent a cache entry or entries from constantly being reloaded through pragma directives (Mozilla for instance has a user-option cache setting which causes the request header always to contain a 'no-cache' indication causing reload). The entire cache may be purged of cached data either from the server administration menu or using command line server control. PERMANENT ENTRIES ----------------- Permanent entries are indicated by a path mapping a SET rule. Permanent entries are designed to allow certain classes of (mainly file) entry, those that are relatively static and/or frequently being used by requests, that once loaded remain in the cache, never validated, never able to be reclaimed or otherwise removed from the cache during routine server activity. These entries, along with the volatile ones, can of course be purged from the cache either via the CLI /DO=CACHE=PURGE command or using Server Administration menu. Permanent entries do not use any of the data memory ([CacheTotalKBytesMax]) set aside for the volatile cache. They use memory from their own specific VM pool. Unlike for volatile entries there is no upper limit (apart from system and server process virtual memory) on the memory that can be allocated for permanent entries (be careful!). Entry slots ([CacheEntriesMax]) are shared between permanent and volatile entries (and can be a constraint on both). GZIP COMPRESSED CONTENT ----------------------- If ZLIB is being used by the server (see GZIP.C) the cache will generate a buffer of GZIP compressed content from the cached binary content. This is then is returned to clients that can accept GZIP encoding saving the processing expense of generating the GZIP compression dynamically with each request. When the cache load is complete the content-type, etc., is tested for suitability for GZIP compression. It is performed as part of the cache post-processing rather than just collecting the original request's GZIP compressed network stream (for instance) for three basic reasons. First; the point at which the deflate is finalized is different to when the cache load is finalized. Second; not all initial cache load requests may accept GZIP encoding. This approach ensures GZIP content is available for any subsequent access. Third; it's slightly more expensive in terms or processing (the possible double GZIPing on the initial cache load) but was fairly straight-forward and self-contained when introducing to the code. VERSION HISTORY --------------- 23-JAN-2020 MGD bugfix; CacheLoadEnd() free rqCache.ContentPtr on fail (seemed to be induced only by HTTP/2 requests - hmmmm) 05-FEB-2019 MGD CacheLoadBegin() CacheLoadEnd() accept 203 as 200 01-JAN-2018 MGD refactor some code paths (sigh! *so* tightly coupled with FILE.C and REQUEST.C modules) 14-JAN-2016 MGD CacheLoadResponse() in line with HTTP/2 changes bugfix; CacheLoadResponse() initialise storage 04-MAR-2011 MGD CacheSearch() implement request cache control 01-MAR-2011 MGD CacheLoadResponse() checks response header for "Cache-Control:" directives and adjusts accordingly 27-FEB-2011 MGD CacheLoadEnd() buffer all content-type data (per JPP) (previous behaviour truncated at ';' or white-space) 05-OCT-2009 MGD use OutputFileBufferSize to maximise transfer 19-AUG-2009 MGD bugfix; CacheAcpInfoAst() byte-range limit negative offset 09-JUN-2007 MGD use STR_DSC 28-FEB-2007 MGD bugfix; CacheNext() don't adjust GZIP content for CGI header 04-JUL-2006 MGD use PercentOf() for more accurate percentages 11-AUG-2005 MGD CacheEntryReclaimCount report item 26-MAR-2005 MGD provide caching and supply of GZIP deflated content 28-JUL-2004 MGD support use of entity tag in file responses 15-MAY-2004 MGD bugfix; content pointer needs to be NULLed before first call to CacheNext() 24-APR-2004 MGD extend cache hit accounting to 100-999 and 1000 plus 09-JUL-2003 MGD rework for non-file caching requirements, support byte-range requests on cached *files* 16-JUN-2003 MGD bugfix; FileSetCharset() moved from FILE.C module 24-MAY-2003 MGD permanent cache entries, path specified maximum file size 11-MAR-2002 MGD bugfix; ensure only one request revalidates a cache entry at a time (multiple could cause eventual channel exhaustion) 22-NOV-2001 MGD ensure there are reasonable cache minima 29-SEP-2001 MGD instance support 04-AUG-2001 MGD use MD5 hash to identify cache entries, modify hash table as fixed 4096 entrie, support module WATCHing 27-MAY-2000 MGD BUGFIX; CacheEntryNotValid() linked list :^{ bugfix; CacheLoadBegin() #else before memcpy() 09-APR-2000 MGD simplified cache search 04-MAR-2000 MGD use FaolToNet(), et.al. 28-DEC-1999 MGD support ODS-2 and ODS-5 using ODS module, add a number of other WATCH points 26-SEP-1999 MGD minor changes in line with RequestExecute(), CacheReport() now only optionally reports cached files, scavenge failure should not result in a sanity check exit 20-JAN-1999 MGD report format refinenment 19-SEP-1998 MGD improve granularity of cache operation, add check for existing entry to CacheLoadBegin() 14-MAY-1998 MGD request-specified content-type ("httpd=content&type=") 18-MAR-1998 MGD use file's VBN and first free byte to check size changes (allows variable record files to be cached more efficiently) 24-FEB-1998 MGD if CacheAcpInfo() reports a problem then let the file module handle/report it, add file size check to entry validation (allow for extend) 10-JAN-1998 MGD added a (much overdue) hash collision list, fixed problem with cache purge (it mostly didn't :^) 22-NOV-1997 MGD sigh, bugfix; need to proactively free memory at capacity 05-OCT-1997 MGD initial development for v4.5 */ /*****************************************************************************/ #ifdef WASD_VMS_V7 #undef _VMS__V6__SOURCE #define _VMS__V6__SOURCE #undef __VMS_VER #define __VMS_VER 70000000 #undef __CRTL_VER #define __CRTL_VER 70000000 #endif /* standard C header files */ #include #include #include #include /* VMS related header files */ #include #include #include #include #include #include #include /* application header files */ #include "wasd.h" #include "gzip.h" #include "md5.h" #define WASD_MODULE "CACHE" /******************/ /* global storage */ /******************/ BOOL CacheEnabled, CacheHashTableInitialised; /* this is run-time storage */ int CacheChunkInBytes, CacheCurrentlyLoading, CacheCurrentlyLoadingInUse, CacheEntriesMax, CacheEntryCount, CacheEntryKBytesMax, CacheFrequentHits, CacheFrequentSeconds, CacheGuardSeconds, CacheHashTableMask, CacheMemoryInUse, CachePermEntryCount, CachePermMemoryInUse, CacheTotalKBytesMax, CacheValidateSeconds; /* these counters may be zeroed when accounting is zeroed */ int CacheGzipDeflateCount, CacheHashTableCollsnCount, CacheHashTableCount, CacheHashTableHitCount, CacheHashTableMissCount, CacheHashTableCollsnHitCount, CacheHashTableCollsnMissCount, CacheHitCount, CacheHits0, CacheHits10, CacheHits100, CacheHits1000, CacheHits1000plus, CacheListHitCount, CacheLoadCount, CacheNotHitCount, CacheNoHitsCount, CacheReclaimCount, CacheSearchCount; unsigned long CacheGzipDeflateBytesIn [2], CacheGzipDeflateBytesOut [2]; LIST_HEAD CacheList; /* the hash table index is the first byte of the MD5 hash */ #define CACHE_HASH_TABLE_ENTRIES 4096 /* 12 bits, 16384 bytes, 32 page(lets) */ struct GenericCacheEntry *CacheHashTable [CACHE_HASH_TABLE_ENTRIES]; /********************/ /* external storage */ /********************/ extern BOOL CliCacheEnabled, CliCacheDisabled, GzipResponse; extern int HttpdTickSecond, OdsExtended, OutputFileBufferSize; extern int ToLowerCase[], ToUpperCase[]; extern short HttpdNumTime[]; extern char ErrorSanityCheck[], HttpProtocol[], SoftwareID[]; extern ACCOUNTING_STRUCT *AccountingPtr; extern CONFIG_STRUCT Config; extern MSG_STRUCT Msgs; extern HTTPD_PROCESS HttpdProcess; extern WATCH_STRUCT Watch; /*****************************************************************************/ /* Initialize cache run-time parameters at startup (even though caching might be initially disabled). Also allocate the hash table when appropriate. */ void CacheInit (BOOL Startup) { int HashValue; /*********/ /* begin */ /*********/ if (WATCH_MODULE(WATCH_MOD_CACHE)) WatchThis (WATCHALL, WATCH_MOD_CACHE, "CacheInit()"); if (Startup) { if (!CliCacheDisabled && (CliCacheEnabled || Config.cfCache.Enabled)) CacheEnabled = true; else CacheEnabled = false; CacheChunkInBytes = CACHE_CHUNK_SIZE; CacheEntriesMax = Config.cfCache.EntriesMax; if (CacheEntriesMax < 256) CacheEntriesMax = 256; CacheEntryKBytesMax = Config.cfCache.FileKBytesMax; if (CacheEntryKBytesMax <= 0) CacheEntryKBytesMax = 65; CacheFrequentHits = Config.cfCache.FrequentHits; CacheFrequentSeconds = Config.cfCache.FrequentSeconds; CacheGuardSeconds = Config.cfCache.GuardSeconds; if (!CacheGuardSeconds) CacheGuardSeconds = 15; CacheTotalKBytesMax = Config.cfCache.TotalKBytesMax; if (CacheTotalKBytesMax < 1024) CacheTotalKBytesMax = 1024; CacheValidateSeconds = Config.cfCache.ValidateSeconds; if (!CacheValidateSeconds) CacheValidateSeconds = 300; CacheEntryCount = CacheMemoryInUse = CachePermEntryCount = CachePermMemoryInUse = 0; CacheZeroCounters (); } if (CacheEntriesMax && CacheEnabled) { /* 100% overhead for ambit memory allocations, fragmentation, etc. */ VmCacheInit (CacheTotalKBytesMax * 2); /* the permanent cache space will be created on the first perm entry */ /** VmPermCacheInit (CacheTotalKBytesMax); **/ for (HashValue = 0; HashValue < CACHE_HASH_TABLE_ENTRIES; HashValue++) CacheHashTable[HashValue] = NULL; CacheHashTableInitialised = true; } } /*****************************************************************************/ /* Just zero the cache-associated counters :^) */ void CacheZeroCounters () { /*********/ /* begin */ /*********/ if (WATCH_MODULE(WATCH_MOD_CACHE)) WatchThis (WATCHALL, WATCH_MOD_CACHE, "CacheZeroCounter()"); CacheGzipDeflateCount = CacheGzipDeflateBytesIn[0] = CacheGzipDeflateBytesIn[1] = CacheGzipDeflateBytesOut[0] = CacheGzipDeflateBytesOut[1] = CacheHitCount = CacheHashTableHitCount = CacheHashTableCollsnCount = CacheHashTableCollsnHitCount = CacheHashTableCollsnMissCount = CacheHashTableCount = CacheHashTableMissCount = CacheNoHitsCount = CacheHits0 = CacheHits10 = CacheHits100 = CacheHits1000 = CacheHits1000plus = CacheLoadCount = CacheReclaimCount = CacheSearchCount = 0; } /*****************************************************************************/ /* Look through the cache list for a resource hash (MD5) that matches. If one is found then call CacheBegin() and return success, otherwise return error status to indicate the search was unsuccessful. NOTE: as with CacheLoadBegin(), the path passed to this function MUST be the mapped path, not the request path. With conditional mapping identical request paths may be mapped to completely different virtual paths. */ int CacheSearch (REQUEST_STRUCT *rqptr) { int status, EntryCount, HashValue; char *cptr, *sptr; FILE_CENTRY *captr; LIST_ENTRY *leptr; MD5_HASH *md5ptr; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "CacheSearch() file:!&B", rqptr->FileTaskPtr); if (!CacheEnabled) return (SS$_NOSUCHFILE); if (!CacheHashTableInitialised) CacheInit (false); /* if the request specified that cache not be used */ if (rqptr->rqHeader.CacheControlNoCache || rqptr->rqHeader.CacheControlNoStore || rqptr->rqHeader.PragmaNoCache) { if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE request no-cache"); return (SS$_NOSUCHFILE); } /* if not interested in anything with a query string */ if (!rqptr->rqPathSet.CacheQuery && rqptr->rqHeader.QueryStringLength) { if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE request query-string"); return (SS$_NOSUCHFILE); } /* any request containing a cookie that is not from a static file */ if (rqptr->rqHeader.CookiePtr && !rqptr->FileTaskPtr && !rqptr->rqPathSet.CacheCookie) { if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE request cookie"); return (SS$_NOSUCHFILE); } /* if not a GET request */ if (rqptr->rqHeader.Method != HTTP_METHOD_GET) return (SS$_NOSUCHFILE); /* if a SET mapping rule has specified the path should not be cached */ if (rqptr->rqPathSet.NoCache) return (SS$_NOSUCHFILE); /* server has specifically set this request not to be cached */ if (rqptr->rqCache.DoNotCache) return (SS$_NOSUCHFILE); /* must be using this for a cache load (e.g. during directory listing) */ if (rqptr->rqCache.EntryPtr) return (SS$_NOSUCHFILE); if (rqptr->FileTaskPtr) md5ptr = &rqptr->FileTaskPtr->Md5Hash; else md5ptr = &rqptr->Md5HashPath; if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE search !&?file\rpath\r !16&H", rqptr->FileTaskPtr, md5ptr); /* check the hash table */ CacheHashTableCount++; /* 12 bit hash value, 0..4095 from a fixed 3 bytes of the MD5 hash */ HashValue = md5ptr->HashLong[0] & 0xfff; if (!(captr = CacheHashTable[HashValue])) { /*************/ /* not found */ /*************/ if (rqptr->rqHeader.CacheControlOnlyIfCached) { rqptr->rqResponse.HttpStatus = 504; ErrorVmsStatus (rqptr, SS$_TIMEOUT, FI_LI); return (SS$_TIMEOUT); } /* nope, no pointer against that hash value */ CacheHashTableMissCount++; return (SS$_NOSUCHFILE); } EntryCount = 0; for ( /*set above*/ ; captr; captr = captr->HashCollisionNextPtr) { if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "file:!&B valid:!&B revalidating:!&B !AZ !16&H !&?match\rno-match\r", captr->FromFile, captr->EntryValid, captr->EntryRevalidating, captr->FileOds.ExpFileName, &captr->Md5Hash, MATCH16 (&captr->Md5Hash, md5ptr)); /* note the first collision entry */ if (++EntryCount == 2) CacheHashTableCollsnCount++; if (!MATCH16 (&captr->Md5Hash, md5ptr)) continue; /********/ /* hit! */ /********/ /* first entry is always the hash table, after that the collision list */ if (EntryCount == 1) CacheHashTableHitCount++; else CacheHashTableCollsnHitCount++; if (!captr->EntryValid || captr->DataLoading || captr->EntryRevalidating) { /**************/ /* not usable */ /**************/ rqptr->rqCache.NotUsable = true; return (SS$_NOSUCHFILE); } /**********/ /* usable */ /**********/ if (rqptr->rqHeader.CacheControlMaxAge && HttpdTickSecond - captr->LoadSeconds > rqptr->rqHeader.CacheControlMaxAge) { /***********/ /* too old */ /***********/ if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE request max-age !UL", rqptr->rqHeader.CacheControlMaxAge); return (SS$_NOSUCHFILE); } if (!ResponseEntityMatch (rqptr, captr->EntityTag)) { /********************/ /* failed condition */ /********************/ /* ResponseEntityMatch() should generate its own HTTP response */ return (SS$_ABORT); } /************/ /* letsdoit */ /************/ rqptr->rqCache.EntryPtr = captr; status = CacheBegin (rqptr); if (VMSnok (status)) rqptr->rqCache.EntryPtr = NULL; return (status); } /***********/ /* not hit */ /***********/ CacheHashTableCollsnMissCount++; return (SS$_NOSUCHFILE); } /*****************************************************************************/ /* Use this cached entry as the contents for the client. If the entry needs to be validated then generate an asynchronous ACP QIO to get the required file details, other wise call the AST processing function directly. NOTE: as with CacheLoadBegin(), the path passed to this function MUST be the mapped path, not the request path. With conditional mapping identical request paths may be mapped to completely different virtual paths. */ int CacheBegin (REQUEST_STRUCT *rqptr) { BOOL RevalidateEntry; FILE_CENTRY *captr; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "CacheBegin() !AZ", rqptr->rqCache.EntryPtr->FileOds.ExpFileName); captr = rqptr->rqCache.EntryPtr; if (captr->EntryPermanent) RevalidateEntry = false; else if (rqptr->NotFromCache && captr->GuardTickSecond < HttpdTickSecond) RevalidateEntry = true; else if (captr->ExpiresAfterPeriod == CACHE_EXPIRES_DAY) if (captr->ExpiresAfterTime != HttpdNumTime[2]) RevalidateEntry = true; else RevalidateEntry = false; else if (captr->ExpiresAfterPeriod == CACHE_EXPIRES_HOUR) if (captr->ExpiresAfterTime != HttpdNumTime[3]) RevalidateEntry = true; else RevalidateEntry = false; else if (captr->ExpiresAfterPeriod == CACHE_EXPIRES_MINUTE) if (captr->ExpiresAfterTime != HttpdNumTime[4]) RevalidateEntry = true; else RevalidateEntry = false; else if (captr->ExpiresTickSecond < HttpdTickSecond) RevalidateEntry = true; else RevalidateEntry = false; if (!RevalidateEntry) { /* status of non-zero used to determine whether the ACPQIO was used */ captr->FileOds.FileQio.IOsb.Status = 0; CacheAcpInfoAst (rqptr); return (SS$_NORMAL); } if (captr->FromFile) { if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE revalidate !AZ", captr->FileOds.ExpFileName); captr->EntryRevalidating = true; captr->InUseCount++; captr->ValidatedCount++; OdsFileAcpInfo (&captr->FileOds, &CacheAcpInfoAst, rqptr); return (SS$_NORMAL); } /* non-file entries that are invalid are purged and re-cached */ if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE stale !AZ", captr->FileOds.ExpFileName); CacheRemoveEntry (captr, false); /* move it to the end of the cache list ready for reuse */ ListMoveTail (&CacheList, captr); return (SS$_NOSUCHFILE); } /*****************************************************************************/ /* Called either explicitly or as an AST from an ACP QIO in CacheBegin(). Check if modified, if not then just reply with a 302 header. If contents should be transfered to the client then begin. If there is a problem at all then declare the AST to continue processing the request and let it worry about it, otherwise begin providing the file data from the cache entry. */ void CacheAcpInfoAst (REQUEST_STRUCT *rqptr) { BOOL RangeValid; int idx, status, ContentLength; unsigned short Length; char *cptr, *sptr, *zptr; RANGE_BYTE *rbptr; REQUEST_AST AstFunction; FILE_CENTRY *captr, *tcaptr; FILE_CONTENT *fcptr; FILE_QIO *fqptr; LIST_ENTRY *leptr; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "CacheAcpInfoAst() !&F !&S !%D !%D", &CacheAcpInfoAst, rqptr->rqCache.EntryPtr->FileOds.FileQio.IOsb.Status, &rqptr->rqCache.EntryPtr->RdtTime64, &rqptr->rqCache.EntryPtr->FileOds.FileQio.RdtTime64); captr = rqptr->rqCache.EntryPtr; fqptr = &captr->FileOds.FileQio; /* proactively remove the association between the entry and the request */ rqptr->rqCache.EntryPtr = NULL; /* status of non-zero used to determine whether the ACPQIO was used */ if (fqptr->IOsb.Status) { /*********************/ /* revalidating file */ /*********************/ /* finished getting the revalidate data */ captr->EntryRevalidating = false; captr->InUseCount--; /* first deassign the channel allocated by OdsFileAcpInfo() */ sys$dassgn (fqptr->AcpChannel); if (VMSnok (fqptr->IOsb.Status)) { /***********************/ /* file access problem */ /***********************/ /* entry no longer valid */ CacheRemoveEntry (captr, false); /* move it to the end of the cache list ready for reuse */ ListMoveTail (&CacheList, captr); CacheEnd (rqptr); return; } if (rqptr->rqResponse.HttpVersion == HTTP_VERSION_1_1) if (captr->EntityTag[0]) strcpy (rqptr->rqResponse.EntityTag, captr->EntityTag); fqptr->EndOfFileVbn = ((fqptr->RecAttr.fat$l_efblk & 0xffff) << 16) | ((fqptr->RecAttr.fat$l_efblk & 0xffff0000) >> 16); fqptr->FirstFreeByte = fqptr->RecAttr.fat$w_ffbyte; if (captr->RdtTime64[0] != fqptr->RdtTime64[0] || captr->RdtTime64[1] != fqptr->RdtTime64[1] || captr->EndOfFileVbn != fqptr->EndOfFileVbn || captr->FirstFreeByte != fqptr->FirstFreeByte) { /************************/ /* data no longer valid */ /************************/ if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE stale !AZ !%D (!%D / !%D) (!UL,!UL / !UL,!UL)", captr->FileOds.ExpFileName, &captr->CdtTime64, &captr->RdtTime64, &fqptr->RdtTime64, captr->EndOfFileVbn, captr->FirstFreeByte, fqptr->EndOfFileVbn, fqptr->FirstFreeByte); /* entry no longer valid */ CacheRemoveEntry (captr, false); /* move it to the end of the cache list ready for reuse */ ListMoveTail (&CacheList, captr); CacheEnd (rqptr); return; } /********************/ /* data still valid */ /********************/ /* note the time the cached data was last validated */ PUT_QUAD_QUAD (rqptr->rqTime.BeginTime64, captr->ValidateTime64); if (captr->ExpiresAfterPeriod == CACHE_EXPIRES_DAY) captr->ExpiresAfterTime = HttpdNumTime[2]; else if (captr->ExpiresAfterPeriod == CACHE_EXPIRES_HOUR) captr->ExpiresAfterTime = HttpdNumTime[3]; else if (captr->ExpiresAfterPeriod == CACHE_EXPIRES_MINUTE) captr->ExpiresAfterTime = HttpdNumTime[4]; else if (captr->ExpiresAfterPeriod == CACHE_EXPIRES_NONE) captr->ExpiresTickSecond = 0xffffffff; else if (captr->ExpiresAfterPeriod) captr->ExpiresTickSecond = HttpdTickSecond + captr->ExpiresAfterPeriod; else captr->ExpiresTickSecond = HttpdTickSecond + CacheValidateSeconds; if (captr->GuardSeconds) captr->GuardTickSecond = HttpdTickSecond + captr->GuardSeconds; else captr->GuardTickSecond = HttpdTickSecond + CacheGuardSeconds; } if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE hit !&?permanent\rvolatile\r !AZ", captr->EntryPermanent, captr->FileOds.ExpFileName); /**************************************/ /* request to be fulfilled from cache */ /**************************************/ /* cancel any no-such-file callback, we've obviously got it! */ if (rqptr->FileTaskPtr) rqptr->FileTaskPtr->NoSuchFileFunction = NULL; PUT_QUAD_QUAD (rqptr->rqTime.BeginTime64, captr->HitTime64); captr->FrequentTickSecond = HttpdTickSecond + CacheFrequentSeconds; CacheHitCount++; if (!captr->HitCount) CacheHits0--; captr->HitCount++; if (captr->HitCount < 10) CacheHits10++; else if (captr->HitCount < 100) CacheHits100++; else if (captr->HitCount < 1000) CacheHits1000++; else CacheHits1000plus++; rqptr->AccountingDone = InstanceGblSecIncrLong (&AccountingPtr->CacheHitCount); /* if this entry has more hits move it to the head of the cache list */ leptr = CacheList.HeadPtr; tcaptr = (FILE_CENTRY*)leptr; if (captr->HitCount > tcaptr->HitCount) ListMoveHead (&CacheList, captr); else { /* if this entry has more hits than the one "above" it swap them */ leptr = captr; if (leptr->PrevPtr) { leptr = leptr->PrevPtr; tcaptr = (FILE_CENTRY*)leptr; if (captr->HitCount > tcaptr->HitCount) { ListRemove (&CacheList, captr); ListAddBefore (&CacheList, leptr, captr, LIST_ENTRY_TYPE_CACHE); } } } if (captr->ContentHandlerFunction) { /***********************************/ /* this file has a content handler */ /***********************************/ if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "!&X", captr->ContentHandlerFunction); /* reassociated the cache entry with this request */ rqptr->rqCache.EntryPtr = captr; /* allow one extra character for a(n implicit) terminating null */ rqptr->FileContentPtr = fcptr = (FILE_CONTENT*) VmGetHeap (rqptr, sizeof(FILE_CONTENT) + captr->ContentLength+1); fcptr->ContentSize = fcptr->ContentSizeMax = captr->ContentLength+1; /* buffer space immediately follows the structured storage */ fcptr->ContentPtr = (char*)fcptr + sizeof(FILE_CONTENT); memcpy (fcptr->ContentPtr, captr->ContentPtr, fcptr->ContentLength = captr->ContentLength); /* populate the file contents structure with some file data */ zptr = (sptr = fcptr->FileName) + sizeof(fcptr->FileName); for (cptr = captr->FileOds.ExpFileName; *cptr && sptr < zptr; *sptr++ = *cptr++); if (sptr >= zptr) { ErrorGeneralOverflow (rqptr, FI_LI); CacheEnd (rqptr); return; } *sptr = '\0'; fcptr->FileNameLength = sptr - fcptr->FileName; PUT_QUAD_QUAD (captr->CdtTime64, fcptr->CdtTime64); PUT_QUAD_QUAD (captr->RdtTime64, fcptr->RdtTime64); fcptr->UicGroup = captr->UicGroup; fcptr->UicMember = captr->UicMember; fcptr->Protection = captr->Protection; fcptr->ContentHandlerFunction = captr->ContentHandlerFunction; CacheEnd (rqptr); return; } if (!rqptr->rqResponse.HeaderGenerated) { /**************************/ /* full response required */ /**************************/ if (captr->FromFile) { /********/ /* file */ /********/ if (rqptr->rqHeader.RangeBytePtr && rqptr->rqHeader.RangeBytePtr->Total) { /**************/ /* byte-range */ /**************/ RangeValid = true; rbptr = rqptr->rqHeader.RangeBytePtr; for (idx = 0; idx < rbptr->Total; idx++) { if (!rbptr->Last[idx]) { /* last byte not specified, set at EOF */ rbptr->Last[idx] = captr->ContentLength - 1; } else if (rbptr->Last[idx] < 0) { /* first byte a negative offset from end, last byte at EOF */ rbptr->First[idx] = captr->ContentLength + rbptr->Last[idx]; if (rbptr->First[idx] < 0) rbptr->First[idx] = 0; rbptr->Last[idx] = captr->ContentLength - 1; } else if (rbptr->Last[idx] >= captr->ContentLength) { /* if the last byte is ambit make it at the EOF */ rbptr->Last[idx] = captr->ContentLength - 1; } /* if the range still does not make sense then back out now */ if (rbptr->Last[idx] < rbptr->First[idx]) { RangeValid = false; rbptr->Length = 0; } else rbptr->Length = rbptr->Last[idx] - rbptr->First[idx] + 1; if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "RANGE !UL !UL-!UL !UL byte!%s!&? INVALID\r\r", idx+1, rbptr->First[idx], rbptr->Last[idx], rbptr->Length, !rbptr->Length); } if (RangeValid) rbptr->Count = idx; } if (rqptr->rqPathSet.CharsetPtr) { cptr = captr->FileOds.ResFileName; if (!*cptr) cptr = captr->FileOds.ExpFileName; rqptr->rqPathSet.CharsetPtr = FileSetCharset (rqptr, cptr); } if (rqptr->rqResponse.HttpVersion == HTTP_VERSION_1_1) if (captr->EntityTag[0]) strcpy (rqptr->rqResponse.EntityTag, captr->EntityTag); status = FileResponseHeader (rqptr, captr->ContentType, captr->ContentLength, &captr->RdtTime64); if (VMSnok (status)) { if (status == LIB$_NEGTIM) { InstanceGblSecIncrLong (&AccountingPtr-> CacheHitNotModifiedCount); captr->HitNotModifiedCount++; } CacheEnd (rqptr); return; } } else { /************/ /* non-file */ /************/ if (captr->ContentType[0]) { /* if any retained CGI header fields add these to the header */ if (captr->CgiHeaderLength) { cptr = captr->ContentPtr; ContentLength = captr->ContentLength - captr->CgiHeaderLength - 1; } else { cptr = NULL; ContentLength = captr->ContentLength; } ResponseHeader (rqptr, 200, captr->ContentType, ContentLength, &captr->RdtTime64, cptr); } else { /* when no associated content-type it's an NPH script */ rqptr->rqResponse.HttpStatus = 200; } } /* quit here if the HTTP method was HEAD */ if (rqptr->rqHeader.Method == HTTP_METHOD_HEAD) { CacheEnd (rqptr); return; } } /************/ /* transfer */ /************/ /* 'CacheInUse' keeps track of whether the entry is in use or not */ captr->InUseCount++; /* initialize this for start-of-transfer detection in CacheNext() */ rqptr->rqCache.ContentPtr = NULL; /* reassociated the cache entry with this request */ rqptr->rqCache.EntryPtr = captr; /* network writes are checked for success, fudge the first one! */ rqptr->NetIoPtr->WriteStatus = SS$_NORMAL; /* begin the transfer */ if (STR_DSC_LEN(&rqptr->NetWriteBufferDsc)) { /* after ensuring the current contents are output */ NetWriteFullFlush (rqptr, &CacheNext); } else CacheNext (rqptr); } /*****************************************************************************/ /* Write the next (or first) block of data from the cache buffer to the client. */ void CacheNext (REQUEST_STRUCT *rqptr) { int status, DataLength; unsigned short Length; unsigned char *DataPtr; FILE_CENTRY *captr; RANGE_BYTE *rbptr; REQUEST_AST AstFunction; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "CacheNext() !&F !&S", &CacheNext, rqptr->NetIoPtr->WriteStatus); if (rqptr->RequestState >= REQUEST_STATE_ABORT) { CacheEnd (rqptr); return; } if (VMSnok (rqptr->NetIoPtr->WriteStatus)) { CacheEnd (rqptr); return; } captr = rqptr->rqCache.EntryPtr; if (!rqptr->rqCache.ContentPtr) { /* first read, initialize content pointer and length */ if (rqptr->rqHeader.RangeBytePtr && rqptr->rqHeader.RangeBytePtr->Count) { /* returning a byte range within the file (partial content) */ rbptr = rqptr->rqHeader.RangeBytePtr; rbptr->Length = rbptr->Last[rbptr->Index] - rbptr->First[rbptr->Index] + 1; if (rbptr->Count > 1) { /* returning 'multipart/byteranges' range content */ char Buffer [256]; FaoToBuffer (Buffer, sizeof(Buffer), &Length, "!AZ--!AZ\r\n\ Content-Type: !AZ\r\n\ Range: bytes !UL-!UL/!UL\r\n\ \r\n", rbptr->Index ? "\r\n" : "", rqptr->rqResponse.MultipartBoundaryPtr, captr->ContentType, rbptr->First[rbptr->Index], rbptr->Last[rbptr->Index], captr->ContentLength); /* synchronous network write (just for the convenience of it!) */ NetWrite (rqptr, NULL, Buffer, Length); } rqptr->rqCache.ContentPtr = captr->ContentPtr + rbptr->First[rbptr->Index]; rqptr->rqCache.ContentLength = rbptr->Length; } else if (captr->GzipContentPtr && rqptr->rqResponse.ContentEncodeAsGzip && /* if the header has been sent this is something like a readme! */ !rqptr->rqResponse.HeaderSent) { /* GZIP content is available and we're going to provide one */ rqptr->rqCache.ContentPtr = captr->GzipContentPtr; rqptr->rqCache.ContentLength = captr->GzipContentLength; rqptr->rqResponse.ContentEncodeAsGzip = false; rqptr->rqResponse.ContentIsEncodedGzip = true; if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE (gzip deflated) !UL->!UL bytes, !UL%", captr->ContentLength, captr->GzipContentLength, PercentOf(captr->GzipContentLength, captr->ContentLength)); } else { /* use the unencoded content */ rqptr->rqCache.ContentPtr = captr->ContentPtr; rqptr->rqCache.ContentLength = captr->ContentLength; if (captr->CgiHeaderLength) { /* adjust body of response for any retained CGI header fields */ rqptr->rqCache.ContentPtr += captr->CgiHeaderLength + 1; rqptr->rqCache.ContentLength -= captr->CgiHeaderLength + 1; } } } if (rqptr->rqCache.ContentLength > OutputFileBufferSize) { DataPtr = rqptr->rqCache.ContentPtr; DataLength = OutputFileBufferSize; rqptr->rqCache.ContentPtr += DataLength; rqptr->rqCache.ContentLength -= DataLength; if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) { WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "CACHE !UL/!UL", DataLength, rqptr->rqCache.ContentLength); WatchDataDump (DataPtr, DataLength); } NetWrite (rqptr, &CacheNext, DataPtr, DataLength); return; } else { DataPtr = rqptr->rqCache.ContentPtr; DataLength = rqptr->rqCache.ContentLength; rqptr->rqCache.ContentLength = 0; if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) { WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "CACHE !UL/!UL", DataLength, rqptr->rqCache.ContentLength); WatchDataDump (DataPtr, DataLength); } NetWrite (rqptr, &CacheEnd, DataPtr, DataLength); return; } } /*****************************************************************************/ /* End of transfer to client using cached contents. */ void CacheEnd (REQUEST_STRUCT *rqptr) { char *cptr, *sptr, *zptr; FILE_CENTRY *captr; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "CacheEnd() !&F", &CacheEnd); if (!(captr = rqptr->rqCache.EntryPtr)) { /* entry is no longer valid */ if (rqptr->FileTaskPtr) FileParse (rqptr); else RequestExecutePostCache1 (rqptr); return; } if (rqptr->rqHeader.RangeBytePtr && rqptr->rqHeader.RangeBytePtr->Count) { /* transfering byte-range(s) */ rqptr->rqHeader.RangeBytePtr->Index++; if (rqptr->rqHeader.RangeBytePtr->Index < rqptr->rqHeader.RangeBytePtr->Count) { /* multiple byte ranges, restart with next range */ rqptr->rqCache.ContentPtr = NULL; rqptr->rqCache.ContentLength = 0; SysDclAst (&CacheNext, rqptr); return; } if (rqptr->rqHeader.RangeBytePtr->Count > 1) { /* end of multiple byte ranges, provide final boundary */ char Buffer [64]; zptr = (sptr = Buffer) + sizeof(Buffer)-1; for (cptr = "\r\n--"; *cptr && sptr < zptr; *sptr++ = *cptr++); for (cptr = rqptr->rqResponse.MultipartBoundaryPtr; *cptr && sptr < zptr; *sptr++ = *cptr++); for (cptr = "--\r\n"; *cptr && sptr < zptr; *sptr++ = *cptr++); *sptr = '\0'; /* synchronous network write (for the convenience of it!) */ NetWrite (rqptr, NULL, Buffer, sptr-Buffer); } } /* this cache entry is no longer associated with this request */ rqptr->rqCache.EntryPtr = NULL; /* this cache entry is no longer in use (if now zero) */ if (captr->InUseCount) captr->InUseCount--; if (captr->Purge && !captr->InUseCount) { if (captr->PurgeCompletely) CacheRemoveEntry (captr, true); else { CacheRemoveEntry (captr, false); /* move it to the end of the cache list ready for reuse */ ListMoveTail (&CacheList, captr); } } rqptr->BytesTxGzipPercent = PercentOf (captr->GzipContentLength, captr->ContentLength); if (rqptr->FileTaskPtr) FileEnd (rqptr); else RequestEnd (rqptr); } /*****************************************************************************/ /* Check that we're interested in caching this particular data and that the requested size is allowed to be cached. Find/create a cache structure ready to contain the data buffered. This also blocks other concurrent loads of the same resource. It is entirely possible, given that the maximum number of cache entries has been reached and all are currently in use (either for data transfer or being loaded) - though this if fairly unlikely, that there will be no cache entry available for this request to use and the load will fail. If available allocate an appropriate chunk of either volatile or permanent cache memory and use that to load the to-be-cached data. */ BOOL CacheLoadBegin ( REQUEST_STRUCT *rqptr, int SizeInBytes, char *ContentTypePtr ) { int secs, MaxKBytes; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "CacheLoadBegin() !UL bytes !&Z !&Z http:!UL in-use:!&B", SizeInBytes, ContentTypePtr, rqptr->rqHeader.QueryStringPtr, rqptr->rqResponse.HttpStatus, rqptr->rqCache.ContentPtr); rqptr->rqCache.LoadCheck = true; /* could be disabled between search and load beginning */ if (!CacheEnabled) return (false); /* only interested in HTTP success status */ if (rqptr->rqResponse.HttpStatus && rqptr->rqResponse.HttpStatus != 200 && rqptr->rqResponse.HttpStatus != 203) return (false); /* if this cache structure is already in use (e.g. directory listing) */ if (rqptr->rqCache.EntryPtr) return (false); /* if not interested in query strings */ if (!rqptr->rqPathSet.CacheQuery && rqptr->rqHeader.QueryStringLength) return (false); /* if a SET mapping rule has specified the path should not be cached */ if (rqptr->rqPathSet.NoCache) return (false); /* if not a GET request */ if (rqptr->rqHeader.Method != HTTP_METHOD_GET) return (false); /* server has specifically set this request not to be cached */ if (rqptr->rqCache.DoNotCache) return (false); MaxKBytes = 0; if (rqptr->rqCgi.ScriptControlCacheMaxKBytes) { /* start off with any script specified value */ MaxKBytes = rqptr->rqCgi.ScriptControlCacheMaxKBytes; if (rqptr->rqPathSet.CacheMaxKBytes) { /* any path setting should limit anything script supplied */ if (MaxKBytes > rqptr->rqPathSet.CacheMaxKBytes) MaxKBytes = rqptr->rqPathSet.CacheMaxKBytes; } else { /* configuration setting should limit anything script supplied */ if (MaxKBytes > CacheEntryKBytesMax) MaxKBytes = CacheEntryKBytesMax; } } else if (rqptr->rqPathSet.CacheMaxKBytes) { /* path specified maximum overrides configuration maximum */ MaxKBytes = rqptr->rqPathSet.CacheMaxKBytes; } /* fall back to using the configuration setting */ if (!MaxKBytes) MaxKBytes = CacheEntryKBytesMax; /* zero indicates exact size is unknown so start with an ambit maximum */ if (SizeInBytes <= 0) SizeInBytes = MaxKBytes << 10; /* if it's larger than the maximum allowed */ if ((SizeInBytes >> 10) > MaxKBytes) { if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE load fail, too large (!ULkB>!ULkB)", SizeInBytes >> 10, MaxKBytes); return (false); } if (!CacheLoadResponse (rqptr)) return (false); /* if we can't get one then just forget it! */ if (!CacheAllocateEntry (rqptr)) return (false); /* calculate the cache chunk */ if (SizeInBytes % CacheChunkInBytes) SizeInBytes += CacheChunkInBytes; SizeInBytes = SizeInBytes / CacheChunkInBytes; SizeInBytes *= CacheChunkInBytes; if (rqptr->rqPathSet.CachePermanent) rqptr->rqCache.ContentPtr = VmGetPermCache (SizeInBytes); else rqptr->rqCache.ContentPtr = VmGetCache (SizeInBytes); rqptr->rqCache.CurrentPtr = rqptr->rqCache.ContentPtr; rqptr->rqCache.ContentRemaining = SizeInBytes; rqptr->rqCache.ContentBufferSize = SizeInBytes; rqptr->rqCache.ContentLength = rqptr->rqCache.RecordBlockLength = 0; rqptr->rqCache.ContentTypePtr = ContentTypePtr; rqptr->rqCache.Loading = true; rqptr->rqCache.LoadStatus = 0; CacheCurrentlyLoadingInUse += SizeInBytes; CacheCurrentlyLoading++; return (true); } /*****************************************************************************/ /* This function checks any cache-control in that and if it should not be cached returns false. It also checks for an entity-tag header and stores it locally to the request, and for the presence of a cookie (responses with cookies or vary headers are never cached). The rqCache.Response.. storage conveys information back to the cache load to determine some characterstics. Return true to allow caching (the default), and false to prohibit caching. This function is very much a collection of pragmatics. */ BOOL CacheLoadResponse (REQUEST_STRUCT *rqptr) { BOOL ok2cache, CookieHit = false, VaryCookie = false; int len, CacheFor = CACHE_EXPIRES_NONE, VaryCount = 0; char *cptr, *sptr, *zptr; DICT_ENTRY_STRUCT *denptr; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "CacheLoadResponse()"); if (denptr = DictLookup (rqptr->rqDictPtr, DICT_TYPE_RESPONSE, "cache-control", 13)) { cptr = DICT_GET_VALUE(denptr); if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE cache-control: !AZ", cptr); while (*cptr) { if (strsame (cptr, "max-age=", len=8)) CacheFor = rqptr->rqCache.ResponseCacheControl = atoi(cptr+8); else if (strsame (cptr, "must-revalidate", len=15)) CacheFor = 0; else if (strsame (cptr, "no-cache", len=8)) CacheFor = 0; else if (strsame (cptr, "no-store", len=8)) CacheFor = 0; else if (strsame (cptr, "private", len=7)) CacheFor = 0; else if (strsame (cptr, "proxy-revalidate", len=16)) CacheFor = 0; else if (strsame (cptr, "public", len=6)) CacheFor = CACHE_EXPIRES_NONE; else if (strsame (cptr, "s-maxage=", len=9)) CacheFor = rqptr->rqCache.ResponseCacheControl = atoi(cptr+9); else len = 0; while (*cptr && !ISLWS(*cptr) && *cptr != ',') cptr++; while (*cptr && (ISLWS(*cptr) || *cptr == ',')) cptr++; } } if (denptr = DictLookup (rqptr->rqDictPtr, DICT_TYPE_RESPONSE, "etag", 4)) { cptr = DICT_GET_VALUE(denptr); if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE etag: !AZ", cptr); zptr = (sptr = rqptr->rqCache.ResponseEntityTag) + sizeof(rqptr->rqCache.ResponseEntityTag)-1; if (*cptr == '\"') cptr++; while (*cptr && !ISLWS(*cptr) && *cptr != '\"' && sptr < zptr) *sptr++ = *cptr++; if (sptr >= zptr) { ErrorNoticed (rqptr, SS$_RESULTOVF, NULL, FI_LI); sptr = rqptr->rqCache.ResponseEntityTag; } *sptr = '\0'; } if (denptr = DictLookup (rqptr->rqDictPtr, DICT_TYPE_RESPONSE, "pragma", 6)) { cptr = DICT_GET_VALUE(denptr); if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE pragma: !AZ", cptr); while (*cptr && ISLWS(*cptr)) cptr++; if (strsame (cptr, "no-cache", 8)) CacheFor = 0; } if (denptr = DictLookup (rqptr->rqDictPtr, DICT_TYPE_RESPONSE, "set-cookie", 10)) { if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE set-cookie:"); CookieHit = true; } if (denptr = DictLookup (rqptr->rqDictPtr, DICT_TYPE_RESPONSE, "vary", 4)) { cptr = DICT_GET_VALUE(denptr); if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE vary: !AZ", cptr); while (*cptr && ISLWS(*cptr)); cptr++; /* server generated response based on specific request characteristics */ VaryCount = 1; VaryCookie = false; while (*cptr) { if (*cptr == ',') VaryCount++; else if (TOUP(*cptr) == 'C' && strsame (cptr, "Cookie", 6)) VaryCookie = true; cptr++; } /* if the only variation is cookie and not one in this response */ if (VaryCount == 1 && VaryCookie && !CookieHit) VaryCount = 0; } /* return true if (all other things being equal) it's OK to cache */ ok2cache = !CookieHit && !VaryCount && CacheFor != 0; if (WATCHING (rqptr, WATCH_RESPONSE)) { if (!ok2cache) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE no"); else if (CacheFor != CACHE_EXPIRES_NONE) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE for !UL", CacheFor); } return (ok2cache); } /*****************************************************************************/ /* Check the success of the load. This is indicated using an end-of-file status. If the load failed or finding a cache entry failed just discard the loaded data and return the memory to the cache pool. If the content-type, etc., lends itself to being GZIP encoded then go ahead and process the cached binary content into a buffer of GZIP compressed content. */ void CacheLoadEnd (REQUEST_STRUCT *rqptr) { int status, ReclaimEntryBytes, ReclaimEntryCount, SizeInBytes; char *cptr, *sptr, *zptr; FILE_CENTRY *captr, *lcaptr; FILE_TASK *ftkptr; LIST_ENTRY *leptr; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) if (rqptr->FileTaskPtr) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "CacheLoadEnd() file:!&B !UL !&S", rqptr->FileTaskPtr, rqptr->rqResponse.HttpStatus, rqptr->rqCache.LoadStatus); else WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "CacheLoadEnd() file:FALSE !UL !&S", rqptr->rqResponse.HttpStatus, rqptr->rqCache.LoadStatus); /* better check, caching could be enabled between ASTs since search */ if (!CacheHashTableInitialised) CacheInit (false); if (!rqptr->rqCache.Loading) ErrorExitVmsStatus (SS$_BUGCHECK, ErrorSanityCheck, FI_LI); captr = rqptr->rqCache.EntryPtr; /* entry is no longer associated with this request */ rqptr->rqCache.EntryPtr = NULL; captr->InUseCount--; if (captr->FromFile = rqptr->rqCache.LoadFromFile) cptr = "FILE"; if (captr->FromNet = rqptr->rqCache.LoadFromNet) cptr = "NET"; if (captr->FromScript = rqptr->rqCache.LoadFromCgi) cptr = "CGI"; rqptr->rqCache.Loading = rqptr->rqCache.LoadFromCgi = rqptr->rqCache.LoadFromFile = rqptr->rqCache.LoadFromNet = false; CacheCurrentlyLoading--; CacheCurrentlyLoadingInUse -= rqptr->rqCache.ContentBufferSize; /* not interested in anything but successful responses */ if ((rqptr->rqResponse.HttpStatus && rqptr->rqResponse.HttpStatus != 200 && rqptr->rqResponse.HttpStatus != 203) || /* an early HTTP success status but an error during processing */ rqptr->rqResponse.ErrorReportPtr) rqptr->rqCache.LoadStatus = SS$_CANCEL; /* end-of-file is used to indicate successful cache data load */ if (rqptr->rqCache.LoadStatus != RMS$_EOF && rqptr->rqCache.LoadStatus != SS$_ENDOFFILE) { /*********************/ /* load unsuccessful */ /*********************/ if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE load from !AZ fail, data %!&M", cptr, rqptr->rqCache.LoadStatus); CacheRemoveEntry (captr, false); /* move it to the end of the cache list ready for reuse */ ListMoveTail (&CacheList, captr); if (rqptr->rqCache.ContentPtr) { if (rqptr->rqPathSet.CachePermanent) VmFreePermCache (rqptr->rqCache.ContentPtr, FI_LI); else VmFreeCache (rqptr->rqCache.ContentPtr, FI_LI); rqptr->rqCache.ContentPtr = NULL; } return; } if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE load from !AZ complete, !UL bytes", cptr, rqptr->rqCache.ContentLength); /**********************/ /* check memory usage */ /**********************/ /* calculate the required cache chunk */ SizeInBytes = rqptr->rqCache.ContentLength; if (SizeInBytes % CacheChunkInBytes) SizeInBytes += CacheChunkInBytes; SizeInBytes = SizeInBytes / CacheChunkInBytes; SizeInBytes *= CacheChunkInBytes; if (!SizeInBytes) SizeInBytes = CacheChunkInBytes; if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "!UL < !UL !&B", SizeInBytes, rqptr->rqCache.ContentBufferSize, SizeInBytes < rqptr->rqCache.ContentBufferSize); if (SizeInBytes < rqptr->rqCache.ContentBufferSize) { /* actually required cache chunk is smaller than original ambit chunk */ cptr = rqptr->rqCache.ContentPtr; if (rqptr->rqPathSet.CachePermanent) { sptr = VmGetPermCache (SizeInBytes); memcpy (sptr, cptr, rqptr->rqCache.ContentLength); VmFreePermCache (cptr, FI_LI); } else { sptr = VmGetCache (SizeInBytes); memcpy (sptr, cptr, rqptr->rqCache.ContentLength); VmFreeCache (cptr, FI_LI); } rqptr->rqCache.ContentPtr = sptr; rqptr->rqCache.ContentBufferSize = SizeInBytes; } /************************/ /* populate cache entry */ /************************/ InstanceGblSecIncrLong (&AccountingPtr->CacheLoadCount); CacheLoadCount++; CacheHits0++; /* move entry to the head of the cache list */ ListMoveHead (&CacheList, captr); if (rqptr->rqPathSet.CachePermanent) { /* permanent cache entry */ captr->EntryPermanent = true; CachePermEntryCount++; CacheEntryCount--; CachePermMemoryInUse += rqptr->rqCache.ContentBufferSize; } else CacheMemoryInUse += rqptr->rqCache.ContentBufferSize; captr->ContentPtr = rqptr->rqCache.ContentPtr; captr->EntrySize = rqptr->rqCache.ContentBufferSize; captr->ContentLength = rqptr->rqCache.ContentLength; rqptr->rqCache.ContentPtr = NULL; captr->HitCount = 0; captr->DataLoading = false; captr->EntryValid = true; /* quadword time file was loaded/validated, created, last modified */ PUT_QUAD_QUAD (rqptr->rqTime.BeginTime64, captr->LoadTime64); PUT_QUAD_QUAD (rqptr->rqTime.BeginTime64, captr->ValidateTime64); /* some file details (if applicable) */ if (ftkptr = rqptr->FileTaskPtr) { captr->EndOfFileVbn = ftkptr->FileOds.FileQio.EndOfFileVbn; captr->FirstFreeByte = ftkptr->FileOds.FileQio.FirstFreeByte; captr->UicGroup = (ftkptr->FileOds.FileQio.AtrUic & 0x0fff0000) >> 16; captr->UicMember = (ftkptr->FileOds.FileQio.AtrUic & 0x0000ffff); captr->Protection = ftkptr->FileOds.FileQio.AtrFpro; PUT_QUAD_QUAD (ftkptr->FileOds.FileQio.CdtTime64, captr->CdtTime64); PUT_QUAD_QUAD (ftkptr->FileOds.FileQio.RdtTime64, captr->RdtTime64); strcpy (captr->EntityTag, ftkptr->EntityTag); /* copy the entire on-disk structure from file to cache */ OdsCopyStructure (&captr->FileOds, &ftkptr->FileOds); /* the cache entry will reuse the original content handler (if any) */ captr->ContentHandlerFunction = ftkptr->ContentHandlerFunction; /* drop through to buffer the content-type */ cptr = ftkptr->ContentTypePtr; } else { captr->FirstFreeByte = captr->EndOfFileVbn = captr->UicGroup = captr->UicMember = captr->Protection = 0; memset (&captr->FileOds, 0, sizeof(captr->FileOds)); PUT_QUAD_QUAD (rqptr->rqTime.BeginTime64, captr->CdtTime64); PUT_QUAD_QUAD (rqptr->rqTime.BeginTime64, captr->RdtTime64); strcpy (captr->EntityTag, rqptr->rqCache.ResponseEntityTag); captr->ContentHandlerFunction = NULL; /* add the path purely for cache report purposes */ zptr = (sptr = captr->FileOds.ExpFileName) + sizeof(captr->FileOds.ExpFileName)-1; for (cptr = rqptr->ServicePtr->ServerHostPort; *cptr && sptr < zptr; *sptr++ = *cptr++); for (cptr = rqptr->rqHeader.RequestUriPtr; *cptr && *cptr != '?' && sptr < zptr; *sptr++ = *cptr++); if (rqptr->rqPathSet.CacheQuery) { /* those cached regardless include any request query string */ while (*cptr && sptr < zptr) *sptr++ = *cptr++; } *sptr = '\0'; /* drop through to buffer the content-type */ cptr = rqptr->rqCache.ContentTypePtr; } /* note if the cached content contains any CGI header fields */ if (captr->FromScript && rqptr->rqCgi.HeaderLength) captr->CgiHeaderLength = rqptr->rqCgi.HeaderLength; /* buffer the content type */ if (!cptr) cptr = ""; zptr = (sptr = captr->ContentType) + sizeof(captr->ContentType)-1; while (*cptr && sptr < zptr) *sptr++ = *cptr++; if (sptr >= zptr) ErrorNoticed (rqptr, SS$_RESULTOVF, NULL, FI_LI); *sptr = '\0'; if (GzipResponse) { /*************************************/ /* generated GZIP compressed content */ /*************************************/ if (captr->ContentLength) if (GzipShouldDeflate (rqptr, captr->ContentType, captr->ContentLength)) GzipDeflateCache (rqptr, captr); } /********************/ /* set entry expiry */ /********************/ /* for max-age, etc. */ captr->LoadSeconds = HttpdTickSecond; captr->ExpiresAfterPeriod = rqptr->rqPathSet.CacheExpiresAfter; if (captr->ExpiresAfterPeriod == CACHE_EXPIRES_DAY) captr->ExpiresAfterTime = HttpdNumTime[2]; else if (captr->ExpiresAfterPeriod == CACHE_EXPIRES_HOUR) captr->ExpiresAfterTime = HttpdNumTime[3]; else if (captr->ExpiresAfterPeriod == CACHE_EXPIRES_MINUTE) captr->ExpiresAfterTime = HttpdNumTime[4]; else if (captr->ExpiresAfterPeriod == CACHE_EXPIRES_NONE) captr->ExpiresTickSecond = 0xffffffff; else if (captr->ExpiresAfterPeriod) captr->ExpiresTickSecond = HttpdTickSecond + captr->ExpiresAfterPeriod; else captr->ExpiresTickSecond = HttpdTickSecond + CacheValidateSeconds; captr->GuardSeconds = rqptr->rqPathSet.CacheGuardSeconds; if (captr->GuardSeconds) captr->GuardTickSecond = HttpdTickSecond + captr->GuardSeconds; else captr->GuardTickSecond = HttpdTickSecond + CacheGuardSeconds; /* if we're using too much cache memory */ if ((CacheMemoryInUse >> 10) > CacheTotalKBytesMax) { /******************/ /* reclaim memory */ /******************/ CacheReclaimCount++; ReclaimEntryCount = ReclaimEntryBytes = 0; /* mark the current one just so *it* won't be reclaimed */ captr->InUseCount++; /* process the cache entry list from least to most recent */ for (leptr = CacheList.TailPtr; leptr; leptr = leptr->PrevPtr) { /* remember, use a separate pointer or we get very confused :^) */ lcaptr = (FILE_CENTRY*)leptr; if (WATCHPNT(rqptr) && WATCH_MODULE(WATCH_MOD_CACHE) && WATCH_MODULE(WATCH_MOD__DETAIL)) WatchThis (WATCHALL, WATCH_MOD_CACHE, "!&Z !UL !UL !&B !&B !UL", captr->FileOds.ExpFileName, captr->EntrySize, captr->ContentLength, captr->EntryValid, captr->EntryRevalidating, captr->InUseCount); /* if it's permanent or in use in some way then just continue */ if (lcaptr->EntryPermanent || lcaptr->InUseCount) continue; ReclaimEntryCount++; ReclaimEntryBytes += lcaptr->EntrySize; /* entry no longer valid */ CacheRemoveEntry (lcaptr, false); lcaptr->EntryReclaimed = true; /* if we've reclaimed enough memory */ if ((CacheMemoryInUse >> 10) <= CacheTotalKBytesMax) break; } /* remove the reclaim prophylactic */ captr->InUseCount--; if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE load, reclaim !UL entries, !UL kBytes", ReclaimEntryCount, ReclaimEntryBytes >> 10); } } /*****************************************************************************/ /* Copy the referenced data into the cache pre-allocated buffer (if there's still space available). */ int CacheLoadData ( REQUEST_STRUCT *rqptr, char *DataPtr, int DataLength ) { /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "CacheLoadData() file:!&B cgi:!&B net:!&B !UL+!UL=!UL !UL-!UL=!UL", rqptr->rqCache.LoadFromFile, rqptr->rqCache.LoadFromCgi, rqptr->rqCache.LoadFromNet, rqptr->rqCache.ContentLength, DataLength, rqptr->rqCache.ContentLength + DataLength, rqptr->rqCache.ContentBufferSize, rqptr->rqCache.ContentLength + DataLength, rqptr->rqCache.ContentBufferSize - rqptr->rqCache.ContentLength - DataLength); if (DataLength <= rqptr->rqCache.ContentRemaining) { memcpy (rqptr->rqCache.CurrentPtr, DataPtr, DataLength); rqptr->rqCache.CurrentPtr += DataLength; rqptr->rqCache.ContentLength += DataLength; rqptr->rqCache.ContentRemaining -= DataLength; return (SS$_NORMAL); } return (rqptr->rqCache.LoadStatus = SS$_BUFFEROVF_ERROR); } /*****************************************************************************/ /* Allocate a cache entry ready for data loading. Returns true if entry available, false if not. Sets 'rqCache.CacheEntry' to point to the allocated entry. */ BOOL CacheAllocateEntry (REQUEST_STRUCT *rqptr) { int HashValue; FILE_CENTRY *captr, *lcaptr; LIST_ENTRY *leptr; MD5_HASH *md5ptr; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "CacheAllocateEntry() !16&H", rqptr->FileTaskPtr ? &rqptr->FileTaskPtr->Md5Hash : &rqptr->Md5HashPath); if (rqptr->FileTaskPtr) md5ptr = &rqptr->FileTaskPtr->Md5Hash; else md5ptr = &rqptr->Md5HashPath; /**********************************/ /* check it doesn't already exist */ /**********************************/ /* 12 bit hash value, 0..4095 from a fixed 3 bytes of the MD5 hash */ HashValue = md5ptr->HashLong[0] & 0xfff; for (captr = CacheHashTable[HashValue]; captr; captr = captr->HashCollisionNextPtr) { if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "file:!&B valid:!&B revalidating:!&B !AZ !16&H !&?match\rno-match\r", captr->FromFile, captr->EntryValid, captr->EntryRevalidating, captr->FileOds.ExpFileName, &captr->Md5Hash, MATCH16 (&captr->Md5Hash, md5ptr)); if (!MATCH16 (&captr->Md5Hash, md5ptr)) continue; if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "entry exists"); return (false); } /**********************/ /* find a cache entry */ /**********************/ captr = (FILE_CENTRY*)CacheList.TailPtr; if (captr && !captr->EntryValid && !captr->InUseCount) { /****************************/ /* reuse invalid tail entry */ /****************************/ if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE load, reuse entry"); } else if (CacheEntryCount + CachePermEntryCount >= CacheEntriesMax) { /***********************/ /* reuse a cache entry */ /***********************/ /* process the cache entry list from least to most recent */ for (leptr = CacheList.TailPtr; leptr; leptr = leptr->PrevPtr) { captr = (FILE_CENTRY*)leptr; if (WATCHPNT(rqptr) && WATCH_MODULE(WATCH_MOD_CACHE) && WATCH_MODULE(WATCH_MOD__DETAIL)) WatchThis (WATCHALL, WATCH_MOD_CACHE, "!&Z !UL !UL !&B !&B !UL", captr->FileOds.ExpFileName, captr->EntrySize, captr->ContentLength, captr->EntryValid, captr->EntryRevalidating, captr->InUseCount); /* if it's permanent or in use in some way then just continue */ if (captr->EntryPermanent || captr->InUseCount) continue; /* if it can be considered frequently hit */ if (captr->EntryValid && CacheFrequentHits && captr->HitCount > CacheFrequentHits && captr->FrequentTickSecond > HttpdTickSecond) continue; /* entry no longer valid */ CacheRemoveEntry (captr, false); if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE load, reuse entry"); break; } /* if we got to the end of the list */ if (!leptr) { /**********************/ /* all entries in use */ /**********************/ if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE load, all entries in use"); return (false); } } else { /*******************/ /* new cache entry */ /*******************/ captr = VmGet (sizeof(FILE_CENTRY)); CacheEntryCount++; /* add it to the list */ ListAddTail (&CacheList, captr, LIST_ENTRY_TYPE_CACHE); if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE load, new entry"); } /**************/ /* init entry */ /**************/ captr->EntryReclaimed = captr->EntryRevalidating = captr->EntryValid = captr->FromScript = captr->FromNet = captr->Purge = captr->PurgeCompletely = false; captr->ContentPtr = NULL; captr->CgiHeaderLength = captr->EntrySize = 0; if (rqptr->FileTaskPtr) captr->FromFile = true; else captr->FromFile = false; captr->DataLoading = true; captr->InUseCount++; /*************************/ /* add to the hash table */ /*************************/ memcpy (&captr->Md5Hash, md5ptr, sizeof(MD5_HASH)); /* 12 bit hash value, 0..4095 from a fixed 3 bytes of the MD5 hash */ HashValue = md5ptr->HashLong[0] & 0xfff; if (!CacheHashTable[HashValue]) { /* set hash table index */ CacheHashTable[HashValue] = captr; captr->HashCollisionPrevPtr = captr->HashCollisionNextPtr = NULL; } else { /* add to head of hash-collision list */ lcaptr = CacheHashTable[HashValue]; lcaptr->HashCollisionPrevPtr = captr; captr->HashCollisionPrevPtr = NULL; captr->HashCollisionNextPtr = lcaptr; CacheHashTable[HashValue] = captr; } rqptr->rqCache.EntryPtr = captr; return (true); } /*****************************************************************************/ /* Purge a cache entry, either just the data, or completely from the cache list. */ void CacheRemoveEntry ( FILE_CENTRY *captr, BOOL Completely ) { int HashValue; /*********/ /* begin */ /*********/ if (WATCH_MODULE(WATCH_MOD_CACHE)) WatchThis (WATCHALL, WATCH_MOD_CACHE, "CacheRemoveEntry() !&B", Completely); if (captr->InUseCount) { if (Completely) captr->Purge = captr->PurgeCompletely = true; else captr->Purge = true; return; } /* note the number of entries loaded but never subsequently hit */ if (captr->EntryValid && !captr->HitCount) CacheNoHitsCount++; if (captr->EntryPermanent) { /* permanent entry reverts to volatile when purged */ captr->EntryPermanent = false; if (captr->ContentPtr) { VmFreePermCache (captr->ContentPtr, FI_LI); CachePermMemoryInUse -= captr->EntrySize; captr->ContentPtr = NULL; captr->EntrySize = 0; } if (captr->GzipContentPtr) { VmFreePermCache (captr->GzipContentPtr, FI_LI); CachePermMemoryInUse -= captr->GzipEntrySize; captr->GzipContentPtr = NULL; captr->GzipContentLength = captr->GzipEntrySize = 0; } CacheEntryCount++; CachePermEntryCount--; } else { if (captr->ContentPtr) { VmFreeCache (captr->ContentPtr, FI_LI); CacheMemoryInUse -= captr->EntrySize; captr->ContentPtr = NULL; captr->EntrySize = 0; } if (captr->GzipContentPtr) { VmFreeCache (captr->GzipContentPtr, FI_LI); CacheMemoryInUse -= captr->GzipEntrySize; captr->GzipContentPtr = NULL; captr->GzipContentLength = captr->GzipEntrySize = 0; } } captr->DataLoading = captr->EntryValid = captr->Purge = captr->PurgeCompletely = false; /* 12 bit hash value, 0..4095 from a fixed 3 bytes of the MD5 hash */ HashValue = captr->Md5Hash.HashLong[0] & 0xfff; if ((FILE_CENTRY*)(CacheHashTable[HashValue]) == captr) { /* must be at the head of any collision list */ CacheHashTable[HashValue] = captr->HashCollisionNextPtr; if (captr->HashCollisionNextPtr) captr->HashCollisionNextPtr->HashCollisionPrevPtr = captr->HashCollisionPrevPtr; } else { /* if somewhere along the collision list */ if (captr->HashCollisionPrevPtr) captr->HashCollisionPrevPtr->HashCollisionNextPtr = captr->HashCollisionNextPtr; /* *** SITE OF ONE OF MY MOST STUPID AND COSTLY PROGRAMMING ERRORS *** If this isn't an argument against coding for speed and efficiency instead of with tested and common code routines I don't know what is! */ if (captr->HashCollisionNextPtr) captr->HashCollisionNextPtr->HashCollisionPrevPtr = captr->HashCollisionPrevPtr; } captr->HashCollisionPrevPtr = captr->HashCollisionNextPtr = NULL; if (Completely) { ListRemove (&CacheList, captr); VmFree (captr, FI_LI); } } /*****************************************************************************/ /* Scan through the cache list. If a cache entry is currently not in use then free the data memory associated with it. If purge completely then also remove the entry from the list and free it's memory. If the entry is currently in use then mark it for purge and if necessary for complete removal. */ void CachePurge ( BOOL Completely, int *PurgeCountPtr, int *MarkedForPurgeCountPtr ) { int PurgeCount, MarkedForPurgeCount; FILE_CENTRY *captr; LIST_ENTRY *leptr; /*********/ /* begin */ /*********/ if (WATCH_MODULE(WATCH_MOD_CACHE)) WatchThis (WATCHALL, WATCH_MOD_CACHE, "CachePurge() !&B", Completely); CacheZeroCounters (); PurgeCount = MarkedForPurgeCount = 0; /* do it backwards! seeing they are pushed to the tail of the list */ leptr = CacheList.TailPtr; while (leptr) { captr = (FILE_CENTRY*)leptr; /* now, before stuffing around with the entry get the next one */ leptr = leptr->PrevPtr; if (WATCH_MODULE(WATCH_MOD_CACHE) && WATCH_MODULE(WATCH_MOD__DETAIL)) WatchThis (WATCHALL, WATCH_MOD_CACHE, "!&Z !UL !UL !&B !&B !UL", captr->FileOds.ExpFileName, captr->EntrySize, captr->ContentLength, captr->EntryValid, captr->EntryRevalidating, captr->InUseCount); if (captr->InUseCount) { if (Completely) captr->Purge = captr->PurgeCompletely = true; else captr->Purge = true; MarkedForPurgeCount++; } else { if (captr->EntrySize) { CacheRemoveEntry (captr, Completely); PurgeCount++; } } } if (PurgeCountPtr) *PurgeCountPtr = PurgeCount; if (MarkedForPurgeCountPtr) *MarkedForPurgeCountPtr = MarkedForPurgeCount; } /*****************************************************************************/ /* Return a report on cache usage. This function blocks while executing. */ void CacheReport ( REQUEST_STRUCT *rqptr, BOOL IncludeEntries ) { static char BeginPageFao [] = "

\n\ \n\
\n\ \ \n\ \n\ \n\ \ \n\ \ \n\ \ \n\ \ \n\ \ \n\ \ \n\ \ \n\ \ \n\ \ \n\ \ \n\ \ \n\ \ \n\ \ \n\ \ \n\ \ \n\ \ \n\
Configuration
Caching:!AZ
Memory  /Permanent:!ULkB
/Volatile:!ULkB
/Max:!ULkB
Entries  /Permanent:!UL
/Volatile:!UL
/Max:!UL
/Valid:!UL
/In-Use:!UL
/Loading:!UL
/Reclaimed:!UL
Max File Size:!ULkB
Guard:!ULseconds
Validate:!ULseconds
Frequent  /Hits:!ULhits
/Within:!ULseconds
\n\ \ 		\n\ \n\ \n\ \n\ \n\ \n\ \ \n\ \ \n\ \n\ \ \n\ \ \n\ \ \n\ \n\ \ \ \ \n\ \n\ \n\ \n\ \n\ \n\
Activity
Search:!UL
 Hash Table
Hit:!UL(!UL%)
Miss:!UL(!UL%)
Collision
Total:!UL(!UL%)
Hit:!UL(!UL%)
Miss:!UL(!UL%)
Memory
Loading:!UL!UL MB
Reclaim:!UL(!UL%)
 GZIP Deflate
Count:!UL
Bytes In:!&,@SQ
Bytes Out:!&,@SQ
Ratio:!UL%
\n\ \ 		\n\ \n\ \n\ \n\ \n\ \ \n\ \ \n\ \ \n\ \ \n\ \ \n\ \ \n\
Entries
Load:!UL
Not Hit:!UL(!UL%)
  Total Hit:!UL(!UL%)
1-9:!UL(!UL%)
10-99:!UL(!UL%)
100-999:!UL(!UL%)
>1000:!UL(!UL%)
\n\ \
\n\ !&@"; /* the final column just adds a little white-space on the page far right */ static char EntriesFao [] = "

\n\ \ \ \ \ \ \ \ \ \ \ \ \n"; /* the empty 99% column just forces the rest left with long request URIs */ static char CacheFao [] = "\ \ \ \ \ \ \ \ \ \ \ \ \n\ \ \ \ \n"; static char EmptyCacheFao [] = "\ \n"; static char EntriesButtonFao [] = "
FlagsSize / LengthGZ Size / Len / %In-UseHashRevisedValidatedLoadedHit / 304
!#ZL  \ !&?P\rV\r\ !&?F\r\r!&?N\r\r!&?S\r\r\ !&?C\rC\r\ !&?T\rT\r\   !UL / !UL!UL / !UL / !UL!UL!UL / !UL!&@!&@!20%D!20%D!UL / !UL
!AZ
000empty
\n\

Entries\n\ !AZ\ \n\ \n\ \n"; static char EndPageFao [] = "\n\ !AZ\ \n\ \n\ \n"; int cnt, status, Count, GzipDeflatePercent, HashCollisionListLength, InUseEntryCount, LoadingEntryCount, ReclaimedEntryCount, ValidEntryCount; unsigned long FaoVector [64]; unsigned long *vecptr; char *cptr, *sptr, *LastColPtr; FILE_CENTRY *captr; LIST_ENTRY *leptr; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "CacheReport()"); InUseEntryCount = LoadingEntryCount = ReclaimedEntryCount = ValidEntryCount = 0; for (leptr = CacheList.HeadPtr; leptr; leptr = leptr->NextPtr) { captr = (FILE_CENTRY*)leptr; if (captr->EntryValid) ValidEntryCount++; if (captr->EntryReclaimed) ReclaimedEntryCount++; if (captr->InUseCount) InUseEntryCount++; if (captr->DataLoading) LoadingEntryCount++; } AdminPageTitle (rqptr, "Cache Report"); vecptr = FaoVector; if (CacheEnabled) *vecptr++ = "[enabled]"; else *vecptr++ = "[disabled]"; *vecptr++ = CachePermMemoryInUse >> 10; *vecptr++ = CacheMemoryInUse >> 10; *vecptr++ = CacheTotalKBytesMax; *vecptr++ = CachePermEntryCount; *vecptr++ = CacheEntryCount; *vecptr++ = CacheEntriesMax; *vecptr++ = ValidEntryCount; *vecptr++ = InUseEntryCount; *vecptr++ = LoadingEntryCount; *vecptr++ = ReclaimedEntryCount; *vecptr++ = CacheEntryKBytesMax; *vecptr++ = CacheGuardSeconds; *vecptr++ = CacheValidateSeconds; *vecptr++ = CacheFrequentHits; *vecptr++ = CacheFrequentSeconds; *vecptr++ = CacheHashTableCount; *vecptr++ = CacheHashTableHitCount; if (CacheHashTableCount) *vecptr++ = CacheHashTableHitCount * 100 / CacheHashTableCount; else *vecptr++ = 0; *vecptr++ = CacheHashTableMissCount; if (CacheHashTableCount) *vecptr++ = CacheHashTableMissCount * 100 / CacheHashTableCount; else *vecptr++ = 0; *vecptr++ = CacheHashTableCollsnCount; if (CacheHashTableCount) *vecptr++ = CacheHashTableCollsnCount * 100 / CacheHashTableCount; else *vecptr++ = 0; *vecptr++ = CacheHashTableCollsnHitCount; if (CacheHashTableCount) *vecptr++ = CacheHashTableCollsnHitCount * 100 / CacheHashTableCount; else *vecptr++ = 0; *vecptr++ = CacheHashTableCollsnMissCount; if (CacheHashTableCount) *vecptr++ = CacheHashTableCollsnMissCount * 100 / CacheHashTableCount; else *vecptr++ = 0; *vecptr++ = CacheCurrentlyLoading; *vecptr++ = CacheCurrentlyLoadingInUse >> 10; *vecptr++ = CacheReclaimCount; if (CacheLoadCount) *vecptr++ = CacheReclaimCount * 100 / CacheLoadCount; else *vecptr++ = 0; *vecptr++ = CacheGzipDeflateCount; *vecptr++ = &CacheGzipDeflateBytesIn; *vecptr++ = &CacheGzipDeflateBytesOut; *vecptr++ = QuadPercentOf (&CacheGzipDeflateBytesOut, &CacheGzipDeflateBytesIn); *vecptr++ = CacheLoadCount; *vecptr++ = CacheHits0; *vecptr++ = PercentOf(CacheHits0,CacheLoadCount); *vecptr++ = CacheHitCount; *vecptr++ = PercentOf(CacheHitCount,CacheLoadCount); *vecptr++ = CacheHits10; *vecptr++ = PercentOf(CacheHits10,CacheLoadCount); *vecptr++ = CacheHits100; *vecptr++ = PercentOf(CacheHits100,CacheLoadCount); *vecptr++ = CacheHits1000; *vecptr++ = PercentOf(CacheHits1000,CacheLoadCount); *vecptr++ = CacheHits1000plus; *vecptr++ = PercentOf(CacheHits1000plus,CacheLoadCount); if (CacheEnabled && !IncludeEntries) { *vecptr++ = EntriesButtonFao; *vecptr++ = ADMIN_REPORT_CACHE_ENTRIES; *vecptr++ = AdminRefresh(); } else *vecptr++ = ""; status = FaolToNet (rqptr, BeginPageFao, &FaoVector); if (VMSnok (status)) ErrorNoticed (rqptr, status, NULL, FI_LI); if (!CacheEnabled || !IncludeEntries) { rqptr->rqResponse.PreExpired = PRE_EXPIRE_ADMIN; ResponseHeader200 (rqptr, "text/html", &rqptr->NetWriteBufferDsc); AdminEnd (rqptr); return; } /*****************/ /* cache entries */ /*****************/ status = FaolToNet (rqptr, EntriesFao, NULL); if (VMSnok (status)) ErrorNoticed (rqptr, status, NULL, FI_LI); Count = 0; if (WATCHPNT(rqptr) && WATCH_MODULE(WATCH_MOD_CACHE) && WATCH_MODULE(WATCH_MOD__DETAIL)) WatchThis (WATCHALL, WATCH_MOD_CACHE, "!&X !&X", CacheList.HeadPtr, CacheList.TailPtr); /* process the cache entry list from most to least recently hit */ for (leptr = CacheList.HeadPtr; leptr; leptr = leptr->NextPtr) { captr = (FILE_CENTRY*)leptr; if (WATCHPNT(rqptr) && WATCH_MODULE(WATCH_MOD_CACHE) && WATCH_MODULE(WATCH_MOD__DETAIL)) WatchThis (WATCHALL, WATCH_MOD_CACHE, "!&X<-!&X->!&X !&X", leptr->PrevPtr, leptr, leptr->NextPtr, captr); HashCollisionListLength = 0; /* count further down the list */ while (captr->HashCollisionNextPtr) { captr = captr->HashCollisionNextPtr; HashCollisionListLength++; } captr = (FILE_CENTRY*)leptr; /* count further up the list */ while (captr->HashCollisionPrevPtr) { captr = captr->HashCollisionPrevPtr; HashCollisionListLength++; } captr = (FILE_CENTRY*)leptr; vecptr = FaoVector; *vecptr++ = &captr->Md5Hash; if (CacheEntriesMax >= 10000) *vecptr++ = 5; else if (CacheEntriesMax >= 1000) *vecptr++ = 4; else *vecptr++ = 3; *vecptr++ = ++Count; *vecptr++ = captr->EntryPermanent; *vecptr++ = captr->FromFile; *vecptr++ = captr->FromNet; *vecptr++ = captr->FromScript; *vecptr++ = captr->CgiHeaderLength; *vecptr++ = captr->ContentType[0]; *vecptr++ = captr->EntrySize; *vecptr++ = captr->ContentLength; *vecptr++ = captr->GzipEntrySize; *vecptr++ = captr->GzipContentLength; *vecptr++ = PercentOf(captr->GzipContentLength,captr->ContentLength); *vecptr++ = captr->InUseCount; /* 12 bit hash value, 0..4095 from a fixed 3 bytes of the MD5 hash */ *vecptr++ = captr->Md5Hash.HashLong[0] & 0xfff; *vecptr++ = HashCollisionListLength; if (captr->FromFile) { *vecptr++ = "!20%D"; *vecptr++ = &captr->RdtTime64; } else *vecptr++ = "n/a"; if (captr->FromFile && captr->EntryValid) { *vecptr++ = "!20%D !UL"; *vecptr++ = &captr->ValidateTime64; *vecptr++ = captr->ValidatedCount; } else { if (captr->EntryReclaimed) *vecptr++ = "RECLAIMED"; else if (!captr->EntryValid) *vecptr++ = "INVALID"; else if (!captr->FromFile) *vecptr++ = "n/a"; else *vecptr++ = "?"; } *vecptr++ = &captr->LoadTime64; *vecptr++ = &captr->HitTime64; *vecptr++ = captr->HitCount; *vecptr++ = captr->HitNotModifiedCount; *vecptr++ = captr->FileOds.ExpFileName; status = FaolToNet (rqptr, CacheFao, &FaoVector); if (VMSnok (status)) ErrorNoticed (rqptr, status, NULL, FI_LI); } if (!CacheList.HeadPtr) { status = FaolToNet (rqptr, EmptyCacheFao, NULL); if (VMSnok (status)) ErrorNoticed (rqptr, status, NULL, FI_LI); } vecptr = FaoVector; *vecptr++ = AdminRefresh(); status = FaolToNet (rqptr, EndPageFao, &FaoVector); if (VMSnok (status)) ErrorNoticed (rqptr, status, NULL, FI_LI); rqptr->rqResponse.PreExpired = PRE_EXPIRE_ADMIN; ResponseHeader200 (rqptr, "text/html", &rqptr->NetWriteBufferDsc); AdminEnd (rqptr); } /*****************************************************************************/ /* Return a report on a single cache entry. */ void CacheReportEntry ( REQUEST_STRUCT *rqptr, char *Md5HashHexString ) { static char BeginPageFao [] = "

\n\ \n\
\n\ \n\ \ \n\ \ \n\ \n\ \n\ \n\ \ \n\ \ \n\ \n\ \n\ \ \n\ \n\ \n\ \n\ \ \n\
Resource:!&;AZ
Entity Tag:!AZ
Flags:\ !&?Permanent\rVolatile\r, \ !&?File\r\r!&?Network\r\r!&?Script\r\r, \ !&?CGI-fields\rCGI-fields\r, \ !&?Content-Type\rContent-Type\r, \
Size:!UL byte!%s
Length:!UL byte!%s
GZIP Size:!UL byte!%s
GZIP Length:!UL byte!%s  (!UL%)
CGI Fields:!UL byte!%s
Content-Type:!&;AZ
In-Use:!UL
Hash/Idx/Colsn:!16&H / !UL / !UL
Revised:!&@
Validated:!&@
Loaded:!20%D
Hit:!20%D, !UL time!%s, 304 response !UL time!%s
\n\
\n\ 

";

   int  idx, status,
        ItemCount,
        HashCollisionListLength;
   unsigned long  FaoVector [64];
   unsigned long  *vecptr;
   char  *cptr;
   MD5_HASH  Md5Hash;
   FILE_CENTRY  *captr;
   LIST_ENTRY  *leptr;

   /*********/
   /* begin */
   /*********/

   if (WATCHMOD (rqptr, WATCH_MOD_CACHE))
      WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE,
                 "CacheReportEntry() !&Z", Md5HashHexString);

   /* convert hex string back into binary hash */
   cptr = Md5HashHexString;
   for (idx = 0; idx < sizeof(Md5Hash); idx++)
   {
      if (*cptr >= '0' && *cptr <= '9')
         Md5Hash.HashChar[idx] = (*cptr - '0') << 4;
      else
      if (TOUP(*cptr) >= 'A' && TOUP(*cptr) <= 'F')
         Md5Hash.HashChar[idx] = (*cptr - '7') << 4;
      if (*cptr) cptr++;
      if (*cptr >= '0' && *cptr <= '9')
         Md5Hash.HashChar[idx] |= (*cptr - '0') & 0xf;
      else
      if (TOUP(*cptr) >= 'A' && TOUP(*cptr) <= 'F')
         Md5Hash.HashChar[idx] |= (*cptr - '7') & 0xf;
      if (*cptr) cptr++;
   }

   captr = NULL;
   ItemCount = 0;
   for (leptr = CacheList.HeadPtr; leptr; leptr = leptr->NextPtr)
   {
      captr = (FILE_CENTRY*)leptr;
      if (!MATCH0 (&captr->Md5Hash, &Md5Hash, sizeof(Md5Hash))) continue;
      break;
   }
   if (!leptr)
   {
      rqptr->rqResponse.HttpStatus = 400;
      ErrorGeneral (rqptr, "Entry not found.", FI_LI);
      AdminEnd (rqptr);
      return;
   }

   AdminPageTitle (rqptr, "Cache Entry Report");

   HashCollisionListLength = 0;
   /* count further down the list */
   while (captr->HashCollisionNextPtr)
   {
         captr = captr->HashCollisionNextPtr;
         HashCollisionListLength++;
   }
   captr = (FILE_CENTRY*)leptr;
   /* count further up the list */
   while (captr->HashCollisionPrevPtr)
   {
      captr = captr->HashCollisionPrevPtr;
      HashCollisionListLength++;
   }
   captr = (FILE_CENTRY*)leptr;

   vecptr = FaoVector;

   *vecptr++ = captr->FileOds.ExpFileName;
   *vecptr++ = captr->EntityTag;
   *vecptr++ = captr->EntryPermanent;
   *vecptr++ = captr->FromFile;
   *vecptr++ = captr->FromNet;
   *vecptr++ = captr->FromScript;
   *vecptr++ = captr->CgiHeaderLength;
   *vecptr++ = captr->ContentType[0];
   *vecptr++ = captr->EntrySize;
   *vecptr++ = captr->ContentLength;
   *vecptr++ = captr->GzipEntrySize;
   *vecptr++ = captr->GzipContentLength;
   *vecptr++ = PercentOf(captr->GzipContentLength,captr->ContentLength);
   *vecptr++ = captr->CgiHeaderLength;
   if (captr->ContentType[0])
      *vecptr++ = captr->ContentType;
   else
      *vecptr++ = "n/a";
   *vecptr++ = captr->InUseCount;
   *vecptr++ = &captr->Md5Hash.HashLong;
   /* 12 bit hash value, 0..4095 from a fixed 3 bytes of the MD5 hash */
   *vecptr++ = captr->Md5Hash.HashLong[0] & 0xfff;
   *vecptr++ = HashCollisionListLength;

   if (captr->FromFile)
   {
      *vecptr++ = "!20%D";
      *vecptr++ = &captr->RdtTime64;
   }
   else
      *vecptr++ = "n/a";

   if (captr->FromFile &&
       captr->EntryValid)
   {
      *vecptr++ = "!20%D, !UL time!%s";
      *vecptr++ = &captr->ValidateTime64;
      *vecptr++ = captr->ValidatedCount;
   }
   else
   {
      if (captr->EntryReclaimed)
         *vecptr++ = "RECLAIMED";
      else
      if (!captr->EntryValid)
         *vecptr++ = "INVALID";
      else
      if (!captr->FromFile)
         *vecptr++ = "n/a";
      else
         *vecptr++ = "?";
   }

   *vecptr++ = &captr->LoadTime64;
   *vecptr++ = &captr->HitTime64;
   *vecptr++ = captr->HitCount;
   *vecptr++ = captr->HitNotModifiedCount;

   status = FaolToNet (rqptr, BeginPageFao, &FaoVector);
   if (VMSnok (status)) ErrorNoticed (rqptr, status, NULL, FI_LI);

   vecptr = FaoVector;

   if (captr->ContentPtr)
   {
      CacheDumpData (rqptr, captr->ContentPtr, captr->ContentLength);
      NetWriteBuffered (rqptr, NULL,
"
", -1);
   }

   if (captr->GzipContentPtr)
   {
      CacheDumpData (rqptr, captr->GzipContentPtr, captr->GzipContentLength);
      NetWriteBuffered (rqptr, NULL,
"
", -1);
   }

   NetWriteBuffered (rqptr, NULL, "
\n\n\n", -1); rqptr->rqResponse.PreExpired = PRE_EXPIRE_ADMIN; ResponseHeader200 (rqptr, "text/html", &rqptr->NetWriteBufferDsc); AdminEnd (rqptr); } /*****************************************************************************/ /* Dump data a la WATCH. */ CacheDumpData ( REQUEST_STRUCT *rqptr, char *DataPtr, int DataLength ) { /* 32 bytes by 128 lines comes out to 4096 bytes, the default buffer-full */ #define MAX_LINES 128 #define BYTES_PER_LINE 32 #define BYTES_PER_GROUP 4 #define GROUPS_PER_LINE (BYTES_PER_LINE / BYTES_PER_GROUP) #define CHARS_PER_LINE ((BYTES_PER_LINE * 3) + GROUPS_PER_LINE + 1) #define HTML_ESCAPE_SPACE 1024 static char HexDigits [] = "0123456789ABCDEF"; static char Woops [] = "ERROR: Buffer overflow!"; int ByteCount, CurrentDataCount, DataCount; char *cptr, *sptr, *zptr, *CurrentDataPtr, *CurrentDumpPtr; char DumpBuffer [(CHARS_PER_LINE * MAX_LINES)+HTML_ESCAPE_SPACE+1]; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_CACHE)) WatchThis (WATCHITM(rqptr), WATCH_MOD_CACHE, "CacheDumpData() !&A !UL", DataPtr, DataLength); if (!DataPtr) return; zptr = (sptr = DumpBuffer) + sizeof(DumpBuffer)-1; cptr = DataPtr; DataCount = DataLength; while (DataCount) { CurrentDumpPtr = sptr; CurrentDataPtr = cptr; CurrentDataCount = DataCount; ByteCount = BYTES_PER_LINE; while (ByteCount && DataCount) { if (sptr < zptr) *sptr++ = HexDigits[*(unsigned char*)cptr >> 4]; if (sptr < zptr) *sptr++ = HexDigits[*(unsigned char*)cptr & 0xf]; cptr++; DataCount--; ByteCount--; if (!(ByteCount % BYTES_PER_GROUP) && sptr < zptr) *sptr++ = ' '; } while (ByteCount) { if (sptr < zptr) *sptr++ = ' '; if (sptr < zptr) *sptr++ = ' '; ByteCount--; if (!(ByteCount % BYTES_PER_GROUP) && sptr < zptr) *sptr++ = ' '; } cptr = CurrentDataPtr; DataCount = CurrentDataCount; ByteCount = BYTES_PER_LINE; while (ByteCount && DataCount) { if (isalnum(*cptr) || ispunct(*cptr) || *cptr == ' ') { if (*cptr == '<') { if (sptr < zptr) *sptr++ = '&'; if (sptr < zptr) *sptr++ = 'l'; if (sptr < zptr) *sptr++ = 't'; if (sptr < zptr) *sptr++ = ';'; cptr++; } else if (*cptr == '&') { if (sptr < zptr) *sptr++ = '&'; if (sptr < zptr) *sptr++ = 'a'; if (sptr < zptr) *sptr++ = 'm'; if (sptr < zptr) *sptr++ = 'p'; if (sptr < zptr) *sptr++ = ';'; cptr++; } else { if (sptr < zptr) *sptr++ = *cptr; cptr++; } } else { if (sptr < zptr) *sptr++ = '.'; cptr++; } DataCount--; ByteCount--; } /* ensure there is a right margin using a couple of spaces */ if (sptr < zptr) *sptr++ = ' '; if (sptr < zptr) *sptr++ = ' '; if (sptr < zptr) *sptr++ = '\n'; if (sptr >= zptr) { NetWriteBuffered (rqptr, NULL, Woops, sizeof(Woops)-1); return; } if (!DataCount || !ByteCount) { *sptr = '\0'; NetWriteBuffered (rqptr, NULL, DumpBuffer, sptr - DumpBuffer); zptr = (sptr = DumpBuffer) + sizeof(DumpBuffer)-1; } } } /*****************************************************************************/