K 10
svn:author
V 7
coneill
K 8
svn:date
V 27
1997-09-08T02:05:04.000000Z
K 7
svn:log
V 2537
Unfortunately this commit contains a few misc things also.

- updates to Makefiles for the removal of -lutil from config.dist
- updates to makehistory and expire to handle rebuilding the dbz files
correctly.

But the major changes here are support for mmaping overview, mmaping
articles, wireformat article and overview indexes.  These should probably
have been committed as separate patches, but they involved such large
changes to article.c that it would have been difficult to separate them.
Now one at a time explanation.

Wireformat Articles
	Articles are assumed to be in wireformat if the first line ends in
\r\n.  Regular articles are still supported and will be transparently
handled.

MMAP'ing articles
	This is a big performance win if you are writing articles in
wireformat, but still helps if your OS has superior mmap performance and/or
supports MADV_DONTNEED.  The article will be mmap'ed when it is requested
and only unmapped when the next article is requested.  This means that
you'll be holding the mapping until the reader requests the next article,
but it also means that you don't have to mmap it twice if the reader does
a mixture of HEAD and BODY commands (rn based readers).  With wireformat
article the article is sent out using a single writev w/no user level
overhead; with regular articles it will be sent out using write, with an 8k
buffer.  If the OS supports MADV_DONTNEED, then nnrpd will issue a madvise
using this, indicating that the OS can free the memory.  At least under
Solaris, this means that your scan rate should go down immensely, and if you
have enough memory to cache overviews then it should be near or at zero.

MMAP'ing overviews
	This reduces the number of context switches needed to access the
overview files and makes the implementation of OVERfind() considerably
simpler.

Overview Indexes
	This eliminates almost all of the cpu overhead and a lot of disk
overhead associated with overview handling.  The basic idea is the
expireover and overchan now understand how to write a binary index into the
same directory that the overview file is stored in.  The index contains
article number and offset pairs, giving nnrpd enough information to find the
start of the overview entry without searching.  NNRPD loads the overview
indexes into memory when XOVER is executed in the group the first time.
The indexes can be changed to store any information that would be useful
with very little work.  This is probably the best place to store an
article # -> token translation.


Obtained from:

END
