[ On Saturday, December 4, 2004 at 22:54:37 (-0800), Pete Carah wrote: ]
> Subject: Re: [exim] Small modification for queue runners?
>
> One other thing to consider here is that unix takes *very* long to open a
> file in a directory with 500k files (the spool of 244k messages mentioned).
As you almost hint that depends on the filesystem, the implementation of
the filesystem, and perhaps also the regularity with which the directory
is used.
> One other approach to speeding this up would be a db file (hash or btree)
> parallel to the spool directory (or indexing into it).
No, I don't think so -- at least not at user level.
Simple directory level hashing at the user level would likely suffice if
this were a really big problem for enough people (see the URL below).
It's almost trivial to implement in this kind of application too, and
it's a very widely used solution (e.g. Cyrus IMAP is one e-mail related
example where this technique is used).
> Freebsd has the dirhash option but I don't know if the kernel memory it
> uses for that is persistent when the dir gets *that* big.
With most modern FFS implemenations there's lots of caching of vnodes
and metadata, and with FFS "soft dependencies" there's also much of the
benefit of "async" and "noatime" mounts without their dangers. These
combine to make the affects of accessing directories with many files
less painful than it once was
FreeBSD's "dirhash" is indeed a potential benefit, though it needs
careful tuning and control.
> I don't know
> what other unices (or similar) do here. Mac HFS would be OK (btree dirs;
> worse for sequential reads but quick for opens) but I don't know if OS-X
> uses those or not.
The design goals for SGI's XFS included solving this problem of handling
many files in one directory and their papers and reports suggest they
met this goal quite well (they use B-tree indexing within the directory
file).
FFSv2 apparently handles many files per directory even better than the
old 4.4BSD FFS with softdep. ReiserFS and one called GFS apparently do
well too.
The scripts included in this message might be useful to anyone who wants
to investigate how well any given system behaves:
http://mail-index.netbsd.org/tech-kern/2000/12/19/0016.html
--
Greg A. Woods
+1 416 218-0098 VE3TCP RoboHack <woods@???>
Planix, Inc. <woods@???> Secrets of the Weird <woods@???>
