Re: [exim] ext3 with maildir++ = huge disk latency and high …

If you just have 45 gigs of data who not go with SSD drives? SSD is
really fast.

On 9/23/2011 1:13 AM, Andrey wrote:
> 23.09.2011 11:31, Janne Pikkarainen пишет:
> Thank you for reply,
>
> BTW, other webserver has almost the same bonnie results (10283ms and
> 5884ms) on ext3 partition with 45GB of data (1.5 millions of files)?!
>
> Hardware and RAID5(also hardware) are the same: HP Proliant DL380 G4
> with SmartArray 6i controller (as I see it comes with 128MB BBWC
> enabler but not kit).
>
> I did not tried to mount fs with barriers disabled. Does it have any
> crititcal risks?
>
> Bonnie tests was performed in the morning when we have a mininmal user
> load.
>
> With regards, Andrey.
>
>> Hello,
>>
>> On 09/23/2011 08:51 AM, Andrey wrote:
>>> Hello,
>>>
>>> I have a production mail server with maildir++ structure and about
>>> 250GB (~10 millions) of files on the ext3 partition on RAID5. It's
>>> mounted with noatime option. These mail server is responsible to local
>>> delivery and storing mail messages.
>>>
>>> System has Debian Squeeze installed and Exim as MDA + Dovecot as
>>> IMAP+POP3 server.
>>>
>>> Bonnie results are terrible. Sequential output for Block and Rewrite
>>> are 10722ms and 9232ms. So if there is a 1000 messages in the mail
>>> queue load is extremely high, delivery time is very big and server can
>>> hang. I did not see such problems with UFS on FreeBSD server.
>>>
>>> As I understand ext3 file system is really bad for such configurations
>>> with Maildir++ (many smaill files)? Is there a way to decrease disk
>>> latency on ext3 or speed up it?
>>>
>>> With regards, Andrey
>>>
>>> ___
>>
>> (replying off-list, so the ext3 developers will not start a flamewar)
>>
>> In my opinion ext3 is a terrible file system for your kind of workload,
>> especially if you have lots of concurrent clients accessing their
>> mailboxes. Even though ext3 has evolved over the years and has gained
>> features such as directory indexes, it still is not good for tens of
>> million of frequently changing small files with lots of concurrency.
>> Been there, done that, not gonna do it again. I administer servers with
>> 50 000 - 100 000 user accounts, with couple of thousands active IMAP
>> connections.
>>
>> Personally I switched from ext3 to ReiserFS many years ago and happily
>> used it between 2004-2008, then after things went downhill from ReiserFS
>> development point of view, I switched to XFS during a server hardware
>> refresh. ReiserFS was excellent, but it really started to slow down if
>> file system was more than 85% full and it also got fragmented over time.
>>
>> XFS has been rock-solid and fast since 2008 for me, but it has an
>> achilles heel of its own: if I need to remove lots of files from a huge
>> directory tree, the delete performance is quite sucky compared to other
>> file systems. This has been improved in the later kernel versions with
>> the new delaylog parameter, but how much, I've not yet tested.
>>
>> All this said, the performance of ext3 should not be THAT bad you are
>> describing. Is the bonnie result done while the server is idle or while
>> it has mail clients accessing it all the time? If you have hardware
>> RAID, is there a battery-backed up write cache and are you sure it's
>> enabled? Also, have you tried to mount your file system with barriers
>> disabled? What kind of server setup you have?
>>
>> Something is very wrong.
>>
>> Best regards,
>>
>> Janne Pikkarainen
>>
>>
>
>