Author: Fabricio Archanjo Date: To: Marc Perkel CC: exim-users Subject: Re: [exim] ext3 with maildir++ = huge disk latency and high load
Did you see the iowait ?
On Sun, Sep 25, 2011 at 2:13 PM, Marc Perkel <marc@???> wrote:
> If you just have 45 gigs of data who not go with SSD drives? SSD is really
> fast.
>
>
> On 9/23/2011 1:13 AM, Andrey wrote:
>
>> 23.09.2011 11:31, Janne Pikkarainen пишет:
>> Thank you for reply,
>>
>> BTW, other webserver has almost the same bonnie results (10283ms and
>> 5884ms) on ext3 partition with 45GB of data (1.5 millions of files)?!
>>
>> Hardware and RAID5(also hardware) are the same: HP Proliant DL380 G4 with
>> SmartArray 6i controller (as I see it comes with 128MB BBWC enabler but not
>> kit).
>>
>> I did not tried to mount fs with barriers disabled. Does it have any
>> crititcal risks?
>>
>> Bonnie tests was performed in the morning when we have a mininmal user
>> load.
>>
>> With regards, Andrey.
>>
>> Hello,
>>>
>>> On 09/23/2011 08:51 AM, Andrey wrote:
>>>
>>>> Hello,
>>>>
>>>> I have a production mail server with maildir++ structure and about
>>>> 250GB (~10 millions) of files on the ext3 partition on RAID5. It's
>>>> mounted with noatime option. These mail server is responsible to local
>>>> delivery and storing mail messages.
>>>>
>>>> System has Debian Squeeze installed and Exim as MDA + Dovecot as
>>>> IMAP+POP3 server.
>>>>
>>>> Bonnie results are terrible. Sequential output for Block and Rewrite
>>>> are 10722ms and 9232ms. So if there is a 1000 messages in the mail
>>>> queue load is extremely high, delivery time is very big and server can
>>>> hang. I did not see such problems with UFS on FreeBSD server.
>>>>
>>>> As I understand ext3 file system is really bad for such configurations
>>>> with Maildir++ (many smaill files)? Is there a way to decrease disk
>>>> latency on ext3 or speed up it?
>>>>
>>>> With regards, Andrey
>>>>
>>>> ___
>>>>
>>>
>>> (replying off-list, so the ext3 developers will not start a flamewar)
>>>
>>> In my opinion ext3 is a terrible file system for your kind of workload,
>>> especially if you have lots of concurrent clients accessing their
>>> mailboxes. Even though ext3 has evolved over the years and has gained
>>> features such as directory indexes, it still is not good for tens of
>>> million of frequently changing small files with lots of concurrency.
>>> Been there, done that, not gonna do it again. I administer servers with
>>> 50 000 - 100 000 user accounts, with couple of thousands active IMAP
>>> connections.
>>>
>>> Personally I switched from ext3 to ReiserFS many years ago and happily
>>> used it between 2004-2008, then after things went downhill from ReiserFS
>>> development point of view, I switched to XFS during a server hardware
>>> refresh. ReiserFS was excellent, but it really started to slow down if
>>> file system was more than 85% full and it also got fragmented over time.
>>>
>>> XFS has been rock-solid and fast since 2008 for me, but it has an
>>> achilles heel of its own: if I need to remove lots of files from a huge
>>> directory tree, the delete performance is quite sucky compared to other
>>> file systems. This has been improved in the later kernel versions with
>>> the new delaylog parameter, but how much, I've not yet tested.
>>>
>>> All this said, the performance of ext3 should not be THAT bad you are
>>> describing. Is the bonnie result done while the server is idle or while
>>> it has mail clients accessing it all the time? If you have hardware
>>> RAID, is there a battery-backed up write cache and are you sure it's
>>> enabled? Also, have you tried to mount your file system with barriers
>>> disabled? What kind of server setup you have?
>>>
>>> Something is very wrong.
>>>
>>> Best regards,
>>>
>>> Janne Pikkarainen
>>>
>>>
>>>
>>
>>
> --
> ## List details at https://lists.exim.org/**mailman/listinfo/exim-users<https://lists.exim.org/mailman/listinfo/exim-users>
> ## Exim details at http://www.exim.org/ > ## Please use the Wiki with this list - http://wiki.exim.org/ >
