Re: [exim] malware and defer_ok

Dennis Davis wrote:

> On Mon, 12 Dec 2005, Bill Hacker wrote:
>
>
>>From: Bill Hacker <wbh@???>
>>To: exim <exim-users@???>
>>Date: Mon, 12 Dec 2005 19:06:10 +0800
>>Subject: Re: [exim] malware and defer_ok
>>
>>Riemer Palstra wrote:
>>
>>
>>>On Mon, Dec 12, 2005 at 11:46:53AM +0200, oxo@??? wrote:
>>>
>>>
>>>>1) Does anyone know of monitoring software for FreeBSD/Linux that
>>>>will restart the daemon if it dies.
>>>
>>>
>>>Install DaemonTools and put the daemon under supervise.
>>
>>Bernstein's Revenge? Better to take thermite to the box.
>
>
> By all means keep away from qmail, but in my experience a *lot* of
> his other sofware does exactly what it sez on the tin.

As did a medication called 'brucin', but it was not, and is not, an
optimal solution.

> I've had no
> problems running clamd -- and a lot of other software -- under the
> control of daemontools. All you need is a simple run file of the
> form:
>
>
> #!/bin/sh
>
> clamd=/usr/local/sbin/clamd
>
> exec $clamd
>
>
> and make sure you set:
>
>
> # Don't fork into background.
> # Default: disabled
> #Foreground
> Foreground
>
>
> in /etc/clamd.conf

As with most of Dan's work - simple-appearing on the surface, 'overly
complex' for the job at hand under the covers.
(production user of Qmail for ~5 years, including one 688+ day
uninterrupted run terminated only by failure of an aging HDD)

Dig deeper:

- his toolset includes 'service' as well 'supervise', and installs a
'mini tree' in each directory of a monitored process, in which is
stored, for example, it's own PID info, etc. Not only complex and
scattered about, but potentially very much at-odds with directory perms
and ownership of one's preferred security model.

- some apps, including those we speak of here, are *designed* to
'panic' and abort themselves under certain error conditions
that may, for example, be indicative of a DoS or similar compromise.
What is wanted in such case may be an e-mail or SMS/pager message
calling for human inspection, NOT (only) a knee-jerk restart of the binary.

- most installations have interdependencies that daemontools does not,
by default, handle well at all. For example:
I want PostgreSQL, a 'slow starter', up and running before clamd or
spamd, and both of those worthies alive and well before starting Exim.
Otherwise they will be unable to acess resources they need.

- if/as/when any one of these fall down, I usually prefer downing the
rest of the team and restarting them in the designated sequence rather
than re-starting only the one that has failed.

- if/s/when I wish to perform maintenance or a refresh, I do NOT want to
have to be forced to use a third-party supervisory tool that may itself
be hard to find, as it is in a 'sleep' cycle for much of its life,
before I can take, for example, Exim off-line for a brief period. Nor
struggle adapting supervise to an upgraded app that may store its binary
under a different name or in a revised location.

By Contrast - just two of the several simpler tools:

1) 'monitord' - Ordinarily run only on the host being monitored:

- keeps all settings in ONE place: ~/etc/monitord.conf

- has human-readable settings for timing between checks AND delays
allowed after possibly-lengthy startup before resumng routine checks.

- uses existing system tools to determine if a binary is running, no
need to pollute the fs or argue over perms or ownership within the fs
for specialized substitutes, dirtrees, nor for a pidfile that may or may
not be available / accurate.

- re-invokes the binary *AND/OR* any chosen script on detection of
failure, *including* failure of a specified, but non-existent binary.
This is both simple and extremely flexible, as the script can perform
tests, shutdown and restart a whole 'team' in sequence, send an e-mail
or SMS message, make custom log entries, any/all of the preceding -
whatever your imagination and scripting skill will support. Or just
kick the binary back to life.

- monitoring a non-existent binary effectively provides a very
fast-running chron-like tool, as it will *always* trigger at the
specified interval, and may be used to see if a flag has been set in
userland, or even on another box half way across the world.
With this, an unprivileged user or event may be allowed to cause some
action to be taken by root, but without otherwise impacting system security.

- the monitord process never hides itself from ps or top, can more
easily be suspended while maintenance is done, restarted later.


2) 'checkservice' - run on either/both the monitored host, and/or from a
'watcher' box elsewhere:

- at specified intervals, looks at a specified IP and port for the
availability of a specified service - standard or otherwise.

- may utilize a 'stock' or custom template of arbitrary complexity to
determine that a service is not only present, but that it also responds
as it should do (specific web-page content, presentation of the expected
smtp banner, EHLO advertising, etc.)

- may also make use of an arbitrary script to send messages, perform
tests, do sequenced control, branch to other scripts/binaries, etc.

Taken as a team, these are far more powerful and flexible than
Daemontools - even (from experience) w/r keeping an otherwise moody
Qmail daemon standing up on its wrong-legged feet.

YMMV, but IMNSHO as a (former) long-term user, Daemontools is too
limited for the underlying complexity and in-your-face intransigence it
adds.

Bill Hacker