Hi people,
============================
3 line summary of this post
============================
- I want exim lookups to be dynamically loadable modules
- I've *almost* got it working
- I just need a bit of help from someone who knows more about this
stuff than I do
============================
As some of you know, packaging of apps (RPM based) is one of my topics
of interest and indeed I have been sort of (or not, as the case may be)
maintaining some Exim RPMs on
ftp.exim.org.
Exim is a bit of a PITA to package, because there are so many different
options that you end up either building every possible option and
ending up in dependency hell where an Exim install pulls in hundreds of
dependencies, or you pick an arbitrary subset of options and leave some
people out in the cold. The main problem is with lookups, where you can
choose to compile in support for many different databases from DBM to
MySQL to LDAP.
One nice solution to this as used by many other apps is to use
dynamically loadable modules, where you have a "core" binary that
contains the core of the app, and most/all options that have external
dependencies are built separately as modules that can be dlopen()d at
runtime. These can then be packaged separately in RPMs (or debs, or
whatever), meaning you end up with stuff like:
exim
exim-mysql
exim-ldap
exim-pgsql
...
and you simply install "exim" plus whichever of the optional extensions
you want.
Now, various people over the years have discussed dlopen()
functionality for lookups but Philip doesn't really have any strong
interest in developing this functionality, not unreasonably. It's been
bugging me though, so I thought that rather than moan about it, I'd try
to do something about it. My C is not up to much so I first decided to
see if anyone had even had a bash at it. After some searching, I
eventually found a proof-of-concept patch posted by Steve Haslam on
exim-users back nearly 6 years ago, against Exim 3.12:
http://www.exim.org/pipermail/exim-users/Week-of-Mon-20000110/016177.html
I've now adapted Steve's patch to Exim 4 (4.52 as it happens) as a
proof-of-concept. It attempts to split out MySQL functionality in a
separate module. After some work, I've got it to the stage where Exim
builds, you get a separate mysql.so and this is dlopen()'d when Exim
starts. However, I have run into a bit of a brick wall and my complete
lack of knowledge or experience in this area means I'm not sure what to
do. Here's the deal:
[exim-4.52]$ ls build-Linux-i386/exim -l
-rwxrwxr-x 1 x x 619339 Oct 17 18:32 build-Linux-i386/exim
[exim-4.52]$ ls build-Linux-i386/lookups/*.so -l
-rwxrwxr-x 1 tjackson tjackson 9893 Oct 17 18:30 build-Linux-i386/
lookups/mysql.so
[exim-4.52]$ ./build-Linux-i386/exim -C src/configure.default
Error loading mysql.so: /path/to/exim-4.52/build-Linux-i386/lookups/
mysql.so: undefined symbol: debug_selector Exim is a Mail Transfer
Agent. It is normally called by Mail User Agents, not directly from a
shell command line. Options and/or arguments control what it does when
called. For a list of options, see the Exim documentation.
[exim-4.52]$ objdump --syms build-Linux-i386/lookups/mysql.so | grep
debug
00000000 *UND* 00000000 debug_selector
00000000 *UND* 00000000 debug_printf
Now to be honest, this as far as my knowledge goes. I don't really know
how to interpret the above. I am convinced that I am within a hair's
breadth of getting this working; indeed, although I may be completely
wrong, I suspect the problem is probably in the build chain rather than
in the actual code, which I know even less about.
So, what I was wondering is if there is anyone who knows more about C,
objects and dynamically loading stuff than I do (not difficult :) who
could give me a hand getting this POC working? I'm not asking for you
to take over this mini-project or anything, just to help me sort out
what's up so I can tidy up the patch and extend it to all modules.
Also, I know Philip's not very keen, but perhaps with the help of
someone experienced, we can even come up with a clean (optional) method
of doing this that is acceptable by all for integration into the main
Exim tree. (If not, I'll maintain it as a separate patch).
Any help whatsoever much appreciated.
Thanks,
Tim
diff -ur exim-4.52.orig/Makefile exim-4.52.dllookups/Makefile
--- exim-4.52.orig/Makefile 2005-07-01 12:09:15.000000000 +0100
+++ exim-4.52.dllookups/Makefile 2005-09-14 21:39:18.000000000 +0100
@@ -79,10 +79,10 @@
cd build-$(buildname); \
/bin/rm -f *.o lookups/*.o lookups/*.a auths/*.o auths/*.a \
routers/*.o routers/*.a transports/*.o transports/*.a \
- pcre/*.o pcre/*.a
+ pcre/*.o pcre/*.a lookups/*.so
clean_exim:; cd build-$(buildname); \
/bin/rm -f *.o lookups/*.o lookups/*.a auths/*.o auths/*.a \
- routers/*.o routers/*.a transports/*.o transports/*.a
+ routers/*.o routers/*.a transports/*.o transports/*.a lookups/*.so
# End of top-level makefile
diff -ur exim-4.52.orig/OS/Makefile-Base exim-4.52.dllookups/OS/Makefile-Base
--- exim-4.52.orig/OS/Makefile-Base 2005-07-01 12:09:15.000000000 +0100
+++ exim-4.52.dllookups/OS/Makefile-Base 2005-09-14 21:40:43.000000000 +0100
@@ -666,6 +666,7 @@
buildlookups:
@cd lookups; $(MAKE) SHELL=$(SHELL) AR="$(AR)" $(MFLAGS) CC="$(CC)" CFLAGS="$(CFLAGS)" \
FE="$(FE)" RANLIB="$(RANLIB)" HDRS="$(PHDRS)" \
+ LOOKUPDLS="$(LOOKUPDLS)" DLFLAGS="$(DLFLAGS)" \
INCLUDE="$(INCLUDE) $(IPV6_INCLUDE) $(TLS_INCLUDE) $(LOOKUP_INCLUDE)"; \
echo " "
diff -ur exim-4.52.orig/scripts/MakeLinks exim-4.52.dllookups/scripts/MakeLinks
--- exim-4.52.orig/scripts/MakeLinks 2005-07-01 12:09:15.000000000 +0100
+++ exim-4.52.dllookups/scripts/MakeLinks 2005-09-14 21:41:22.000000000 +0100
@@ -195,6 +195,7 @@
ln -s ../src/osfunctions.h osfunctions.h
ln -s ../src/store.h store.h
ln -s ../src/structs.h structs.h
+ln -s ../src/lookupapi.h lookupapi.h
ln -s ../src/acl.c acl.c
ln -s ../src/buildconfig.c buildconfig.c
diff -ur exim-4.52.orig/src/config.h.defaults exim-4.52.dllookups/src/config.h.defaults
--- exim-4.52.orig/src/config.h.defaults 2005-07-01 12:09:15.000000000 +0100
+++ exim-4.52.dllookups/src/config.h.defaults 2005-09-14 19:36:50.000000000 +0100
@@ -73,6 +73,7 @@
#define LOOKUP_LDAP
#define LOOKUP_LSEARCH
#define LOOKUP_MYSQL
+#define LOOKUP_MYSQL_DYNAMIC
#define LOOKUP_NIS
#define LOOKUP_NISPLUS
#define LOOKUP_ORACLE
@@ -117,6 +118,9 @@
#define SUPPORT_PAM
#define SUPPORT_TLS
#define SUPPORT_TRANSLATE_IP_ADDRESS
+#define SUPPORT_DLOOKUPS
+#define LOOKUP_MODULE_DIR
+
#define SYSLOG_LOG_PID
#define SYSLOG_LONG_LINES
diff -ur exim-4.52.orig/src/drtables.c exim-4.52.dllookups/src/drtables.c
--- exim-4.52.orig/src/drtables.c 2005-07-01 12:09:15.000000000 +0100
+++ exim-4.52.dllookups/src/drtables.c 2005-10-17 18:32:30.000000000 +0100
@@ -10,6 +10,7 @@
#include "exim.h"
+#include <dlfcn.h>
/* This module contains tables that define the lookup methods and drivers
that are actually included in the binary. Its contents are controlled by
@@ -112,7 +113,9 @@
searched by binary chop, having got rather large for the original linear
searching. */
-lookup_info lookup_list[] = {
+lookup_info *lookup_list;
+
+static lookup_info static_lookup_list[] = {
/* cdb lookup in single file */
@@ -304,23 +307,6 @@
#endif
},
-/* MYSQL lookup */
-
- {
- US"mysql", /* lookup name */
- lookup_querystyle, /* query-style lookup */
-#ifdef LOOKUP_MYSQL
- mysql_open, /* open function */
- NULL, /* no check function */
- mysql_find, /* find function */
- NULL, /* no close function */
- mysql_tidy, /* tidy function */
- mysql_quote /* quoting function */
-#else
- NULL, NULL, NULL, NULL, NULL, NULL /* lookup not present */
-#endif
- },
-
/* NIS lookup, excluding trailing 0 from key */
{
@@ -827,4 +813,133 @@
{ US"", NULL, NULL, NULL, 0, NULL, NULL, NULL, NULL, FALSE }
};
+typedef struct lookupmodulestr
+{
+ void *dl;
+ struct lookup_module_info *info;
+ struct lookupmodulestr *next;
+};
+
+static struct lookupmodulestr *lookupmodules;
+
+static void addlookupmodule(void *dl, struct lookup_module_info *info)
+{
+ struct lookupmodulestr *p = store_malloc(sizeof(struct lookupmodulestr));
+ p->dl = dl;
+ p->info = info;
+ if (lookupmodules)
+ lookupmodules->next = p;
+ lookupmodules = p;
+}
+
+void init_lookup_list(void)
+{
+ DIR *dd;
+ struct dirent *ent;
+ pcre *regex_islookupmod = regex_must_compile("\\.so$", FALSE, TRUE);
+ int countmodules = 0;
+ int moduleerrors = 0;
+
+ /* Our failsafe is to stick to the lookups that were compiled
+ in statically */
+ lookup_list = static_lookup_list;
+
+#if defined(LOOKUP_MYSQL) && !defined(LOOKUP_MYSQL_DYNAMIC)
+ addlookupmodule(NULL, &mysql_lookup_module_info);
+#endif
+
+ dd = opendir(LOOKUP_MODULE_DIR);
+ if (dd == NULL) {
+ DEBUG(5) debug_printf("Couldn't open %s: not loading lookup modules\n", LOOKUP_MODULE_DIR);
+ }
+ else {
+ DEBUG(9) debug_printf("Loading lookup modules from %s\n", LOOKUP_MODULE_DIR);
+ while ((ent = readdir(dd)) != NULL) {
+ char *name = ent->d_name;
+ int len = (int)strlen(name);
+ if (pcre_exec(regex_islookupmod, NULL, name, len, 0, PCRE_EOPT, NULL, 0) >= 0) {
+ int pathnamelen = len + (int)strlen(LOOKUP_MODULE_DIR) + 2;
+ void *dl;
+ struct lookup_module_info *info;
+ char *errormsg;
+
+ /* SRH: am I being paranoid here or what? */
+ if (pathnamelen > big_buffer_size) {
+ fprintf(stderr, "%s/%s: name too long\n", LOOKUP_MODULE_DIR, name);
+ continue;
+ }
+
+ /* SRH: snprintf here? */
+ sprintf(big_buffer, "%s/%s", LOOKUP_MODULE_DIR, name);
+
+ dl = dlopen(big_buffer, RTLD_NOW);// TJ was LAZY
+ if (dl == NULL) {
+ fprintf(stderr, "Error loading %s: %s\n", name, dlerror());
+ moduleerrors++;
+ continue;
+ }
+
+ info = (struct lookup_module_info*) dlsym(dl, "_lookup_module_info");
+ if ((errormsg = dlerror()) != NULL) {
+ fprintf(stderr, "%s does not appear to be a lookup module (%s)\n", name, errormsg);
+ dlclose(dl);
+ moduleerrors++;
+ continue;
+ }
+ if (info->magic != LOOKUP_MODULE_INFO_MAGIC) {
+ fprintf(stderr, "Lookup module %s is not compatible with this version of Exim\n", name);
+ dlclose(dl);
+ moduleerrors++;
+ continue;
+ }
+
+ addlookupmodule(dl, info);
+ DEBUG(9) debug_printf("Loaded \"%s\" (%d lookup types)\n", name, info->lookupcount);
+ countmodules++;
+ }
+ }
+ closedir(dd);
+ }
+
+ DEBUG(9) debug_printf("Loaded %d lookup modules\n", countmodules);
+ if (lookupmodules) {
+ struct lookupmodulestr *p;
+ int staticlookups = sizeof(static_lookup_list)/sizeof(static_lookup_list[0]) - 1;
+ int i;
+ int dynamiclookups = 0;
+
+ for (p = lookupmodules; p; p = p->next) {
+ dynamiclookups += p->info->lookupcount;
+ }
+
+ DEBUG(9) debug_printf("Appending %d dynamic lookups to %d static lookups\n", dynamiclookups, staticlookups);
+
+ lookup_list = store_malloc(sizeof(struct lookup_info) * dynamiclookups + sizeof(static_lookup_list));
+
+ /* static lookups get low numbers */
+ memcpy(lookup_list, static_lookup_list, sizeof(static_lookup_list));
+ i = staticlookups;
+
+ /* add dynamic lookups */
+ for (p = lookupmodules; p; p=p->next) {
+ int j;
+ for (j = 0; j < p->info->lookupcount; j++) {
+ lookup_list[i++] = *p->info->lookups[j];
+ }
+ }
+
+ /* copy terminator */
+ memcpy(lookup_list + i, static_lookup_list + staticlookups, sizeof(struct lookup_info));
+
+ DEBUG(4) {
+ lookup_info *li;
+ debug_printf("Lookups:", i);
+ for (li = lookup_list, i = 0; li->name[0] != 0; i++, li++) {
+ debug_printf(" %s=%d", li->name, i);
+ }
+ debug_printf("\n");
+ }
+ }
+}
+
/* End of drtables.c */
diff -ur exim-4.52.orig/src/EDITME exim-4.52.dllookups/src/EDITME
--- exim-4.52.orig/src/EDITME 2005-07-01 12:09:15.000000000 +0100
+++ exim-4.52.dllookups/src/EDITME 2005-09-14 21:57:45.000000000 +0100
@@ -301,6 +301,8 @@
# specified in INCLUDE. The settings below are just examples; -lpq is for
# PostgreSQL, -lgds is for Interbase.
+LOOKUP_INCLUDE=-I /usr/include/mysql
+LOOKUP_LIBS=-I /usr/lib/mysql
# LOOKUP_INCLUDE=-I /usr/local/ldap/include -I /usr/local/mysql/include -I /usr/local/pgsql/include
# LOOKUP_LIBS=-L/usr/local/lib -lldap -llber -lmysqlclient -lpq -lgds
diff -ur exim-4.52.orig/src/exim.c exim-4.52.dllookups/src/exim.c
--- exim-4.52.orig/src/exim.c 2005-07-01 12:09:15.000000000 +0100
+++ exim-4.52.dllookups/src/exim.c 2005-09-14 20:25:25.000000000 +0100
@@ -3248,6 +3248,9 @@
}
#endif /* EXIM_PERL */
+/* Initialise lookup_list */
+init_lookup_list();
+
/* Log the arguments of the call if the configuration file said so. This is
a debugging feature for finding out what arguments certain MUAs actually use.
Don't attempt it if logging is disabled, or if listing variables or if
diff -ur exim-4.52.orig/src/globals.h exim-4.52.dllookups/src/globals.h
--- exim-4.52.orig/src/globals.h 2005-07-01 12:09:15.000000000 +0100
+++ exim-4.52.dllookups/src/globals.h 2005-10-17 18:14:07.000000000 +0100
@@ -398,7 +398,7 @@
extern BOOL log_testing_mode; /* TRUE in various testing modes */
extern BOOL log_timezone; /* TRUE to include the timezone in log lines */
extern uschar *login_sender_address; /* The actual sender address */
-extern lookup_info lookup_list[]; /* Vector of available lookups */
+extern lookup_info *lookup_list; /* Array of available lookups */
extern int lookup_list_count; /* Number of entries in the list */
extern int lookup_open_max; /* Max lookup files to cache */
extern uschar *lookup_value; /* Value looked up from file */
Only in exim-4.52.dllookups/src: lookupapi.h
diff -ur exim-4.52.orig/src/lookups/Makefile exim-4.52.dllookups/src/lookups/Makefile
--- exim-4.52.orig/src/lookups/Makefile 2005-07-01 12:09:15.000000000 +0100
+++ exim-4.52.dllookups/src/lookups/Makefile 2005-10-17 18:29:35.000000000 +0100
@@ -8,6 +8,8 @@
OBJ = cdb.o dbmdb.o dnsdb.o dsearch.o ibase.o ldap.o lsearch.o mysql.o nis.o \
nisplus.o oracle.o passwd.o pgsql.o spf.o testdb.o whoson.o lf_check_file.o \
lf_quote.o
+
+all: lookups.a $(LOOKUPDLS)
lookups.a: $(OBJ)
@/bin/rm -f lookups.a
@@ -16,10 +18,13 @@
$(RANLIB) $@
@/bin/rm -rf ../drtables.o
-.SUFFIXES: .o .c
+.SUFFIXES: .o .c .so
.c.o:; @echo "$(CC) $*.c"
$(FE)$(CC) -c $(CFLAGS) $(INCLUDE) $*.c
+mysql.so: $(HDRS) mysql.c mysql.h
+ $(CC) -DDYNLOOKUP -rdynamic $(CFLAGS) $(INCLUDE) -L /usr/lib/mysql $(DLFLAGS) $*.c -lmysqlclient -o $@
+
lf_check_file.o: $(HDRS) lf_check_file.c lf_functions.h
lf_quote.o: $(HDRS) lf_quote.c lf_functions.h
diff -ur exim-4.52.orig/src/lookups/mysql.c exim-4.52.dllookups/src/lookups/mysql.c
--- exim-4.52.orig/src/lookups/mysql.c 2005-07-01 12:09:15.000000000 +0100
+++ exim-4.52.dllookups/src/lookups/mysql.c 2005-09-16 10:20:25.000000000 +0100
@@ -25,10 +25,33 @@
in a dummy argument to stop even pickier compilers complaining about infinite
loops. */
-#ifndef LOOKUP_MYSQL
+#if !defined(LOOKUP_MYSQL) || (defined(LOOKUP_MYSQL_DYNAMIC) && !defined(SUPPORT_DLOOKUPS))
static void dummy(int x) { dummy(x-1); }
#else
+/* These are the lookup_info blocks for this driver */
+
+lookup_info mysql_lookup_info = {
+ US"mysql", /* lookup name */
+ lookup_querystyle, /* query-style lookup */
+#ifdef LOOKUP_MYSQL
+ mysql_open, /* open function */
+ NULL, /* no check function */
+ mysql_find, /* find function */
+ NULL, /* no close function */
+ mysql_tidy, /* tidy function */
+ mysql_quote /* quoting function */
+#else
+ NULL, NULL, NULL, NULL, NULL, NULL /* lookup not present */
+#endif
+};
+
+#ifdef SUPPORT_DLOOKUPS
+#define mysql_lookup_module_info _lookup_module_info
+#endif
+
+static lookup_info *_lookup_list[] = { &mysql_lookup_info };
+lookup_module_info mysql_lookup_module_info = { LOOKUP_MODULE_INFO_MAGIC, _lookup_list, 1 };
#include <mysql.h> /* The system header */
diff -ur exim-4.52.orig/src/lookups/mysql.h exim-4.52.dllookups/src/lookups/mysql.h
--- exim-4.52.orig/src/lookups/mysql.h 2005-07-01 12:09:15.000000000 +0100
+++ exim-4.52.dllookups/src/lookups/mysql.h 2005-09-16 21:45:34.000000000 +0100
@@ -9,6 +9,8 @@
/* Header for the mysql lookup functions */
+extern lookup_module_info mysql_lookup_module_info;
+
extern void *mysql_open(uschar *, uschar **);
extern int mysql_find(void *, uschar *, uschar *, int, uschar **, uschar **,
BOOL *);
diff -ur exim-4.52.orig/src/structs.h exim-4.52.dllookups/src/structs.h
--- exim-4.52.orig/src/structs.h 2005-07-01 12:09:15.000000000 +0100
+++ exim-4.52.dllookups/src/structs.h 2005-09-14 21:42:52.000000000 +0100
@@ -320,34 +320,7 @@
/* Structure for holding information about a lookup type. */
-typedef struct lookup_info {
- uschar *name; /* e.g. "lsearch" */
- int type; /* query/singlekey/abs-file */
- void *(*open)( /* open function */
- uschar *, /* file name for those that have one */
- uschar **); /* for error message */
- BOOL (*check)( /* file checking function */
- void *, /* handle */
- uschar *, /* file name */
- int, /* modemask for file checking */
- uid_t *, /* owners for file checking */
- gid_t *, /* owngroups for file checking */
- uschar **); /* for error messages */
- int (*find)( /* find function */
- void *, /* handle */
- uschar *, /* file name or NULL */
- uschar *, /* key or query */
- int, /* length of key or query */
- uschar **, /* for returning answer */
- uschar **, /* for error message */
- BOOL *); /* to request cache cleanup */
- void (*close)( /* close function */
- void *); /* handle */
- void (*tidy)(void); /* tidy function */
- uschar *(*quote)( /* quoting function */
- uschar *, /* string to quote */
- uschar *); /* additional data from quote name */
-} lookup_info;
+#include "lookupapi.h"
/* Structure for holding information about the configured authentication
