Re: [exim-dev] Embedding Python

Startseite
Nachricht löschen
Nachricht beantworten
Autor: Todd Lyons
Datum:  
To: Jeremy Harris, exim-dev, Phil Pennock
Betreff: Re: [exim-dev] Embedding Python
On Thu, May 2, 2013 at 1:36 PM, Phil Pennock <pdp@???> wrote:
>
> Todd should just be able to use PyList_Insert() to modify sys.path,
> after importing sys. Note that this has to happen after the interpreter
> is loaded, since Py_GetPath() is used to provide the default value.


I've been experimenting with this. So far I cannot get path
manipulation to work using the list methods. I can only get it to
work using string methods.

Doing this works:

char *path, *newpath;
path = Py_GetPath();
newpath = (char *)string_sprintf("%s:%s", syspath,"/etc/exim");
PySys_SetPath(newpath);
/* Now check it */
path = Py_GetPath();

And using the list method (again, very simplified, no error checking
yet) does not:

char *path;
PyObject *syspath, *pName;
syspath = PySys_GetObject("path");
pName = PyString_FromString("/etc/exim");
PyList_Insert(syspath, 0, pName);
PySys_SetObject("path", syspath);
/* Now check it */
path = Py_GetPath();

When I print out the path string, the "/etc/exim" is not in it.

> The main things to figure out and document are:
>
>  1. Does site initialisation get loaded?  If not, does another fixed
>     module name get loaded instead, letting administrators symlink it to
>     pull in any site customisations of sys.path done therein?  This is
>     `site.py` inside the main python system modules directory, and it
>     affects whether or not distutils-provided packages can be used, etc
>     etc.


It will depend upon if Py_Initialize() does the site loading (which in
my googling seems to imply that it does). To answer a second part of
your question, my plan was to take the python_startup setting (for
example "/etc/exim/exim_python.py"), split the path and filename, then
load the module "exim_python". That's why I need to be able to add
the path to the python search paths. If there is a better way of
doing this, I'm all ears. It didn't seem difficult when reading all
of the api docs, but now it seems very complicated.

> 2. Is threading initialised? If not, threading calls will all succeed


No plans to initialize threading, and if I can actually block it, I will.

>     but the other threads will never actually be scheduled.  It's
>     GIL-based, and OS-level threads are unaffected, since effectively
>     it's just a single real thread, but given that Exim is traditionally
>     single-threaded, even just thinking about explaining and helping
>     debug this might cause issues.
>     I suspect we explicitly document that threading is not initialised,
>     because of Exim's forking model, and that folks who want to use
>     threaded python architect to have a separate long-lived daemon run
>     entirely outside of Exim and then perhaps use the Python-in-Exim
>     support to talk to that daemon.


Agreed.

>  3. Signal handler registration?  Yes or no?  Probably not.  Still needs
>     to be documented as a limitation.


Agreed.

>  4. What do we do about character sets, and what data is passed to the
>     Python code from Exim strings?  How does this change with Python 3
>     versus Python 2?  Py3 uses Unicode extensively and a lot of existing
>     interfaces pass around byte arrays instead.


I really only considered ASCII. If we get outside of that, I'll need
hand holding.

>  5. What happens with the lifetime and scope of globals?  Is there any
>     persistence within a process between calls, or is there a fresh
>     global dict at each invocation, snapshotted from any initialisation
>     atstart-style function?


Once I get a working system, I'll see how that actually plays out. I
had hoped it would be just like the perl code, where globals can be
defined and are maintained (through the life of that exim process).

>  6. What happens with garbage collection?  Do we GC at every return to
>     top-level Exim config?  Do we never GC?  This affects when return
>     values have to be copied to Exim's storage pools, or even if they
>     need to be, and whether subsequent calls to Python might invalidate
>     existing returned string pointers, if we don't
>     copy(-and-perhaps-GC).


Is GC more than just calling the appropriate Py_DECREF() when
initializing, importing code, and calling the functions?

> Those are the issues that come immediately to mind. There may be
> others.


Undoubtedly. Thanks Phil.

...Todd
--
The total budget at all receivers for solving senders' problems is $0.
If you want them to accept your mail and manage it the way you want,
send it the way the spec says to. --John Levine