[Twisted-Python] Implementing Postfix Inet Policy Check Client

[Tom Boland]

Greetings all.

This may be a very basic question. I'm hoping to implement a postfix
policy check client in twisted. It's a simple protocol. You send
newline separated key value pairs like:

recipient=***@ddr.ess
sender=***@ddr.ess
size=1024
helo_name=mail.server
etc..

you terminate the request with an additional newline. The response
comes back like

action=OK

You can send mutliple requests in the same connection. What I'm
envisaging is a module that can be used to provide a deferred
request/response pairing to my calling application. The module class
will manage the single connection to the postfix policy daemon (I'm
actually going to have persistent connections to a few daemons), and
reconnect when necessary etc. Any requests will return a deferred that
I can add callbacks to. How would you design this with twisted? I can
easily envisage a way of using a clientfactory to instantiate separate
connections for each request/response, but actually being able to simply
send a request and receive the single response for that request is
something I'm struggling to do within a LineReceiver instance (for
instance). Would the twisted.protocols.amp module help given that I
can't change the server-side protocol?

Any advice much appreciated!

Thanks. Tom.

[Ex Vito]

Tom,

I guess LineReceiver can help you with that, it's a matter of using a
different delimiter. Here is a rough sketch of something you can start
exploring:

from __future__ import print_function

from twisted.internet import reactor, protocol, endpoints, defer
from twisted.protocols import basic


class PostfixProtocol(basic.LineReceiver):

# Assuming Postfix sends '\r\n' line breaks (does it?)
delimiter = '\r\n\r\n'

def lineReceived(self, multi_line):
input_dict = {
k: v for k, v in (
line.split('=') for line in multi_line.split('\r\n')
)
}
self.gotPostfixRequest(input_dict)

def gotPostfixRequest(self, request_dict):
# Silly deferred-based implementation.
d = defer.Deferred()
d.addCallback(self.sendPostfixAction)
# Simulate a deferred being fired with success.
reactor.callLater(1, d.callback, 'OK')

def sendPostfixAction(self, response):
# NOTE: Sends self.delimiter after the payload.
# Use self.tranport.write if you don't want it.
self.sendLine('action={}'.format(response))


if __name__ == '__main__':

ep = endpoints.serverFromString(reactor, 'tcp:10000')
f = protocol.Factory()
f.protocol = PostfixProtocol
ep.listen(f)
reactor.run()

Key ideas:
- PostfixProtocol overrides LineReceiver's delimiter (setting it to
'\r\n\r\n')
- lineReceived parses the muti_line and calls gotPostfixRequest.
- gotPostfixRequest should decide (or delegate, you choice) what kind of
response to send back.
- sendPostfixAction sends a response back to Postfix.

Notes:
- The multi_line parsing code is short(ish) but not very robust. It may
fail in bad ways with invalid input.
- The gotPostfixRequest implementation is written for the sake of a
deferred-based example.

Does this help you in any way?
Cheers,
--
exvito

Post by Tom Boland
Greetings all.
This may be a very basic question. I'm hoping to implement a postfix
policy check client in twisted. It's a simple protocol. You send
size=1024
helo_name=mail.server
etc..
you terminate the request with an additional newline. The response
comes back like
action=OK
You can send mutliple requests in the same connection. What I'm
envisaging is a module that can be used to provide a deferred
request/response pairing to my calling application. The module class
will manage the single connection to the postfix policy daemon (I'm
actually going to have persistent connections to a few daemons), and
reconnect when necessary etc. Any requests will return a deferred that
I can add callbacks to. How would you design this with twisted? I can
easily envisage a way of using a clientfactory to instantiate separate
connections for each request/response, but actually being able to simply
send a request and receive the single response for that request is
something I'm struggling to do within a LineReceiver instance (for
instance). Would the twisted.protocols.amp module help given that I
can't change the server-side protocol?
Any advice much appreciated!
Thanks. Tom.
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Tom Boland]

Hi Exvito,

Thanks very much for this. I think I might just give a bit of context
about using this.

I have an SMTP proxy using Nginx. It uses queries an HTTP service to
establish whether mail should be forwarded, and where to (on to
postfix/dovecot/exchange backends). It uses Twisted for the HTTP
authorisation service, and I've been using a similar model for years for
POP and IMAP in production, and for the last year on an outbound SMTP
relay. The Twisted http service handles password authentication as well
lookups that point connections on to the correct backend servers. All
of our custom business logic is in the python (database lookups and the
like). Twisted has performed brilliantly as the HTTP service for years.

I'm now writing an inbound SMTP proxy service, and initially, it just
does DNSBL checks, which will actually stop around 12 million emails a
day from reaching postfix and tying up smtpd processes! I fire off a
number of DNSBL checks in a deferred chain, and just come back to it
once the chain has completed. The postfix policy check was going to be
added to this chain of events and initially I was going to use it to
query the dovecot quota daemon running on the appropriate backend for a
given mailbox, therefore I'd have a few tens of connections open, one to
each of the policy daemons on the dovecot backend servers. This is just
to avoid queuing and trying to deliver mail to a mailbox that's over-quota.

I think what you've provided me with is useful for me, but I think it's
backwards for my purposes, as I need to be connecting to the policy
daemon rather than being the policy daemon!

I wanted to do this with deferred calls in case one of the policy
daemons becomes unreachable and blocks my application. Do you think I
should do something differently in that regard? My SQL lookups are done
synchronously. If the database server goes away, I've got bigger
problems anyway!

Many thanks for your help. I'll work a bit on this this morning and
come back!

Thanks again. Tom.

Post by Ex Vito
Tom,
I guess LineReceiver can help you with that, it's a matter of using a
different delimiter. Here is a rough sketch of something you can start
from __future__ import print_function
from twisted.internet import reactor, protocol, endpoints, defer
from twisted.protocols import basic
# Assuming Postfix sends '\r\n' line breaks (does it?)
delimiter = '\r\n\r\n'
input_dict = {
k: v for k, v in (
line.split('=') for line in multi_line.split('\r\n')
)
}
self.gotPostfixRequest(input_dict)
# Silly deferred-based implementation.
d = defer.Deferred()
d.addCallback(self.sendPostfixAction)
# Simulate a deferred being fired with success.
reactor.callLater(1, d.callback, 'OK')
# NOTE: Sends self.delimiter after the payload.
# Use self.tranport.write if you don't want it.
self.sendLine('action={}'.format(response))
ep = endpoints.serverFromString(reactor, 'tcp:10000')
f = protocol.Factory()
f.protocol = PostfixProtocol
ep.listen(f)
reactor.run()
- PostfixProtocol overrides LineReceiver's delimiter (setting it to
'\r\n\r\n')
- lineReceived parses the muti_line and calls gotPostfixRequest.
- gotPostfixRequest should decide (or delegate, you choice) what kind
of response to send back.
- sendPostfixAction sends a response back to Postfix.
- The multi_line parsing code is short(ish) but not very robust. It
may fail in bad ways with invalid input.
- The gotPostfixRequest implementation is written for the sake of a
deferred-based example.
Does this help you in any way?
Cheers,
--
exvito

Post by Tom Boland
Greetings all.
This may be a very basic question. I'm hoping to implement a postfix
policy check client in twisted. It's a simple protocol. You send
size=1024
helo_name=mail.server
etc..
you terminate the request with an additional newline. The response
comes back like
action=OK
You can send mutliple requests in the same connection. What I'm
envisaging is a module that can be used to provide a deferred
request/response pairing to my calling application. The module class
will manage the single connection to the postfix policy daemon (I'm
actually going to have persistent connections to a few daemons), and
reconnect when necessary etc. Any requests will return a deferred that
I can add callbacks to. How would you design this with twisted? I can
easily envisage a way of using a clientfactory to instantiate separate
connections for each request/response, but actually being able to simply
send a request and receive the single response for that request is
something I'm struggling to do within a LineReceiver instance (for
instance). Would the twisted.protocols.amp module help given that I
can't change the server-side protocol?
Any advice much appreciated!
Thanks. Tom.
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

--
--
exvito
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[exvito here]

Post by Tom Boland
I think what you've provided me with is useful for me, but I think it's
backwards for my purposes, as I need to be connecting to the policy daemon
rather than being the policy daemon!
I wanted to do this with deferred calls in case one of the policy daemons
becomes unreachable and blocks my application. Do you think I should do
something differently in that regard? My SQL lookups are done
synchronously. If the database server goes away, I've got bigger problems
anyway!

So maybe something like this is more likely to be useful:

#!/usr/bin/env python

from __future__ import print_function

from twisted.internet import reactor, protocol, endpoints, defer
from twisted.protocols import basic


class PostfixProtocol(basic.LineReceiver):

# Assuming Postfix uses '\r\n' line breaks (does it?)
delimiter = '\r\n'

def __init__(self):
self.action = None
self.action_deferred = None

def lineReceived(self, line):
if '=' in line:
self.action = line.split('=')[1]
elif line == '':
self.action_deferred.callback(self.action)
self.action_deferred = None
else:
# oops, bad input
pass

def sendPostfixRequest(self, request_dict):
if not self.action_deferred is None:
raise Exception('transaction pending')
for k, v in request_dict.items():
self.sendLine('{}={}'.format(k,v))
# Empty line signals we're done
self.sendLine('')
self.action_deferred = defer.Deferred()
return self.action_deferred

@defer.inlineCallbacks
def checkPostfixPolicy(request_dict):
ep = endpoints.clientFromString(reactor, 'tcp:host=127.0.0.1:
port=10000')
p = yield endpoints.connectProtocol(ep, PostfixProtocol())
action = yield p.sendPostfixRequest(request_dict)
print('got: {}'.format(action))
reactor.stop()


if __name__ == '__main__':

request_dict = {
'recipient': '***@ddr.ess',
'sender': '***@ddr.ess',
}
reactor.callWhenRunning(checkPostfixPolicy, request_dict)
reactor.run()

Highlights:
- This is not the same protocol as before, in particular it uses a
different delimiter.
- It assumes the response is also terminated with an empty line (does it?).
- It more than one outstanding response: a different exception should be
used.
- The input processing is very rudimentary and failure-prone.
- checkPostfixPolicy could, of course, return instead of printing. :)

Cheers,
--
exvito

[Tom Boland]

Thanks again for this. It's really useful. It turns out that the
delimiter is a plain old \n. Who knows how consistent this will be
between different policy daemons, I don't know!

I've modified it to manage a DeferredQueue which hopefully means I can
just keep throwing requests at it without every being bounced away. A
stripped down example (with even less error checking!) of what I've
managed to work your example in to is here:

class PostfixPolicyClient(LineReceiver):

delimiter = '\n'

def __init__(self):
self.resultQueue = DeferredQueue()


def lineReceived(self, line):
if '=' in line:
self.resultQueue.put(True if line.split('=')[1] == 'OK' else
False)


def sendPostfixPolicyClientRequest(self, request_dict):
for k, v in request_dict.items():
self.sendLine('{}={}'.format(k, v))
self.sendLine('')
return self.resultQueue.get()



Now, this isn't a working example, it's just the minimum that will
demonstrate my idea. I just wonder if what I've done with the
DeferredQueue is sane. If I return the .get() entry from the
DeferredQueue when doing the request, and then do a put() in
lineReceived, am I guaranteeing that I will get my results in the
correct order?

Thanks again for all your help!

Many thanks. Tom.

Post by Tom Boland
I think what you've provided me with is useful for me, but I think
it's backwards for my purposes, as I need to be connecting to the
policy daemon rather than being the policy daemon!
I wanted to do this with deferred calls in case one of the policy
daemons becomes unreachable and blocks my application. Do you
think I should do something differently in that regard? My SQL
lookups are done synchronously. If the database server goes away,
I've got bigger problems anyway!
#!/usr/bin/env python
from __future__ import print_function
from twisted.internet import reactor, protocol, endpoints, defer
from twisted.protocols import basic
# Assuming Postfix uses '\r\n' line breaks (does it?)
delimiter = '\r\n'
self.action = None
self.action_deferred = None
self.action = line.split('=')[1]
self.action_deferred.callback(self.action)
self.action_deferred = None
# oops, bad input
pass
raise Exception('transaction pending')
self.sendLine('{}={}'.format(k,v))
# Empty line signals we're done
self.sendLine('')
self.action_deferred = defer.Deferred()
return self.action_deferred
@defer.inlineCallbacks
ep = endpoints.clientFromString(reactor,
'tcp:host=127.0.0.1:port=10000')
p = yield endpoints.connectProtocol(ep, PostfixProtocol())
action = yield p.sendPostfixRequest(request_dict)
print('got: {}'.format(action))
reactor.stop()
request_dict = {
}
reactor.callWhenRunning(checkPostfixPolicy, request_dict)
reactor.run()
- This is not the same protocol as before, in particular it uses a
different delimiter.
- It assumes the response is also terminated with an empty line (does it?).
- It more than one outstanding response: a different exception should
be used.
- The input processing is very rudimentary and failure-prone.
- checkPostfixPolicy could, of course, return instead of printing. :)
Cheers,
--
exvito
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[exvito here]

Post by Tom Boland
Thanks again for this. It's really useful. It turns out that the
delimiter is a plain old \n. Who knows how consistent this will be between
different policy daemons, I don't know!

I would check the Postfix docs for that.

Post by Tom Boland
Now, this isn't a working example, it's just the minimum that will
demonstrate my idea. I just wonder if what I've done with the
DeferredQueue is sane. If I return the .get() entry from the DeferredQueue
when doing the request, and then do a put() in lineReceived, am I
guaranteeing that I will get my results in the correct order?

The DeferredQueue is a nice approach: it ensures the get() results come out
in the same order as the put() calls.

The key question is whether or not the server handles multiple outstanding
requests within the same connection. Given your informal protocol
description, if the server supports it, it seems the responses must come
back in the same order as the requests were sent, otherwise there is
apparently no way to relate them; that being the case, what's the advantage
of pushing more than one request at a time if one slow request/response
"transaction" will delay a subsequent fast request/response "transaction"?

A variation of this, assuming the server only handles one outstanding
request at a time per connection, could be a protocol implementation that
would queue requests ensuring only one was sent at a time: this might
provide a cleaner client side API.

PS: My references to "Postfix docs" might need to be replaced by "your
policy server docs" which, hopefully, will match Postfix's... (you would
know that) :)

Cheers,
--
exvito