[Twisted-Python] Problem with Deferreds

[Chris Norman]

Hi all,
I recently got over myself and forced myself to read about Deferreds
rather than using threading opperations.

I've used them successfully in a couple of places, but this one has me
flummoxed:

Here's the code for the Deferred and it's sub-commands:

def do_command(self, cmd, **kwargs):
"""Process a command in true Deferred style."""
if cmd.permissions(self.connection):
cmd(self.connection, **kwargs)
else:
logger.warning('Blocked from running command %s which is secured
with %s.', cmd.__name__, cmd.permissions.__name__)
raise CommandError('You have insufficient privileges to perform this
action. The staff have been notified.')

def handle_error(self, err):
"""Handle an error from do_command."""
if isinstance(err, CommandError):
return self.send_error(e.message, disconnect = e.disconnect)
else:
self.log(e, level = 'exception')
self.send_error('Sorry, but a problem with the server means your
command was not executed. The staff have been notified.')

def lineReceived(self, line):
"""Parse an incoming command."""
global lines
lines += 1 # Increment the line count.
data = line.decode(settings.ENCODING)
try:
command, kwargs = json.loads(data)
if not isinstance(command, string_types) or not isinstance(kwargs,
dict):
raise TypeError('Expecting [str, dict]. Got [%s, %s] instead.' %
(type(command), type(kwargs)))
except (TypeError, ValueError) as e:
self.log('Invalid command string: %s', data, level = 'error')
self.log(e, level = 'exception')
return self.send_error('Invalid command.', disconnect = True)
cmd = commands.commands.get(command, None)
if cmd is None:
self.log('Unrecognised command: %s.', command, level = 'warning')
elif self.connection.player or not cmd.login_required:
d = defer.Deferred()
print('Adding callback.')
d.addCallback(self.do_command, **kwargs)
print('Adding errback.')
d.addErrback(self.handle_error)
print('Calling callback.')
d.callback(cmd)
print('Called.') # Never gets this far.
return d
else:
return self.send_error('Not authenticated.', disconnect = True)

Here's the traceback I get when the callback gets called:

Unhandled Error
Traceback (most recent call last):
File "server/functions.py", line 88, in server_start
reactor.run()
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py",
line 1194, in run
self.mainLoop()
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py",
line 1206, in mainLoop
self.doIteration(t)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/epollreactor.py",
line 396, in doPoll
log.callWithLogger(selectable, _drdw, selectable, fd, event)
--- <exception caught here> ---
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/log.py",
line 101, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/log.py",
line 84, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/context.py",
line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/context.py",
line 81, in callWithContext
return func(*args,**kw)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/posixbase.py",
line 610, in _doReadOrWrite
self._disconnectSelectable(selectable, why, inRead)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/posixbase.py",
line 258, in _disconnectSelectable
selectable.connectionLost(failure.Failure(why))
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/failure.py",
line 232, in __init__
tb = self.value.__traceback__
builtins.AttributeError: 'Deferred' object has no attribute '__traceback__'

Anyone have any ideas?

Cheers,

Chris

[Kevin Conway]

Hi Chris,

tl;dr: Returning a value from 'dataReceived', or any of its extensions such
as 'lineReceived' in the 'LineReceiver' Protocol subclass, triggers a
disconnect and uses the returned value as the 'reason'. A 'reason' must be
an Exception or t.p.Failure object as other values will trigger this error.

Are you quite certain that your last line is not getting printed? I'm not
sure exactly where this feature is documented, but returning any non-None
value from a Protocol's 'dataReceived' method can result in this behaviour.
The t.protocols.basic.LineReceiver calls 'lineReceived' from 'dataReceived'
and returns any value it gets from your implementation. The value returned
from 'dataReceived' is passed along to the transport's 'doRead' which,
again, returns it to the portion of the reactor handling selectables. The
reactor assumes that anything returned from a transport during a read or
write operation is a bad thing and disconnects the transport. During the
disconnect process the reactor is generating a t.p.failure.Failure object
and passing in your returned value as the 'why' which is expected to be an
Exception or Failure and not a Deferred. Try returning None instead of your
Deferred. That should resolve this particular issue.

Post by Chris Norman
Hi all,
I recently got over myself and forced myself to read about Deferreds
rather than using threading opperations.
I've used them successfully in a couple of places, but this one has me
"""Process a command in true Deferred style."""
cmd(self.connection, **kwargs)
logger.warning('Blocked from running command %s which is secured
with %s.', cmd.__name__, cmd.permissions.__name__)
raise CommandError('You have insufficient privileges to perform this
action. The staff have been notified.')
"""Handle an error from do_command."""
return self.send_error(e.message, disconnect = e.disconnect)
self.log(e, level = 'exception')
self.send_error('Sorry, but a problem with the server means your
command was not executed. The staff have been notified.')
"""Parse an incoming command."""
global lines
lines += 1 # Increment the line count.
data = line.decode(settings.ENCODING)
command, kwargs = json.loads(data)
if not isinstance(command, string_types) or not isinstance(kwargs,
raise TypeError('Expecting [str, dict]. Got [%s, %s] instead.' %
(type(command), type(kwargs)))
self.log('Invalid command string: %s', data, level = 'error')
self.log(e, level = 'exception')
return self.send_error('Invalid command.', disconnect = True)
cmd = commands.commands.get(command, None)
self.log('Unrecognised command: %s.', command, level = 'warning')
d = defer.Deferred()
print('Adding callback.')
d.addCallback(self.do_command, **kwargs)
print('Adding errback.')
d.addErrback(self.handle_error)
print('Calling callback.')
d.callback(cmd)
print('Called.') # Never gets this far.
return d
return self.send_error('Not authenticated.', disconnect = True)
Unhandled Error
File "server/functions.py", line 88, in server_start
reactor.run()
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py",
line 1194, in run
self.mainLoop()
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py",
line 1206, in mainLoop
self.doIteration(t)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/epollreactor.py",
line 396, in doPoll
log.callWithLogger(selectable, _drdw, selectable, fd, event)
--- <exception caught here> ---
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/log.py",
line 101, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/log.py",
line 84, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/context.py",
line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/context.py",
line 81, in callWithContext
return func(*args,**kw)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/posixbase.py",
line 610, in _doReadOrWrite
self._disconnectSelectable(selectable, why, inRead)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/posixbase.py",
line 258, in _disconnectSelectable
selectable.connectionLost(failure.Failure(why))
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/failure.py",
line 232, in __init__
tb = self.value.__traceback__
builtins.AttributeError: 'Deferred' object has no attribute '__traceback__'
Anyone have any ideas?
Cheers,
Chris
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Chris Norman]

Hi,
Yes, that sorted the problem out no worries.

Thank you so much.

Post by Kevin Conway
Hi Chris,
tl;dr: Returning a value from 'dataReceived', or any of its extensions
such as 'lineReceived' in the 'LineReceiver' Protocol subclass,
triggers a disconnect and uses the returned value as the 'reason'. A
'reason' must be an Exception or t.p.Failure object as other values
will trigger this error.
Are you quite certain that your last line is not getting printed? I'm
not sure exactly where this feature is documented, but returning any
non-None value from a Protocol's 'dataReceived' method can result in
this behaviour. The t.protocols.basic.LineReceiver calls
'lineReceived' from 'dataReceived' and returns any value it gets from
your implementation. The value returned from 'dataReceived' is passed
along to the transport's 'doRead' which, again, returns it to the
portion of the reactor handling selectables. The reactor assumes that
anything returned from a transport during a read or write operation is
a bad thing and disconnects the transport. During the disconnect
process the reactor is generating a t.p.failure.Failure object and
passing in your returned value as the 'why' which is expected to be an
Exception or Failure and not a Deferred. Try returning None instead of
your Deferred. That should resolve this particular issue.
On Sat, Apr 2, 2016 at 4:39 AM Chris Norman
Hi all,
I recently got over myself and forced myself to read about Deferreds
rather than using threading opperations.
I've used them successfully in a couple of places, but this one has me
"""Process a command in true Deferred style."""
cmd(self.connection, **kwargs)
logger.warning('Blocked from running command %s which is secured
with %s.', cmd.__name__, cmd.permissions.__name__)
raise CommandError('You have insufficient privileges to perform this
action. The staff have been notified.')
"""Handle an error from do_command."""
return self.send_error(e.message, disconnect = e.disconnect)
self.log(e, level = 'exception')
self.send_error('Sorry, but a problem with the server means your
command was not executed. The staff have been notified.')
"""Parse an incoming command."""
global lines
lines += 1 # Increment the line count.
data = line.decode(settings.ENCODING)
command, kwargs = json.loads(data)
if not isinstance(command, string_types) or not isinstance(kwargs,
raise TypeError('Expecting [str, dict]. Got [%s, %s] instead.' %
(type(command), type(kwargs)))
self.log('Invalid command string: %s', data, level = 'error')
self.log(e, level = 'exception')
return self.send_error('Invalid command.', disconnect = True)
cmd = commands.commands.get(command, None)
self.log('Unrecognised command: %s.', command, level = 'warning')
d = defer.Deferred()
print('Adding callback.')
d.addCallback(self.do_command, **kwargs)
print('Adding errback.')
d.addErrback(self.handle_error)
print('Calling callback.')
d.callback(cmd)
print('Called.') # Never gets this far.
return d
return self.send_error('Not authenticated.', disconnect = True)
Unhandled Error
File "server/functions.py", line 88, in server_start
reactor.run()
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py",
line 1194, in run
self.mainLoop()
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py",
line 1206, in mainLoop
self.doIteration(t)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/epollreactor.py",
line 396, in doPoll
log.callWithLogger(selectable, _drdw, selectable, fd, event)
--- <exception caught here> ---
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/log.py",
line 101, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/log.py",
line 84, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/context.py",
line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/context.py",
line 81, in callWithContext
return func(*args,**kw)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/posixbase.py",
line 610, in _doReadOrWrite
self._disconnectSelectable(selectable, why, inRead)
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/posixbase.py",
line 258, in _disconnectSelectable
selectable.connectionLost(failure.Failure(why))
File
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/failure.py",
line 232, in __init__
tb = self.value.__traceback__
builtins.AttributeError: 'Deferred' object has no attribute '__traceback__'
Anyone have any ideas?
Cheers,
Chris
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python