[Twisted-Python] Returning a Deferred as a result from another Deferred

[Ryan Fugger]

Returning a Deferred as a result from another Deferred seems to be
disallowed by an assertion at the start of the callback chain. Why is
this? Right now I have worked around this limitation by wrapping my
Deferred in a list and then unwrapping it in the callback.

Maybe someone can suggest a better approach. My application takes a
command from one connection and defers processing to a worker thread
to permit potentially-blocking database access. That worker thread
often will need to send data over another connection to fulfill the
command, and await a reply before the reply can be sent back over the
original command connection. The worker thread sends data over the
second connection using callFromThread, and this send operation itself
returns a Deferred which fires when the reply is received. It is this
Deferred that I want to return as a result from the original Deferred
created by deferring to the worker thread in the command handler, so I
can wait until the reply comes from the second connection, and then
reply on the original command connection. Can someone suggest a
better approach?

Thanks.

Ryan

[Jonathan Lange]

Post by Ryan Fugger
Returning a Deferred as a result from another Deferred seems to be
disallowed by an assertion at the start of the callback chain. Why is
this? Right now I have worked around this limitation by wrapping my
Deferred in a list and then unwrapping it in the callback.

No it's not. I can guarantee this.

What's the content of the AssertionError that you are getting?

jml

[Ryan Fugger]

Post by Jonathan Lange

Post by Ryan Fugger
Returning a Deferred as a result from another Deferred seems to be
disallowed by an assertion at the start of the callback chain. Why is
this? Right now I have worked around this limitation by wrapping my
Deferred in a list and then unwrapping it in the callback.

No it's not. I can guarantee this.
What's the content of the AssertionError that you are getting?

No content in the error. Line 238 of twisted/internet/defer.py (first
line in Deferred.callback) is:

assert not isinstance(result, Deferred)

That's pretty explicit in disallowing Deferreds... I'm working with
the version 2.5 release.

Ryan

[David Reid]

Hi Ryan,

Post by Ryan Fugger
Returning a Deferred as a result from another Deferred seems to be
disallowed by an assertion at the start of the callback chain. Why is
this? Right now I have worked around this limitation by wrapping my
Deferred in a list and then unwrapping it in the callback.

The following will cause an AssertionError

d = Deferred()
d2 = Deferred()

d2.callback(d)

There isn't really a good reason to do this anyway. Since there is
likely no reason to not just return d with your callbacks attached.
If you do need two deferreds here for some reason, you _can_ return a
deferred from a callback. In which case further callbacks will not
be called until the second deferred has fired.

-David

[Ryan Fugger]

Post by David Reid
Hi Ryan,

Post by Ryan Fugger
Returning a Deferred as a result from another Deferred seems to be
disallowed by an assertion at the start of the callback chain. Why is
this? Right now I have worked around this limitation by wrapping my
Deferred in a list and then unwrapping it in the callback.

The following will cause an AssertionError
d = Deferred()
d2 = Deferred()
d2.callback(d)
There isn't really a good reason to do this anyway. Since there is
likely no reason to not just return d with your callbacks attached.
If you do need two deferreds here for some reason, you _can_ return a
deferred from a callback. In which case further callbacks will not
be called until the second deferred has fired.

What if I need to return a Deferred from a deferToThread?

Ryan

[David Reid]

Post by Ryan Fugger
What if I need to return a Deferred from a deferToThread?

That isn't really a "What if I" question, that's a "Why do you think
you" question.

deferToThread is best used for taking uncontrollably blocking APIs
(preferably C APIs that release GIL) and getting a Deferred back. I
personally can not think of a single reason why you would ever take a
Deferred returning API and shuffle it off to a thread.

-David

[Ryan Fugger]

Post by David Reid

Post by Ryan Fugger
What if I need to return a Deferred from a deferToThread?

That isn't really a "What if I" question, that's a "Why do you think
you" question.
deferToThread is best used for taking uncontrollably blocking APIs
(preferably C APIs that release GIL) and getting a Deferred back. I
personally can not think of a single reason why you would ever take a
Deferred returning API and shuffle it off to a thread.

See my original email for the specifics. I receive a command on a
connection, deferToThread for handling, which involves database
access, and then callFromThread to send a message on another
connection, which returns a Deferred that fires when the reply is
received, triggering a reply to the command on the original
connection.

Ryan

[g***@divmod.com]

Post by Ryan Fugger

Post by David Reid
deferToThread is best used for taking uncontrollably blocking APIs
(preferably C APIs that release GIL) and getting a Deferred back. I
personally can not think of a single reason why you would ever take a
Deferred returning API and shuffle it off to a thread.

See my original email for the specifics. I receive a command on a
connection, deferToThread for handling, which involves database
access, and then callFromThread to send a message on another
connection, which returns a Deferred that fires when the reply is
received, triggering a reply to the command on the original
connection.

I assume the briefest Python summary of your code looks like this:

def thingInThread():
databaseResult = databaseWork()
deferred = reactor.callFromThread(otherConnection.sendRequest,
databaseResult)
return deferred

def thingInReactor():
return threads.deferToThread(thingInThread)

This isn't really idiomatic Twisted code, because while it is
technically safe (thingInThread doesn't *do* anything with its Deferred)
it is slightly misleading. Deferreds are not thread safe, and there's
nothing you *could* do with that Deferred in thingInThread; if you
wanted to add a callback to it, for example, you can't.

What you actually want is something more like this:

def thingInReactor():
deferred = threads.deferToThread(databaseWork)
def tellOtherConnection(databaseResult):
return otherConnection.sendRequest(databaseResult)
deferred.addCallback(tellOtherConnection)
return deferred

This deferred can safely have callbacks added to it and generally be
treated normally, since it never sees a thread. Also, now you don't
have the problem where .callback() is being called with Deferred as an
argument; instead, you are returning one Deferred to another, which
results in the outer Deferred ("deferred") receiving the inner
Deferred's result (whatever "otherConnection.sendRequest" would have
fired with).

[Jean-Paul Calderone]

Post by Ryan Fugger

Post by Jonathan Lange

Post by Ryan Fugger
Returning a Deferred as a result from another Deferred seems to be
disallowed by an assertion at the start of the callback chain. Why is
this? Right now I have worked around this limitation by wrapping my
Deferred in a list and then unwrapping it in the callback.

No it's not. I can guarantee this.
What's the content of the AssertionError that you are getting?

No content in the error. Line 238 of twisted/internet/defer.py (first
assert not isinstance(result, Deferred)
That's pretty explicit in disallowing Deferreds... I'm working with
the version 2.5 release.

You've misread the code. Jonathan is correct. The rest of this thread
seems to go off on some tangent about threads which may or may not be
relevant to whatever problem you're trying to solve. What led you to
be reading this code in the first place?

Glyph did make a good point, though. You shouldn't ever have a Deferred
in a thread other than the reactor thread. You shouldn't make them, you
shouldn't add callbacks to them, you shouldn't call them back. It might
be legitimate to get one from the reactor thread, hold it for a while,
then send it back to the reactor thread, but I can't think of a case
where this would really be very useful.

Jean-Paul

Post by Ryan Fugger
Ryan
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Ryan Fugger]

Post by Jean-Paul Calderone

Post by Ryan Fugger

Post by Jonathan Lange

Post by Ryan Fugger
Returning a Deferred as a result from another Deferred seems to be
disallowed by an assertion at the start of the callback chain. Why is
this? Right now I have worked around this limitation by wrapping my
Deferred in a list and then unwrapping it in the callback.

No it's not. I can guarantee this.
What's the content of the AssertionError that you are getting?

No content in the error. Line 238 of twisted/internet/defer.py (first
assert not isinstance(result, Deferred)
That's pretty explicit in disallowing Deferreds... I'm working with
the version 2.5 release.

You've misread the code. Jonathan is correct.

What is that assertion guarding against then? Seems to me like it is
explicitly preventing you from doing this:

d1 = Deferred()
d2 = Deferred()
d1.callback(d2)

In fact, that exact code raises the AssertionError in question. If
the intention is to allow this, then the assertion needs to be
removed.

Post by Jean-Paul Calderone
The rest of this thread
seems to go off on some tangent about threads which may or may not be
relevant to whatever problem you're trying to solve. What led you to
be reading this code in the first place?
Glyph did make a good point, though. You shouldn't ever have a Deferred
in a thread other than the reactor thread. You shouldn't make them, you
shouldn't add callbacks to them, you shouldn't call them back. It might
be legitimate to get one from the reactor thread, hold it for a while,
then send it back to the reactor thread, but I can't think of a case
where this would really be very useful.

Yes, and thanks for that. I was creating a Deferred and adding a
callback in a thread, so in fixing that, the assertion above no longer
comes into play.

Ryan

[Jean-Paul Calderone]

Post by Ryan Fugger

Post by Jean-Paul Calderone

Post by Ryan Fugger
Returning a Deferred as a result from another Deferred seems to be
disallowed by an assertion at the start of the callback chain.

You've misread the code. Jonathan is correct.

What is that assertion guarding against then? Seems to me like it is
d1 = Deferred()
d2 = Deferred()
d1.callback(d2)
In fact, that exact code raises the AssertionError in question. If
the intention is to allow this, then the assertion needs to be
removed.

Here's what it sounds like "returning a Deferred as a result from another
Deferred" describes:

def f(result):
return Deferred()

d = Deferred()
d.addCallback(f)
d.callback(None)

This is a useful and very common thing to do. I see this isn't what you
were attempting to describe now, though.

Jean-Paul