[Twisted-Python] yielding from within deferToThread

[Naveen Michaud-Agrawal]

I'm trying to use sqlalchemy from a twisted application (by running all
blocking queries using deferToThread). Is it possible to yield from within
the function running in deferToThread? For example:


def threadRunQuery(engine, query):
conn = engine.connect()
res = conn.execute(query)

while True:
results = res.fetchmany(1000)
if not results:
break
yield results


@defer.inlineCallbacks
def stream_results():
engine = sqlalchemy.create_engine(...)
query = "select * from table"
result_iter = yield threads.deferToThread(threadRunQuery, engine, query)
for results in result_iter:
print results


It seems that the thread returns a generator, and so everything within
threadRunQuery is actually running on the main reactor thread. Is there
anyway to stream back results from a deferToThread?

Regards
Naveen Michaud-Agrawal

[Naveen Michaud-Agrawal]

To answer my own question, it looks like i can just pass sqlalchemy's
ResultProxy into deferToThread to make a blocking call to fetch the next
set of results:

@defer.inlineCallbacks
def stream_results():
engine = sqlalchemy.create_engine(...)
query = "select * from table"
proxy = yield threads.deferToThread(engine.execute, query)
while True:
results = yield threads.deferToThread(proxy.fetchmany, 1000)
if not results:
break
print results



On Thu, Dec 3, 2015 at 2:41 PM, Naveen Michaud-Agrawal <

Post by Naveen Michaud-Agrawal
I'm trying to use sqlalchemy from a twisted application (by running all
blocking queries using deferToThread). Is it possible to yield from within
conn = engine.connect()
res = conn.execute(query)
results = res.fetchmany(1000)
break
yield results
@defer.inlineCallbacks
engine = sqlalchemy.create_engine(...)
query = "select * from table"
result_iter = yield threads.deferToThread(threadRunQuery, engine, query)
print results
It seems that the thread returns a generator, and so everything within
threadRunQuery is actually running on the main reactor thread. Is there
anyway to stream back results from a deferToThread?
Regards
Naveen Michaud-Agrawal