[Twisted-Python] Streaming Requests

[Mark Williams]

* What?
A new year means renewed ambition. So let's talk about receiving
streaming requests!

* Why?
Twisted's HTTP server implementation does not allow application code
to interact with a request until its body has been entirely
received. It also doesn't allow incremental access to the request's
body as it arrives. This shortcoming has been an issue for a while;
see http://twistedmatrix.com/trac/ticket/288

Some of the discussion in #288 focuses on twisted.web.server and
twisted.web.resource. The approach I'll propose in this email will
not. There are two reasons for this.

The first: I want twisted.web.proxy.Proxy to support the CONNECT
HTTP method. That requires that Request.process be called before
any part of the body has been written by the client. I'd also like
to write proxies that connected the incoming request as an
IPushProducer to an outgoing one as an IConsumer. It just so
happens that Proxy inherits directly from HTTPChannel and doesn't
touch any of twisted.web.server.

The second: the consensus after some discussion on IRC in
#twisted-dev seems to be that we have to fix HTTPChannel first
anyway, and that progress there can be made entirely in Twisted's
private API. Once we have some kind of Request-like thing that
HTTPChannel can begin processing before the body has arrived, we can
work out how to integrate it in twisted.web.server and and
twisted.web.resource.

In other words, we can make this change incrementally and
backwards-compatibly, and get a better Proxy implementation out of
it, too.

* Quickly: How?
1. Define the Request interface HTTPChannel currently uses. It will
be private. Call it _IDeprecatedHTTPChannelToRequestInterface
because requests should eventually always be streaming. There's
a ticket here: https://twistedmatrix.com/trac/ticket/8981 and
some code here:
https://github.com/twisted/twisted/compare/twisted:88a7194...markrwilliams:ed19197
2. Define a new streaming Request interface that HTTPChannel knows
how to use. It will be private. Call it
_IHTTPChannelToStreamingRequest. It won't have a .content, but
it will have a way to specify a protocol that receives the body
incrementally. The interaction will probably look a lot like the
patch in https://twistedmatrix.com/trac/ticket/8143. It won't be
HTTPChannel's default requestFactory.
3. Use the private _IHTTPChannelToStreamingRequest implementation in
a new proxy implementation that supports CONNECT and also
producer/consumer streaming between the client and proxy
requests.
4. Take stock and figure out how to make things work for
twisted.web.server.

* Slowly: How?
(Note: attributions are for posterity only. Any mistakes in
reasoning are because I transcribed something badly.)

Tom Prince explained that HTTPChannel doesn't provide Request with
the HTTP method, URI, or version until the body has arrived.
Request.requestReceived, the method that receives these, calls
Request.process, which means without change this behavior we can't
change Proxy or Site, both of which begin their work by overriding
Request.process. So we have to start with HTTPChannel. (For what
it's worth, http://twistedmatrix.com/trac/ticket/288#comment:31
supports this approach.)

He also noted that the Request interface with which HTTPChannel
interacts is mostly not described by twisted.iweb.IRequest. That
means we can augment the ways HTTPChannel talks to Request-like
things without breaking many public APIs.

Glyph said we should make this existing interface explicit but
private. That will let HTTPChannel (eventually) use the interface
provided by requestFactory to determine whether to treat the Request
as streaming or not.

We can then define a new interface, _IHTTPChannelToStreamingRequest,
and a new implementation that's completely separate from
twisted.web.http.Request. Both will be private.

Tom Prince pointed out that with these two in place, we can then
write a replacement for twisted.web.proxy.Proxy that uses these
private APIs to provide HTTPS support via HTTP's CONNECT method.
HTTPChannel's default requestFactory will continue to be
twisted.web.http.Request. The new proxy code will use the new
_IHTTPChannelToStreamingRequest implementation.

Exarkun pointed out that this new proxy implementation can be
completely separate and indeed deprecate the existing one, avoiding
the need to make twisted.web.proxy.ProxyRequest.process work with
both the new _IHTTPChannelToStreamingRequest process()
implementation and the existing one. I am hopeful this new
implementation will also close
https://twistedmatrix.com/trac/ticket/8961

If all that works, we can then work out an IStreamingRequest
interface that will enable Twisted's web server utilize the private
streaming request APIs.

* Comments?
Will this approach break a public API? Does it sound terrible? Or
good? Please share your thoughts!

Let's hope 2017 is the year of the streaming request!

-Mark

[Jean-Paul Calderone]

Post by Mark Williams
* What?
A new year means renewed ambition. So let's talk about receiving
streaming requests!

Hi Mark,

Thanks for tackling this long-standing issue!

I want to start off by responding to just one part of your email. This is
actually the only part that looks potentially problematic to me so far,
which I take to be a good sign. :)

Post by Mark Williams
He also noted that the Request interface with which HTTPChannel
interacts is mostly not described by twisted.iweb.IRequest. That
means we can augment the ways HTTPChannel talks to Request-like
things without breaking many public APIs.

How will this affect applications which have implemented Request-alike
classes (for example, for testing purposes)? The interface may not be
explicitly defined as a Zope Interface right now but it is defined by the
current implementation and it is largely public (ie, non-_-prefixed
names). Tests may have hooked in to parts of Request not documented by
IRequest out of necessity. It would be a shame to break such carefully
crafted automated tests.

Here's one example I know of off the top of my head, <
https://github.com/ScatterHQ/flocker/blob/master/flocker/restapi/testtools.py#L316>.
This one isn't a from-scratch re-implementation of the implicit Request
interface but a Request subclass, instead. However, it still interacts
with a lot of important pieces of Request which aren't part of IRequest.

Jean-Paul

[Mark Williams]

Post by Jean-Paul Calderone
Hi Mark,
Thanks for tackling this long-standing issue!

I hope I actually make some progress :)

Post by Jean-Paul Calderone

Post by Mark Williams
He also noted that the Request interface with which HTTPChannel
interacts is mostly not described by twisted.iweb.IRequest. That
means we can augment the ways HTTPChannel talks to Request-like
things without breaking many public APIs.

How will this affect applications which have implemented Request-alike
classes (for example, for testing purposes)? The interface may not be
explicitly defined as a Zope Interface right now but it is defined by the
current implementation and it is largely public (ie, non-_-prefixed
names). Tests may have hooked in to parts of Request not documented by
IRequest out of necessity. It would be a shame to break such carefully
crafted automated tests.

The new streaming request interface will define how HTTPChannel talks
to a completely separate Request-like implementation. HTTPChannel can
then inspect what interface is provided by the object returned from
its requestFactory and either use the existing code path or the new
streaming one. The default requestFactory will remain
twisted.web.http.Request.

These things taken together should mean that existing code doesn't
need to change at all. I hope to use Twisted's own tests to verify
this, but I might use Flocker's as well.

I'd like to point out that this approach means that defining
_IDeprecatedHTTPChannelToRequestInterface isn't strictly necessary.
I'd like to do so anyway to help me track the existing API so I'm less
likely to break it, and to make the dispatch logic in HTTPChannel more
robust.

[Glyph Lefkowitz]

Here's one example I know of off the top of my head, <https://github.com/ScatterHQ/flocker/blob/master/flocker/restapi/testtools.py#L316 <https://github.com/ScatterHQ/flocker/blob/master/flocker/restapi/testtools.py#L316>>. This one isn't a from-scratch re-implementation of the implicit Request interface but a Request subclass, instead. However, it still interacts with a lot of important pieces of Request which aren't part of IRequest.

Mark already addressed how he won't be breaking this use-case (which is hugely important, and core to the whole idea of a compatibility policy, so that is as it should be).

However, this kind of test-mocking is, I think, ultimately done at the wrong layer. It's trying to override some very vaguely-specified internals in the middle of an implementation.

Really, twisted.web should provide its own testing tools, of course. But if you're going to implement something that does this sort of overriding, I think the idiom to follow would be treq.testing: https://github.com/twisted/treq/blob/fcf5deb976c955ca6ef6484f414d25839932940e/src/treq/testing.py <https://github.com/twisted/treq/blob/fcf5deb976c955ca6ef6484f414d25839932940e/src/treq/testing.py>, rather than any of the various implementations of DummyRequest (including more than a few I'm sure I've written).

Internally, treq.testing uses MemoryReactor and the (not-technically-public-but-it-really-should-be) iosim module. Therefore the layer it's plugging in at is at the very well-defined interface between IAgent and IStreamClientEndpoint, and between HTTPChannel and Transport. This gives the determinism and (most of the) speed of an in-memory implementation, but a great deal of the realism of a full-blown integration test. In particular, it binds to the appropriate objects within Twisted and will throw deprecation warnings if their interfaces change. (Also of note; since what it's building a fake for is really 99% Agent, the bulk of this could probably move up another level into Twisted with very little effort.)

I think following this idiom of pushing I/O through a fake reactor and having a back-end that can be controlled via test-specific APIs is probably what we should all be striving for, and in Twisted we should be working to make the memory-reactor stuff more complete and convenient, rather than adding dummy implementations at each layer of the stack.

This is a set of ideas that I've been gradually arriving at over a long period of time, but it's probably high time for some public discussion by now, even if it's just everybody saying "yeah, that sounds good" :-).

On a related note, 'proto_helpers' is in a super awkward place. 'twisted.testing', anyone? :)

-glyph

[Jean-Paul Calderone]

Post by Jean-Paul Calderone
Here's one example I know of off the top of my head, <
https://github.com/ScatterHQ/flocker/blob/master/flocker/
restapi/testtools.py#L316>. This one isn't a from-scratch
re-implementation of the implicit Request interface but a Request subclass,
instead. However, it still interacts with a lot of important pieces of
Request which aren't part of IRequest.
Mark already addressed how he won't be breaking this use-case (which is
hugely important, and core to the whole idea of a compatibility policy, so
that is as it should be).
However, this kind of test-mocking is, I think, ultimately done at the
wrong layer. It's trying to override some very vaguely-specified internals
in the middle of an implementation.

Absolutely.

Post by Jean-Paul Calderone
Really, twisted.web should provide its own testing tools, of course. But
if you're going to implement something that does this sort of overriding, I
think the idiom to follow would be treq.testing: https://github.
com/twisted/treq/blob/fcf5deb976c955ca6ef6484f414d25
839932940e/src/treq/testing.py, rather than any of the various
implementations of DummyRequest (including more than a few I'm sure I've
written).

Though, note, the link I gave above was support code for something very
similar to this treq code:
https://github.com/ScatterHQ/flocker/blob/master/flocker/restapi/testtools.py#L460

To clarify your point a bit (I think):


- MemoryAgent (from Flocker) provides a testing IAgent by implementing
an IRequest that does in-memory resource traversal to dispatch requests and
generate responses.
- RequestTraversalAgent (from treq) provides a testing IAgent by
implementing (using) iosim to do in-memory protocol/transport interacts to
drive an in-memory HTTP conversation that runs all of the regular Twisted
Web HTTP processing machinery.

RequestTraversalAgent's approach is better because the protocol/transport
interface is better defined. Because it's better
defined, RequestTraversalAgent doesn't have to touch pieces that we might
want to consider implementation details (whether they're _-prefixed or
not). It also invokes a larger portion of the real implementation code
making it more likely to be a realistic simulation of real-world use of the
code.

Having spelled this out, what occurs to me now is
that RequestTraversalAgent is really just a step or so up the ladder and
there's further to go. For example, RequestTraversalAgent only needs to
exist at all because `iosim` has a distinct interface. This distinct
interface means you need a RequestTraversalAgent-like thing for each
reactor-using thing for which you want to provide a test double. If this
adaption behavior were bundled up differently then I think we'd get a lot
more in-memory test doubles for free (or closer to it - we'd be another
rung up the ladder, at least). That seems like it would be a big win given
how *few* of these testing helpers Twisted has historically managed to
provide (suggesting it's just too hard to do so given the current tools).

Post by Jean-Paul Calderone
This is a set of ideas that I've been gradually arriving at over a long
period of time, but it's probably high time for some public discussion by
now, even if it's just everybody saying "yeah, that sounds good" :-).

So, it pretty much sounds good, though with the above refinements. :)

Post by Jean-Paul Calderone
On a related note, 'proto_helpers' is in a super awkward place.
'twisted.testing', anyone? :)

Yes. I almost suggested this about a week ago when I was preparing to
contribute some testing code but then realized I couldn't contribute the
code after all. :(

Jean-Paul

[Glyph Lefkowitz]

Post by Glyph Lefkowitz

Here's one example I know of off the top of my head, <https://github.com/ScatterHQ/flocker/blob/master/flocker/restapi/testtools.py#L316 <https://github.com/ScatterHQ/flocker/blob/master/flocker/restapi/testtools.py#L316>>. This one isn't a from-scratch re-implementation of the implicit Request interface but a Request subclass, instead. However, it still interacts with a lot of important pieces of Request which aren't part of IRequest.

Mark already addressed how he won't be breaking this use-case (which is hugely important, and core to the whole idea of a compatibility policy, so that is as it should be).
However, this kind of test-mocking is, I think, ultimately done at the wrong layer. It's trying to override some very vaguely-specified internals in the middle of an implementation.
Absolutely.

Yay! Glad to have some consensus on this.

Post by Glyph Lefkowitz
Really, twisted.web should provide its own testing tools, of course. But if you're going to implement something that does this sort of overriding, I think the idiom to follow would be treq.testing: https://github.com/twisted/treq/blob/fcf5deb976c955ca6ef6484f414d25839932940e/src/treq/testing.py <https://github.com/twisted/treq/blob/fcf5deb976c955ca6ef6484f414d25839932940e/src/treq/testing.py>, rather than any of the various implementations of DummyRequest (including more than a few I'm sure I've written).
Though, note, the link I gave above was support code for something very similar to this treq code: https://github.com/ScatterHQ/flocker/blob/master/flocker/restapi/testtools.py#L460 <https://github.com/ScatterHQ/flocker/blob/master/flocker/restapi/testtools.py#L460>
MemoryAgent (from Flocker) provides a testing IAgent by implementing an IRequest that does in-memory resource traversal to dispatch requests and generate responses.
RequestTraversalAgent (from treq) provides a testing IAgent by implementing (using) iosim to do in-memory protocol/transport interacts to drive an in-memory HTTP conversation that runs all of the regular Twisted Web HTTP processing machinery.
RequestTraversalAgent's approach is better because the protocol/transport interface is better defined. Because it's better defined, RequestTraversalAgent doesn't have to touch pieces that we might want to consider implementation details (whether they're _-prefixed or not). It also invokes a larger portion of the real implementation code making it more likely to be a realistic simulation of real-world use of the code.

100% agree.

Post by Glyph Lefkowitz
Having spelled this out, what occurs to me now is that RequestTraversalAgent is really just a step or so up the ladder and there's further to go.
For example, RequestTraversalAgent only needs to exist at all because `iosim` has a distinct interface. This distinct interface means you need a RequestTraversalAgent-like thing for each reactor-using thing for which you want to provide a test double.

I think that there are pieces of RequestTraversalAgent for which this is true; the parts where it has to instantiate its own FakeTransport objects and IPv4Addresses and whatever. This could be streamlined a lot more.

But at the higher level, each layer will want its own public API entrypoint, to facilitate the orchestration and setup of domain-specific client/server objects and reduce boilerplate for testing at a given level.

Post by Glyph Lefkowitz
If this adaption behavior were bundled up differently then I think we'd get a lot more in-memory test doubles for free (or closer to it - we'd be another rung up the ladder, at least).

This I definitely agree with. Each abstraction-level-specific fake should be a paper-thin layer over the one below, and each abstraction layer should cover a fair amount of ground for itself. For example, you can test Klein apps trivially with treq.testing, without adding very much extra logic at all.

Post by Glyph Lefkowitz
That seems like it would be a big win given how few of these testing helpers Twisted has historically managed to provide (suggesting it's just too hard to do so given the current tools).

Here I think there are a few mitigating factors:

I don't actually think that it's that hard using current tools. It's pretty easy, in fact, to build up from the bottom and layer subsequent things on top of another.
The way I see the problem that we're facing is that we realized we should be doing this many years after doing the actual implementation, and we realized this several layers up. So we built mid-tier test mocks and then tried to build on top of them, rather than heading straight for the bottom layer. This has lead to awkwardness like needing to monkeypatch HostnameEndpoint to avoid deferToThread calls.
So, even now, we have an anemic bottom layer. task.Clock is pretty good, but MemoryReactor still leaves a lot to be desired.
Therefore, I don't think that we really need to build that many different abstraction layers; if we really focus on nailing down (A) MemoryReactor+iosim+something with addresses so 'connectTCP' can be hooked up to 'listenTCP' when both are given the same address, and (B) the vast majority of what folks want to be able to test is HTTP API type stuff, so integrating treq.testing (treq is license-compatible, so we can just move the code if we want) would get us a _long_ way there, and would popularize this style with a lot of other developers who might then be motivated to help improve the more obscure corners, like, say, DNS.

I've been slowly improving the testability of Twisted's core; the recent name-resolver change made a LOT of things easier. We should be making deterministicResolvingReactor and twisted.threads (along with MemoryWorker) public, and a lot of the circuitous routes that higher-level code needed to traverse to get to "no I/O" will just go away. I'm probably forgetting a few other building blocks, but I've noticed that each additional one that I remove makes the task more like exponentially than linearly easier.

Post by Glyph Lefkowitz
This is a set of ideas that I've been gradually arriving at over a long period of time, but it's probably high time for some public discussion by now, even if it's just everybody saying "yeah, that sounds good" :-).
So, it pretty much sounds good, though with the above refinements. :)

Cool. I suspect I haven't even covered all of it with my refine-refinements above, but at this point it seems clear we can all pull in the same direction. I wonder if a themed sprint at some point in the next year might be worthwhile?

Post by Glyph Lefkowitz
On a related note, 'proto_helpers' is in a super awkward place. 'twisted.testing', anyone? :)
Yes. I almost suggested this about a week ago when I was preparing to contribute some testing code but then realized I couldn't contribute the code after all. :(

Well, now it's just a matter of time :).

-g

[Jean-Paul Calderone]

Post by Jean-Paul Calderone

Post by Glyph Lefkowitz
On a related note, 'proto_helpers' is in a super awkward place.
'twisted.testing', anyone? :)

Yes. I almost suggested this about a week ago when I was preparing to
contribute some testing code but then realized I couldn't contribute the
code after all. :(
Well, now it's just a matter of time :).

Some further thoughts on this... What I've actually been doing elsewhere (
*mostly*) is to put the testing implementations quite near the real
implementations. For example, I recently introduced
`workproject.subscription_manager.{network_client,memory_client}`. But
contrary to this, in txAWS I'm following the pre-existing pattern which is
more like `txaws.service.AWSServiceRegion.get_ec2_client` &
`txaws.testing.service.FakeAWSServiceRegion.get_ec2_client`.

Which I prefer, I'm not really sure yet. Role-named modules are easy to
recognize. On the other hand, testing code next to implementation code is
easier to stumble across and you don't *have* to recognize a separate
testing module to find it. Also, role-named modules tend to pile up and
get in each other's way (start working with `twisted.testing`,
`txaws.testing`, etc and things start to get confusing quickly).

Additionally, a related idea is that often the testing implementation might
actually serve as a good starting place for the real implementation. I
think the code in txAWS is a good example of this (ie, the real and fake
implementations duplicate a lot of logic with some pointless divergences).
In the olden days I would have said "the testing implementation is often a
good *base class* for the real implementation". But now I'll mumble
something about composition. Put another way:

- Clock should be the real implementation of all of the time-source
independent logic of IReactorTime
- FakeAWSServiceRegion should be the real implementation of the "client
factory" logic (get_XXX_client) of AWSServiceRegion

and so on.

Perhaps this also speaks to the idea you mentioned that there should be a
stack of thin abstractions but I snipped that content already so I'll try
responding to that separately, later.

Jean-Paul

[Glyph Lefkowitz]

Post by Glyph Lefkowitz

Post by Glyph Lefkowitz
On a related note, 'proto_helpers' is in a super awkward place. 'twisted.testing', anyone? :)
Yes. I almost suggested this about a week ago when I was preparing to contribute some testing code but then realized I couldn't contribute the code after all. :(

Well, now it's just a matter of time :).
Some further thoughts on this... What I've actually been doing elsewhere (mostly) is to put the testing implementations quite near the real implementations. For example, I recently introduced `workproject.subscription_manager.{network_client,memory_client}`. But contrary to this, in txAWS I'm following the pre-existing pattern which is more like `txaws.service.AWSServiceRegion.get_ec2_client` & `txaws.testing.service.FakeAWSServiceRegion.get_ec2_client`.

"Memory" implementations are definitely a step up the value hierarchy from "fake" implementations.

I guess maybe both of these patterns make sense for certain things. It makes sense to put a memory implementation next to the real one (c.f. SQLite), but it doesn't make as much sense to put a testing implementation there. I would make this distinction by saying a "testing" implementation is one which has assertion method helpers, simulated I/O triggers, and similar.

Post by Glyph Lefkowitz
Which I prefer, I'm not really sure yet. Role-named modules are easy to recognize. On the other hand, testing code next to implementation code is easier to stumble across and you don't have to recognize a separate testing module to find it. Also, role-named modules tend to pile up and get in each other's way (start working with `twisted.testing`, `txaws.testing`, etc and things start to get confusing quickly).

Yeah, and I guess it'll actually be `twisted.internet.testing`, `twisted.python.testing`, `twisted.web.testing` etc. Which we explicitly rejected with `interfaces` because it got too confusing; the existence of multiple `endpoints` modules has been mildly irritating while working on HostnameEndpoint for the last couple of days.

Post by Glyph Lefkowitz
Additionally, a related idea is that often the testing implementation might actually serve as a good starting place for the real implementation. I think the code in txAWS is a good example of this (ie, the real and fake implementations duplicate a lot of logic with some pointless divergences). In the olden days I would have said "the testing implementation is often a good base class for the real implementation". But now I'll mumble something about composition.

Yeah, I think this is the broader point, actually - the best possible "fake" implementation is a completely real implementation where the lower level it's composing against is fake.

Post by Glyph Lefkowitz
Clock should be the real implementation of all of the time-source independent logic of IReactorTime

One minor tweak: Clock itself should be split into two objects, one of which has just 'advance' and one of which has IReactorTime <https://glyph.twistedmatrix.com/2015/05/separate-your-fakes-and-your-inspectors.html <https://glyph.twistedmatrix.com/2015/05/separate-your-fakes-and-your-inspectors.html>>. Possibly three objects, since it gets its `seconds` from somewhere else?

Post by Glyph Lefkowitz
FakeAWSServiceRegion should be the real implementation of the "client factory" logic (get_XXX_client) of AWSServiceRegion
and so on.

Yes, definitely.

Post by Glyph Lefkowitz
Perhaps this also speaks to the idea you mentioned that there should be a stack of thin abstractions but I snipped that content already so I'll try responding to that separately, later.

Ideally, the only "fake" implementation we should need is StringTransport and Clock; the abstractions in the middle would be wiring together the layers in a way that's convenient for test clients to access.

-glyph

[steven meiers]

hi,



this websocket code works but now i would like to be able to import it
to another class to have a way to send messages without waiting for the
answer.

the code is below here and also pasted here: https://bpaste.net/show/78
36ba3b8008 for some nice syntax highlighting.


just running websockets.py works...sending and receiving messages.
importing does not, says:

builtins.AttributeError: 'MyClientProtocol' object has no attribute
'deferred'


any ideas why it does give me this error?



but since the code works on its own i dont see why it can be imported.

first code paste is the import test which fails, after that is the
error mesage and at the end is the code (mywebsocket.py) that is to be
imported.



from mywebsocket import runIt
import sys
from autobahn.twisted.websocket import WebSocketClientProtocol, \
            WebSocketClientFactory
from mywebsocket import MyClientProtocol
from twisted.python import log
from twisted.internet import reactor

if __name__ == '__main__':
    log.startLogging(sys.stdout)

    factory = WebSocketClientFactory(u"ws://127.0.0.1:8080/javaee7-
websocket-chat/chat/arduino")
    factory.protocol = MyClientProtocol
    reactor.connectTCP("127.0.0.1", 8080, factory)

    d, factory = runIt()
    def test_sendMessage(result):
        print("test_sesndMessage():", result)
        factory.p.sendMessage('python client', 'very nice message')

    d.addCallback(test_sendMessage)
    reactor.run()






python testimport.py 
2017-01-17 12:57:43+0100 [-] Log opened.
2017-01-17 12:57:43+0100 [-] Starting factory
<autobahn.twisted.websocket.WebSocketClientFactory object at
0x7fea4c21d358>
2017-01-17 12:57:43+0100 [-] Starting factory <mywebsocket.MyFactory
object at 0x7fea48a34b38>
2017-01-17 12:57:43+0100 [-] Server connected: tcp4:127.0.0.1:8080
2017-01-17 12:57:43+0100 [-] WebSocket connection open.
2017-01-17 12:57:43+0100 [-] test_sesndMessage(): yep that worked
2017-01-17 12:57:43+0100 [-] Server connected: tcp4:127.0.0.1:8080
2017-01-17 12:57:43+0100 [-] WebSocket connection open.
2017-01-17 12:57:43+0100 [MyClientProtocol,client] Unhandled Error
Traceback (most recent call last):
  File "/home/julius/code/python/twisted/webchat-
client/venv/lib/python3.5/site-packages/twisted/python/log.py", line
103, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/home/julius/code/python/twisted/webchat-
client/venv/lib/python3.5/site-packages/twisted/python/log.py", line
86, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args,
**kw)
  File "/home/julius/code/python/twisted/webchat-
client/venv/lib/python3.5/site-packages/twisted/python/context.py",
line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func,
*args, **kw)
  File "/home/julius/code/python/twisted/webchat-
client/venv/lib/python3.5/site-packages/twisted/python/context.py",
line 81, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/home/julius/code/python/twisted/webchat-
client/venv/lib/python3.5/site-packages/twisted/internet/posixbase.py",
line 597, in _doReadOrWrite
    why = selectable.doRead()
  File "/home/julius/code/python/twisted/webchat-
client/venv/lib/python3.5/site-packages/twisted/internet/tcp.py", line
208, in doRead
    return self._dataReceived(data)
  File "/home/julius/code/python/twisted/webchat-
client/venv/lib/python3.5/site-packages/twisted/internet/tcp.py", line
214, in _dataReceived
    rval = self.protocol.dataReceived(data)
  File "/home/julius/code/python/twisted/webchat-
client/venv/lib/python3.5/site-packages/autobahn/twisted/websocket.py",
line 132, in dataReceived
    self._dataReceived(data)
  File "/home/julius/code/python/twisted/webchat-
client/venv/lib/python3.5/site-
packages/autobahn/websocket/protocol.py", line 1183, in _dataReceived
    self.consumeData()
  File "/home/julius/code/python/twisted/webchat-
client/venv/lib/python3.5/site-
packages/autobahn/websocket/protocol.py", line 1212, in consumeData
    self.processHandshake()
  File "/home/julius/code/python/twisted/webchat-
client/venv/lib/python3.5/site-
packages/autobahn/websocket/protocol.py", line 3784, in
processHandshake
    self._onOpen()
  File "/home/julius/code/python/twisted/webchat-
client/venv/lib/python3.5/site-packages/autobahn/twisted/websocket.py",
line 142, in _onOpen
    self.onOpen()
  File "/home/julius/code/python/twisted/webchat-
client/mywebsocket.py", line 18, in onOpen
    self.deferred.callback('yep that worked')
builtins.AttributeError: 'MyClientProtocol' object has no
attribute 'deferred'

2017-01-17 12:57:43+0100 [-] WebSocket connection closed: connection
was closed uncleanly (peer dropped the TCP connection without previous
WebSocket closing handshake)
2017-01-17 12:57:43+0100 [-] Stopping factory
<autobahn.twisted.websocket.WebSocketClientFactory object at
0x7fea4c21d358>
2017-01-17 12:57:43+0100 [-] Text message received: {"message":"very
nice message","sender":"python client","received":"Tue Jan 17 12:57:43
CET 2017"}
^C2017-01-17 12:57:44+0100 [-] Received SIGINT, shutting down.
2017-01-17 12:57:44+0100 [-] WebSocket connection closed: connection
was closed uncleanly (peer dropped the TCP connection without previous
WebSocket closing handshake)
2017-01-17 12:57:44+0100 [-] Stopping factory <mywebsocket.MyFactory
object at 0x7fea48a34b38>
2017-01-17 12:57:44+0100 [-] Main loop terminated.







mywebsocket.py
from autobahn.twisted.websocket import WebSocketClientProtocol, \
    WebSocketClientFactory
import sys

from twisted.python import log
from twisted.internet import reactor, defer

import json

class MyClientProtocol(WebSocketClientProtocol):

    def onConnect(self, response):
        tmp = "Server connected: {0}".format(response.peer)
        print(tmp)

    def onOpen(self):
        print("WebSocket connection open.")
        self.deferred.callback('yep that worked')

        def hello():
            print("sending message")
            message = json.dumps({"message":"1" ,"sender":"2",
"received":"3"}).encode('utf-8')

            
            self.sendMessage(message)
            self.factory.reactor.callLater(10, hello)


    def sendMessage(self, sender, message):
        message = json.dumps({"message": message, "sender": sender,
"received": "notusedyet"})
        super(MyClientProtocol, self).sendMessage(message.encode('utf-
8'))

    def onMessage(self, payload, isBinary):
        if isBinary:
            print("Binary message received: {0}
bytes".format(len(payload)))
        else:
            print("Text message received:
{0}".format(payload.decode('utf8')))

    def onClose(self, wasClean, code, reason):
        print("WebSocket connection closed: {0}".format(reason))




class MyFactory(WebSocketClientFactory):

    def __init__(self, deferred, *args, **kwargs):
        self.d = deferred
        super(MyFactory, self).__init__(*args, **kwargs)

    def buildProtocol(self, addr):
        p = super(MyFactory, self).buildProtocol(addr)
        self.p = p
        p.deferred = self.d
        return p



def runIt():
    log.startLogging(sys.stdout)


    d = defer.Deferred()

    factory = MyFactory(d, u"ws://127.0.0.1:8080/javaee7-websocket-
chat/chat/arduino")


    factory.protocol = MyClientProtocol
    
    def test_sendMessage(result):
        print("test_sesndMessage():", result)
        factory.p.sendMessage('python client', 'very nice message')

    reactor.connectTCP("127.0.0.1", 8080, factory)
    return (d, factory)


if __name__ == '__main__':
    runIt()
    reactor.run()

[Cory Benfield]

Post by Mark Williams
* Comments?
Will this approach break a public API? Does it sound terrible? Or
good? Please share your thoughts!
Let's hope 2017 is the year of the streaming request!
-Mark

I’m very excited to see someone tackling this again: I began taking a swing at it last year and abandoned it in favour of some of the other HTTP/2 work. However, I think it’s a vital addition and I really look forward to seeing it land.

For my part, I should note that HTTPChannel has a twin in HTTP/2 land, which is the combination of t.w._http2.H2Connection and t.w._http2.H2Stream. These will also want updating. However, once you’ve defined the shape of the patch for HTTPChannel I am happy to help out with reviewing that and making corresponding changes to the HTTP/2 side of things too.

Cory