[Twisted-Python] web.client.readBody

[John Aherne]

I have been using deliverBody to process in responses but decided that
readBody might be a better fit.

So far it works with http, but as soon as I switch to https it fails.

I get a response http code 200 OK but no data.

I've puzzled over this but can't see what the problem is.

Has anyone else seen this problem and found out what the solution is.

Thanks



Response received <twisted.web._newclient.Response object at 0x0362EB50>
Response version: ('HTTP', 1, 1)
Response code: 200
Response phrase: OK
Response headers: Headers({'alternate-protocol': ['443:quic'],
'x-xss-protection': ['1; mode=block'], 'accept-ranges': ['none'],
'expires': ['Mon, 04 Apr 2016 13:32:31 GMT'], 'vary':
['Accept-Language,Accept-Encoding'], 'server': ['mafe'], 'cache-control':
['public, max-age=86400'], 'date': ['Sun, 03 Apr 2016 13:32:31 GMT'],
'x-frame-options': ['SAMEORIGIN'], 'alt-svc': ['quic=":443"; ma=2592000;
v="32,31,30,29,28,27,26,25"'], 'content-type': ['application/json;
charset=UTF-8']})
LENGTH twisted.web.iweb.UNKNOWN_LENGTH
[('Alternate-Protocol', ['443:quic']),
('X-XSS-Protection', ['1; mode=block']),
('Accept-Ranges', ['none']),
('Expires', ['Mon, 04 Apr 2016 13:32:31 GMT']),
('Vary', ['Accept-Language,Accept-Encoding']),
('Server', ['mafe']),
('Cache-Control', ['public, max-age=86400']),
('Date', ['Sun, 03 Apr 2016 13:32:31 GMT']),
('X-Frame-Options', ['SAMEORIGIN']),
('Alt-Svc', ['quic=":443"; ma=2592000; v="32,31,30,29,28,27,26,25"']),
('Content-Type', ['application/json; charset=UTF-8'])]
Got Error
error res [Failure instance: Traceback (failure with no frames): <class
'twisted.web.client.PartialDownloadError'>: 200 OK
]
SHUTDOWN

[Cory Benfield]

I have been using deliverBody to process in responses but decided that readBody might be a better fit.
So far it works with http, but as soon as I switch to https it fails.
I get a response http code 200 OK but no data.
I've puzzled over this but can't see what the problem is.
Has anyone else seen this problem and found out what the solution is.
Thanks
LENGTH twisted.web.iweb.UNKNOWN_LENGTH

This indicates that the remote server (which, while your logs don’t outright say it, is clearly a Google server) is doing something particularly stupid: that is, they’re sending a response that is neither chunked nor content-length-delimited. This means that message completion can only be signalled by the closing of the connection once the response is complete.

Twisted, correctly, gets a bit nervous about this: it’s very difficult to tell the actual completion of the response from any number of error conditions where the connection gets abruptly torn down. For this reason, the docstring of IResponse.deliverBody says that in a case like this: "The protocol's connectionLost method will be called with: PotentialDataLoss, which indicates that it cannot be determined if the entire response body has been delivered.”

When readBody’s protocol connectionLost method is called with PotentialDataLoss, it calls the errback with PartialDownloadError, which is what you’re seeing. The body of the response is available on the PartialDownloadError as PartialDownloadError.response, so if you’re interested in continuing to use readBody you’ll probably want to register an errback on the deferred that checks for this error and handles it appropriately: in your case, probably by converting the error to a safe response and then calling the callback!

In this instance, I’d *also* recommend that you reach out to whatever Google service is sending this response. RFC 7230 says that a server SHOULD send a Content-Length header if not sending a Transfer-Encoding: chunked header, and points out that not doing so exists primarily for backward compatibility with HTTP/1.0.

I hope that all helps!

Cory

[John Aherne]

Cory

Thanks for the detailed reply.

I'll see what I can make of it.

I must admit I don't see Google paying much attention to this.

It does work if I use response.deliverBody, but I thought getting the
complete response in one hit would be better.

John

[Cory Benfield]

Post by John Aherne
Cory
Thanks for the detailed reply.
I'll see what I can make of it.
I must admit I don't see Google paying much attention to this.
It does work if I use response.deliverBody, but I thought getting the complete response in one hit would be better.
John

Yeah, so the difference is that with response.deliverBody you’re not checking the code in connectionLost. If you were, you’d be seeing PotentialDataLoss as well. ;)

There’s nothing wrong with using readBody, you just need to adjust your errback/callback chain to transform the response appropriately in this case.

And yeah, I doubt Google will listen. =P

Cory