[Twisted-Python] web client FileBodyProducer - transfer encoding

[Kevin Mcintyre]

hey all - quick question. Trying to understand FileBodyProducer as it
pertains to POST.

json_body = FileBodyProducer(StringIO(json.dumps({'key': 'value'})))

agent.request("POST", ~*uri*, Headers({'User-Agent': ['AkamaiTest']}),
json_body)

Does the agent chunk the POST body? Is this controllable?

Any insight appreciated!

[Glyph Lefkowitz]

hey all - quick question. Trying to understand FileBodyProducer as it pertains to POST.
json_body = FileBodyProducer(StringIO(json.dumps({'key': 'value'})))
agent.request("POST", ~uri, Headers({'User-Agent': ['AkamaiTest']}), json_body)
Does the agent chunk the POST body?

Maybe!

Is this controllable?

Not really. The current implementation will certainly use chunked encoding sometimes, but there isn't a strong API guarantee of this anywhere.

Any insight appreciated!

You probably don't want to try to control this. I believe that proxies are within their rights to mess around with chunk boundaries and re-buffer things, so you don't have any strong guarantees that chunk sizes will be preserved.

Why is it that you want to control chunking in the first place?

-glyph

[Kevin Mcintyre]

Dealing with older apache, ruby, passenger setup. Hoping to mimic behavior
and show OPS that chunked encoding isn't working correctly.

With python requests (http://docs.python-requests.org/en/master/) all POST
calls work correctly, but twisted requests are failing.

I'm thinking (and hoping) it's failing because twisted is chunking the post
body. When I send the requests to a twisted endpoint like below the
requests are successful and the response is identical.

from twisted.web import server, resource
from twisted.internet import reactor

class Simple(resource.Resource):
isLeaf = True
def render_POST(self, request):
print request.content.getvalue()
return request.content.getvalue()

site = server.Site(Simple())
reactor.listenTCP(8080, site)
reactor.run()

hey all - quick question. Trying to understand FileBodyProducer as it pertains to POST.
json_body = FileBodyProducer(StringIO(json.dumps({'key': 'value'})))
agent.request("POST", ~*uri*, Headers({'User-Agent': ['AkamaiTest']}), json_body)
Does the agent chunk the POST body?
Maybe!
Is this controllable?
Not really. The current implementation will certainly use chunked
encoding sometimes, but there isn't a strong API guarantee of this anywhere.
Any insight appreciated!
You probably don't want to try to control this. I believe that proxies
are within their rights to mess around with chunk boundaries and re-buffer
things, so you don't have any strong guarantees that chunk sizes will be
preserved.
Why is it that you want to control chunking in the first place?
-glyph
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Glyph Lefkowitz]

Gotcha. I guess what I meant was that you shouldn't care about this at the application level, but you're talking about an operational concern, not an application-level concern.

Perhaps this should be a tunable on Agent somehow. Can you file a ticket?

-glyph

Dealing with older apache, ruby, passenger setup. Hoping to mimic behavior and show OPS that chunked encoding isn't working correctly.
With python requests (http://docs.python-requests.org/en/master/ <http://docs.python-requests.org/en/master/>) all POST calls work correctly, but twisted requests are failing.
I'm thinking (and hoping) it's failing because twisted is chunking the post body. When I send the requests to a twisted endpoint like below the requests are successful and the response is identical.
from twisted.web import server, resource
from twisted.internet import reactor
isLeaf = True
print request.content.getvalue()
return request.content.getvalue()
site = server.Site(Simple())
reactor.listenTCP(8080, site)
reactor.run()

hey all - quick question. Trying to understand FileBodyProducer as it pertains to POST.
json_body = FileBodyProducer(StringIO(json.dumps({'key': 'value'})))
agent.request("POST", ~uri, Headers({'User-Agent': ['AkamaiTest']}), json_body)
Does the agent chunk the POST body?

Maybe!

Is this controllable?

Not really. The current implementation will certainly use chunked encoding sometimes, but there isn't a strong API guarantee of this anywhere.

Any insight appreciated!

You probably don't want to try to control this. I believe that proxies are within their rights to mess around with chunk boundaries and re-buffer things, so you don't have any strong guarantees that chunk sizes will be preserved.
Why is it that you want to control chunking in the first place?
-glyph
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python <http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Jean-Paul Calderone]

Post by Glyph Lefkowitz
Gotcha. I guess what I meant was that you shouldn't care about this at
the application level, but you're talking about an operational concern, not
an application-level concern.
Perhaps this should be a tunable on Agent somehow. Can you file a ticket?

Well... It *is* sort of tuneable, as you implied earlier.

If you pass an IBodyProducer with a non-None length, Agent will send a
Content-Length header - not use chunked Transfer-Encoding.
FileBodyProducer doesn't know how to determine the length of a StringIO, so
you get chunked with this example. If you write the JSON to a regular file
and FileBodyProducer(open(the file)) you'll get a Content-Length request.
You could also write a new (trivial) IBodyProducer that does know how to
compute the length of a StringIO.

The documentation doesn't exactly spell this out - but the only reason
`length` is part of the interface is to be able to generate the
Content-Length header.

What would the ticket be? Expanded documentation to make this behavior
into an explicit guarantee of the interface? A new toggle somewhere to
force Agent.request into one mode or the other (regardless of the
performance consequences)?

And that stuff you said about proxies before was true ... So even if you
can control this in Agent, you're still not *guaranteed* the server will
see what you send.

Jean-Paul

[Kevin Mcintyre]

Thanks - that's exactly what I was looking for.

*But note* when using FileBodyProducer(StringIO(json.dumps(~blah))) -- I
still see a content-length in the request header as it's received by
twisted. Am I correct to assume the request is agnostic...meaning it's
shaped the same for twisted as it is for apache.

Headers({'host': ['localhost:8080'], 'connection': ['close'],
'content-length': ['671'], 'user-agent': ['PassengerTest']})

from twisted.web import server, resource
from twisted.internet import reactor

class Simple(resource.Resource):
isLeaf = True
def render_POST(self, request):
print request.requestHeaders
return request.content.getvalue()

site = server.Site(Simple())
reactor.listenTCP(8080, site)
reactor.run()



On Mon, Jan 30, 2017 at 4:59 PM, Jean-Paul Calderone <

Post by Jean-Paul Calderone

Post by Glyph Lefkowitz
Gotcha. I guess what I meant was that you shouldn't care about this at
the application level, but you're talking about an operational concern, not
an application-level concern.
Perhaps this should be a tunable on Agent somehow. Can you file a ticket?

Well... It *is* sort of tuneable, as you implied earlier.
If you pass an IBodyProducer with a non-None length, Agent will send a
Content-Length header - not use chunked Transfer-Encoding.
FileBodyProducer doesn't know how to determine the length of a StringIO, so
you get chunked with this example. If you write the JSON to a regular file
and FileBodyProducer(open(the file)) you'll get a Content-Length request.
You could also write a new (trivial) IBodyProducer that does know how to
compute the length of a StringIO.
The documentation doesn't exactly spell this out - but the only reason
`length` is part of the interface is to be able to generate the
Content-Length header.
What would the ticket be? Expanded documentation to make this behavior
into an explicit guarantee of the interface? A new toggle somewhere to
force Agent.request into one mode or the other (regardless of the
performance consequences)?
And that stuff you said about proxies before was true ... So even if you
can control this in Agent, you're still not *guaranteed* the server will
see what you send.
Jean-Paul
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Glyph Lefkowitz]

Post by Glyph Lefkowitz
Gotcha. I guess what I meant was that you shouldn't care about this at the application level, but you're talking about an operational concern, not an application-level concern.
Perhaps this should be a tunable on Agent somehow. Can you file a ticket?
Well... It is sort of tuneable, as you implied earlier.
If you pass an IBodyProducer with a non-None length, Agent will send a Content-Length header - not use chunked Transfer-Encoding. FileBodyProducer doesn't know how to determine the length of a StringIO, so you get chunked with this example.

from twisted.web.client import FileBodyProducer
from io import BytesIO
from cStringIO import StringIO
FileBodyProducer(BytesIO(b"some bytes")).length

10L

Post by Glyph Lefkowitz

FileBodyProducer(StringIO(b"some bytes")).length

10

Post by Glyph Lefkowitz
If you write the JSON to a regular file and FileBodyProducer(open(the file)) you'll get a Content-Length request. You could also write a new (trivial) IBodyProducer that does know how to compute the length of a StringIO.
The documentation doesn't exactly spell this out - but the only reason `length` is part of the interface is to be able to generate the Content-Length header.
What would the ticket be? Expanded documentation to make this behavior into an explicit guarantee of the interface? A new toggle somewhere to force Agent.request into one mode or the other (regardless of the performance consequences)?

Rather than "regardless" of the consequences, perhaps just a maximum body size. If we made it an explicit guarantee in Agent, perhaps the interface change would not be in '.request' (which, as a formal interface used both inside and outside of Twisted, is fairly fixed), but rather a new `NonChunkedBodyProducer` concrete class that would implement IBodyProducer in terms of another IBodyProducer which either does or doesn't have a `length`. (Or a function that does same, always returning its argument if `length` is already set...)

Post by Glyph Lefkowitz
And that stuff you said about proxies before was true ... So even if you can control this in Agent, you're still not guaranteed the server will see what you send.
Thanks - that's exactly what I was looking for.
But note when using FileBodyProducer(StringIO(json.dumps(~blah))) -- I still see a content-length in the request header as it's received by twisted. Am I correct to assume the request is agnostic...meaning it's shaped the same for twisted as it is for apache.

It is shaped the same. The reason you're seeing the error is due to the issue I pointed out above.

Post by Glyph Lefkowitz
Headers({'host': ['localhost:8080'], 'connection': ['close'], 'content-length': ['671'], 'user-agent': ['PassengerTest']})

-glyph

[Kevin Mcintyre]

"It is shaped the same. The reason you're seeing the error is due to the
issue I pointed out above."

Just to be clear you mean that proxies can reshape the call in flight?

Post by Kevin Mcintyre
On Mon, Jan 30, 2017 at 4:59 PM, Jean-Paul Calderone <

Post by Jean-Paul Calderone

Post by Glyph Lefkowitz
Gotcha. I guess what I meant was that you shouldn't care about this at
the application level, but you're talking about an operational concern, not
an application-level concern.
Perhaps this should be a tunable on Agent somehow. Can you file a ticket?

Well... It *is* sort of tuneable, as you implied earlier.
If you pass an IBodyProducer with a non-None length, Agent will send a
Content-Length header - not use chunked Transfer-Encoding.
FileBodyProducer doesn't know how to determine the length of a StringIO, so
you get chunked with this example.

Not quite true: FileBodyProducer *does* know how to compute the length of

Post by Jean-Paul Calderone

Post by Glyph Lefkowitz

from twisted.web.client import FileBodyProducer
from io import BytesIO
from cStringIO import StringIO
FileBodyProducer(BytesIO(b"some bytes")).length

10L

Post by Jean-Paul Calderone

Post by Glyph Lefkowitz

FileBodyProducer(StringIO(b"some bytes")).length

10
If you write the JSON to a regular file and FileBodyProducer(open(the

Post by Jean-Paul Calderone
file)) you'll get a Content-Length request. You could also write a new
(trivial) IBodyProducer that does know how to compute the length of a
StringIO.
The documentation doesn't exactly spell this out - but the only reason
`length` is part of the interface is to be able to generate the
Content-Length header.
What would the ticket be? Expanded documentation to make this behavior
into an explicit guarantee of the interface? A new toggle somewhere to
force Agent.request into one mode or the other (regardless of the
performance consequences)?

Rather than "regardless" of the consequences, perhaps just a maximum body
size. If we made it an explicit guarantee in Agent, perhaps the interface
change would not be in '.request' (which, as a formal interface used both
inside and outside of Twisted, is fairly fixed), but rather a new
`NonChunkedBodyProducer` concrete class that would implement IBodyProducer
in terms of another IBodyProducer which either does or doesn't have a
`length`. (Or a function that does same, always returning its argument if
`length` is already set...)
And that stuff you said about proxies before was true ... So even if you

Post by Jean-Paul Calderone
can control this in Agent, you're still not *guaranteed* the server will
see what you send.

Thanks - that's exactly what I was looking for.
*But note* when using FileBodyProducer(StringIO(json.dumps(~blah))) -- I
still see a content-length in the request header as it's received by
twisted. Am I correct to assume the request is agnostic...meaning it's
shaped the same for twisted as it is for apache.
It is shaped the same. The reason you're seeing the error is due to the
issue I pointed out above.
Headers({'host': ['localhost:8080'], 'connection': ['close'],
'content-length': ['671'], 'user-agent': ['PassengerTest']})
-glyph
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Glyph Lefkowitz]

"It is shaped the same. The reason you're seeing the error is due to the issue I pointed out above."
Just to be clear you mean that proxies can reshape the call in flight?

No, I mean that FileBodyObserver specifically calculates a length if it can, and it can for StringIO.

-glyph