[Twisted-Python] treq POST abborting with: err: ('Could not adapt', '{"....", "..."} <InterfaceClass twisted.web.iweb.IBodyProducer>)

Discussion:

steven meiers

2017-01-06 19:03:47 UTC

Permalink

hi,

sending a simple post request with a body (cookies) results in a error
here.

im using python 3.x with the latest twisted and treq installed via pip
in a virtualenv.

the code gets a chatroom login page, extracts two values, gets the
cookies and logs a user into the chat. (incomplete)
thing is for being logged in i need to send my current phpsessionid in
the post request...

glyph (without seeing the the code or backtrace) suggested that:
"I think the issue might be that it doesn't support bytes() on py3,
it's registered against str() or something"

here is my code:

from myhttp import getLoginPage, login
from twisted.internet import reactor

def myPrint(result):
    print("result:", result)

def myError(failure):
    print("failure:", failure)

def lala():
    #d = getLoginPage()
    d = login()
    d.addCallback(myPrint)
    d.addErrback(myError)

if __name__ == '__main__':
    lala()
    reactor.run()

from myhttp import getLoginPage, login
from twisted.internet import reactor

def myPrint(result):
    print("result:", result)

def myError(failure):
    print("failure:", failure)

def lala():
    #d = getLoginPage()
    d = login()
    d.addCallback(myPrint)
    d.addErrback(myError)

if __name__ == '__main__':
    lala()
    reactor.run()
(venv) ***@t560:~/code/twisted/webchat-client$ cat myhttp.py
from bs4 import BeautifulSoup
import random
from twisted.internet import reactor, defer
from twisted.web.client import getPage
import treq
import urllib

url = b'http://chatroom2000.de'
useragent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36".encode('utf-8')

cookies = {}

@defer.inlineCallbacks
def getLoginPage():
    global cookies
    response = yield treq.get(url)
    final_page = yield response.content()
    cookies = response.cookies()
    defer.returnValue(final_page)

@defer.inlineCallbacks
def login():
    response = yield getLoginPage()

    print(cookies)
    print("rsponse:", response)
    soup = BeautifulSoup(response, "lxml")
    hmac = soup.input["name"]
    key = soup.input["value"]
    print("hmac:key", hmac,key)
    username_list = ["Demetrius", "Tyrone", "Marshawn"]
    username = random.choice(username_list)
    body = urllib.parse.urlencode({hmac: key,
        'username': username, 'pw':'', 'gender':'m', 'aaaa':''})

    target_url = url + b"/?CheckUserName"


    headers = {'Content-Type': 'application/x-www-form-
urlencoded','cache-control': ['no-store, no-cache, must-revalidate,
pre-check=0, post-check=0, max-age=0']}

    import json
    try:
        print("cookies2:", cookies)
        somepage = yield treq.post(target_url, json.dumps({hmac: key,
"username": username, 'pw':'', 'gender':'m', 'aaaa':''}),
headers=headers, cookies=cookies )
    except Exception as err:
        print("err:", err)
    response = yield somepage.content()
    defer.returnValue(response)

if __name__ == '__main__':
    #url = 'http://chatroom2000.de'
    getPage(url)

    reactor.run()

and the error:
cookies2: <RequestsCookieJar[<Cookie
PHPSESSID=jevr00m25j0clfv6aibbgbro26 for chatroom2000.de/>, <Cookie
db1_cookie_test=1483729313 for chatroom2000.de/>]>
err: ('Could not adapt', '{"pw": "", "aaaa": "", "username":
"Demetrius", "cd5a6e4774acfd7e090467c9ed67f7e6": "1483729313",
"gender": "m"}', <InterfaceClass twisted.web.iweb.IBodyProducer>)
failure: [Failure instance: Traceback: <class 'UnboundLocalError'>:
local variable 'somepage' referenced before assignment
/home/julius/code/twisted/webchat-client/venv/lib/python3.5/site-
packages/twisted/internet/defer.py:457:callback
/home/julius/code/twisted/webchat-client/venv/lib/python3.5/site-
packages/twisted/internet/defer.py:565:_startRunCallbacks
/home/julius/code/twisted/webchat-client/venv/lib/python3.5/site-
packages/twisted/internet/defer.py:651:_runCallbacks
/home/julius/code/twisted/webchat-client/venv/lib/python3.5/site-
packages/twisted/internet/defer.py:1355:gotResult
--- <exception caught here> ---
/home/julius/code/twisted/webchat-client/venv/lib/python3.5/site-
packages/twisted/internet/defer.py:1299:_inlineCallbacks
/home/julius/code/twisted/webchat-client/myhttp.py:54:login

i followd some files in the traceback, but got nowhere :(

steven meiers

2017-01-06 19:03:39 UTC

Permalink

Tristan Seligmann

2017-01-06 23:36:24 UTC

Permalink

Post by steven meiers
"I think the issue might be that it doesn't support bytes() on py3,
it's registered against str() or something"

Actually, the problem is the reverse of this; only bytes is supported, but
you're passing str (unicode) in.

Post by steven meiers
somepage = yield treq.post(target_url, json.dumps({hmac: key,
"username": username, 'pw':'', 'gender':'m', 'aaaa':''}),

json.dumps returns str (ie. unicode) on py3; throwing in a .encode('utf-8')
here is probably what you want.

Glyph Lefkowitz

2017-01-07 01:22:37 UTC

Permalink

Post by steven meiers
"I think the issue might be that it doesn't support bytes() on py3,
it's registered against str() or something"
Actually, the problem is the reverse of this; only bytes is supported, but you're passing str (unicode) in.

Aah, thanks for pointing this out.

Post by steven meiers
somepage = yield treq.post(target_url, json.dumps({hmac: key,
"username": username, 'pw':'', 'gender':'m', 'aaaa':''}),
json.dumps returns str (ie. unicode) on py3; throwing in a .encode('utf-8') here is probably what you want.

Maybe we should support unicode for the body as well. We can set the charset in the mime-type and everything so that it will be properly intelligible by the server, which doesn't happen if the user manually encodes like this.

Even if we do this though - is there any way to convince json.dumps to behave consistently between 2/3 for the purposes of examples?

-glyph

Tristan Seligmann

2017-01-07 02:14:14 UTC

Permalink

Post by Glyph Lefkowitz
Even if we do this though - is there any way to convince json.dumps to
behave consistently between 2/3 for the purposes of examples?

As far as I know, the only way to handle this is with an if PY3 or if
isinstance(âŠ). I don't know of a way to make json.dumps on Python 2
consistently return unicode, and I don't know of a way to make json.dumps
on Python 3 return bytes.

Tristan Seligmann

2017-01-07 02:18:16 UTC

Permalink

Oh, forgot to comment on this point; in the specific case of JSON, it isn't
necessary to specify UTF-8 in Content-Type[1], but for HTML or XML it's a
pretty good idea. However, I'm not sure if it's possible to modify
Content-Type in a generic fashion to make this sort of thing work; for
example, "Content-Type: application/octet-stream; charset=UTF-8" is
nonsense. I'll defer to some HTTP experts here ;)

[1] https://tools.ietf.org/html/rfc7159#section-8.1

Cory Benfield

2017-01-07 12:00:06 UTC

Permalink

Post by Glyph Lefkowitz
Maybe we should support unicode for the body as well. We can set the charset in the mime-type and everything so that it will be properly intelligible by the server, which doesn't happen if the user manually encodes like this.
Oh, forgot to comment on this point; in the specific case of JSON, it isn't necessary to specify UTF-8 in Content-Type[1], but for HTML or XML it's a pretty good idea. However, I'm not sure if it's possible to modify Content-Type in a generic fashion to make this sort of thing work; for example, "Content-Type: application/octet-stream; charset=UTF-8" is nonsense. I'll defer to some HTTP experts here ;)
Note: No "charset" parameter is defined for this registration. Adding one really has no effect on compliant recipients.

steven meiers

2017-01-07 16:21:00 UTC

Permalink

thanks for the quick replys guys, you were right. a .encode('utf-8')
helps.

actually the json.dumps part in the code is wrong.
it has to be:
somepage = yield treq.post(target_url, body.encode('utf-8'),
headers=headers, cookies=cookies )

Maybe we should support unicode for the body as well. We can set
the charset in the mime-type and everything so that it will be
properly intelligible by the server, which doesn't happen if the
user manually encodes like this.
Oh, forgot to comment on this point; in the specific case of JSON,
it isn't necessary to specify UTF-8 in Content-Type[1], but for
HTML or XML it's a pretty good idea. However, I'm not sure if it's
possible to modify Content-Type in a generic fashion to make this
sort of thing work; for example, "Content-Type: application/octet-
stream; charset=UTF-8" is nonsense. I'll defer to some HTTP experts
here ;)

This is really not simple, for the reason that many MIME types do not
define a charset extension. In the case of JSON, it’s not just not
necessary to specify UTF-8 in Content-Type, but the standard

Note: No "charset" parameter is defined for this registration.
Adding one really has no effect on compliant recipients.

Strictly a completely compliant implementation would not emit charset
details for content types that have no charset registration. Such a
thing is pretty tricky to do. Knowing that, it’s probably best to
YOLO your way though, or forbid unicode in bodies.
Cory
[0]: https://tools.ietf.org/html/rfc7159#section-11
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Glyph Lefkowitz

2017-01-07 23:57:34 UTC

Permalink

Post by steven meiers
thanks for the quick replys guys, you were right. a .encode('utf-8')
helps.

Glad you were able to get it working!

-g

Glyph Lefkowitz

2017-01-07 23:57:15 UTC

Permalink

I see you your standards pedantry, and raise you!

MIME defines content-type to always have a charset:

https://tools.ietf.org/html/rfc1521#section-4 <https://tools.ietf.org/html/rfc1521#section-4>

Post by Glyph Lefkowitz

Among the defined parameters is a "charset" parameter by which the character set used in the body may be declared.

and from the spec you're citing,

Post by Glyph Lefkowitz

Adding one really has no effect on compliant recipients

Given that we'd always choose utf-8 anyway it would be fine.

Post by Glyph Lefkowitz
Note: No "charset" parameter is defined for this registration. Adding one really has no effect on compliant recipients.

A bigger part of the problem here is that treq has no way of knowing that the characters you're sending are JSON at all, let alone what encoding you want them in. So if we actually want to do MIME stuff, we'd probably want to have treq be the layer to call treq.dumps on the dict anyway, so that it knows what it's dealing with. Automatic treatment of unicode would likely set the type to text/plain;charset=utf8.

-glyph

Tristan Seligmann

2017-01-08 08:52:20 UTC

Permalink

Post by Glyph Lefkowitz
I see you your standards pedantry, and raise you!
*https://tools.ietf.org/html/rfc1521#section-4
<https://tools.ietf.org/html/rfc1521#section-4>*

RFC 1521 is obsoleted by RFC 2045 which has more nuanced semantics:
https://tools.ietf.org/html/rfc2045#section-5

But actually, HTTP is not MIME; so we should be looking here instead:
https://tools.ietf.org/html/rfc7231#section-3.1.1.1

Glyph Lefkowitz

2017-01-09 02:49:13 UTC

Permalink

Post by steven meiers
hi,
sending a simple post request with a body (cookies) results in a error
here.
im using python 3.x with the latest twisted and treq installed via pip
in a virtualenv.

Something that we discussed on IRC but did not come up in the mailing list thread may be the one issue that *should* be fixed in treq:

Because of this issue, this example - http://treq.readthedocs.io/en/latest/#post <http://treq.readthedocs.io/en/latest/#post> - which the OP was following, doesn't work on Python 3.

We should fix the docs so that it does.

-glyph

steven meiers

2017-01-10 01:20:21 UTC

Permalink

hi,

i use twisted to send a web request over a proxy, this works over
treq.get(url, agent=myagent) without problems.

but in the process (started out with agent.request) i could not figure
out from what line of twisted code this error:

[Failure instance: Traceback (failure with no frames): <class
'twisted.web._newclient.RequestGenerationFailed'>:
[<twisted.python.failure.Failure builtins.TypeError: sequence item 0:
expected a bytes-like object, str found>]

came from.

btw, is there a way to have a debugger attached to twisted code that
shows what code is currently executed and has a step forward option?

the error does come up when you give agent.request a "GET" instead of a
b"GET".

heres the test code:
from twisted.python.log import err
from twisted.web.client import ProxyAgent
from twisted.internet import reactor, defer
from twisted.internet.endpoints import TCP4ClientEndpoint
import treq

def display(response):
    print("Received response")
    print(response.content)

def err(failure):
    print(failure)

def main():
    endpoint = TCP4ClientEndpoint(reactor, "223.25.102.186", 8080)
    agent = ProxyAgent(endpoint)
    d = agent.request("GET", bytes("http://somedomain.de", 'utf-8'))
    d.addCallbacks(display)
    d.addErrback(err)

if __name__ == "__main__":
    main()
    reactor.run()

this code will produce this error:
[Failure instance: Traceback (failure with no frames): <class
'twisted.web._newclient.RequestGenerationFailed'>:
[<twisted.python.failure.Failure builtins.TypeError: sequence item 0:
expected a bytes-like object, str found>]

and some .py files with line numbers. none of which really are the
problem.

as it turns out, the "GET" as a string gets problematic here:
https://github.com/twisted/twisted/blob/twisted-16.6.0/src/twisted/web/
_newclient.py#L651

requestLines.append(b' '.join([self.method, self.uri,
                b'HTTP/1.1\r\n']))

wouldnt it be better to have something like this:

try:
            requestLines.append(b' '.join([self.method, self.uri,
                b'HTTP/1.1\r\n']))
        except TypeError as e:
            raise TypeError("could not join: " + str(self.method) + " "
+ str(self.uri) + ' ' + 'HTTP/1.1\r\n' )

which will produce:
<class 'twisted.python.failure.Failure'> [Failure instance: Traceback:
<class 'TypeError'>: could not join: GET b'http://somedomain.de'
HTTP/1.1

to give the user a chance to find out what he did wrong?
...when he uses agent instead of treq for whatever reason.

Glyph Lefkowitz

2017-01-10 07:12:00 UTC

Permalink

Post by steven meiers
the error does come up when you give agent.request a "GET" instead of a
b"GET".

That's expected; you do have to pass the method as `bytes`, as documented here: https://twistedmatrix.com/documents/16.6.0/api/twisted.web.iweb.IAgent.html#request <https://twistedmatrix.com/documents/16.6.0/api/twisted.web.iweb.IAgent.html#request>

-g

steven meiers

2017-01-10 23:19:57 UTC

Permalink

Post by Glyph Lefkowitz

Post by steven meiers
the error does come up when you give agent.request a "GET" instead of a
b"GET".

That's expected; you do have to pass the method as `bytes`, as
documented here: https://twistedmatrix.com/documents/16.6.0/api/twist
ed.web.iweb.IAgent.html#request

True, but since python 2.x is still shipped with debian testing for
example there will be some people running into this issue.
Just to make twisted more approachable this could he handled.

Tristan Seligmann

2017-01-11 04:01:39 UTC

Permalink

Post by steven meiers
True, but since python 2.x is still shipped with debian testing for
example there will be some people running into this issue.
Just to make twisted more approachable this could he handled.

The b'GET' syntax for byte strings works on Python 2.7 (you get a str) as
well as Python 3, so I would suggest we use it everywhere in the
documentation / examples to make it easier on programmers using both Python
2 and 3.

steven meiers

2017-01-10 23:19:38 UTC

Permalink

Post by Glyph Lefkowitz

Post by steven meiers
the error does come up when you give agent.request a "GET" instead of a
b"GET".

That's expected; you do have to pass the method as `bytes`, as
documented here: https://twistedmatrix.com/documents/16.6.0/api/twist
ed.web.iweb.IAgent.html#request

True, but since python 2.x is still shipped with debian testing for
example there will be some people running into this issue.
Just to make twisted more approachable this could he handled.

Glyph Lefkowitz

2017-01-12 00:49:15 UTC

Permalink

Post by steven meiers

Post by Glyph Lefkowitz

Post by steven meiers
the error does come up when you give agent.request a "GET" instead of a
b"GET".

That's expected; you do have to pass the method as `bytes`, as
documented here: https://twistedmatrix.com/documents/16.6.0/api/twist
ed.web.iweb.IAgent.html#request

True, but since python 2.x is still shipped with debian testing for
example there will be some people running into this issue.
Just to make twisted more approachable this could he handled.

I think that Tristan's suggestion is the best way to address this.

-glyph

Tom Most

2017-01-11 02:43:34 UTC

Permalink

Post by Glyph Lefkowitz

Post by steven meiers
hi,
sending a simple post request with a body (cookies) results in a error
here.
im using python 3.x with the latest twisted and treq installed via pip
in a virtualenv.

Something that we discussed on IRC but did not come up in the mailing
Because of this issue, this example
- http://treq.readthedocs.io/en/latest/#post - which the OP was
following, doesn't work on Python 3.
We should fix the docs so that it does.

I submitted a PR to fix this a while ago:
https://github.com/twisted/treq/pull/152

Also, the docs on RTD are still out of date. Who owns them? I'd be happy
to help.

---Tom

Glyph Lefkowitz

2017-01-12 00:44:10 UTC

Permalink

Post by steven meiers
hi,
sending a simple post request with a body (cookies) results in a error
here.
im using python 3.x with the latest twisted and treq installed via pip
in a virtualenv.

Because of this issue, this example - <http://treq.readthedocs.io/en/latest/#post>http://treq.readthedocs.io/en/latest/#post <http://treq.readthedocs.io/en/latest/#post> - which the OP was following, doesn't work on Python 3.
We should fix the docs so that it does.

I submitted a PR to fix this a while ago: https://github.com/twisted/treq/pull/152 <https://github.com/twisted/treq/pull/152>
Also, the docs on RTD are still out of date. Who owns them? I'd be happy to help.

Thanks, hopefully the next time my periodic review comes around I'll catch that one.

Looks like I own the docs. Builds are apparently failing - should I add you to the project?

-glyph

Tom Most

2017-01-12 06:05:21 UTC

Permalink

Sure, I'd be happy to help (but won't get to it until the weekend).
Username is twm.

Thanks,
Tom

Post by Glyph Lefkowitz

Post by Tom Most

Post by Glyph Lefkowitz

Post by steven meiers
hi,
sending a simple post request with a body (cookies) results in a error
here.
im using python 3.x with the latest twisted and treq installed via pip
in a virtualenv.

Something that we discussed on IRC but did not come up in the
Because of this issue, this example
- http://treq.readthedocs.io/en/latest/#post - which the OP was
following, doesn't work on Python 3.
We should fix the docs so that it does.

https://github.com/twisted/treq/pull/152
Also, the docs on RTD are still out of date. Who owns them? I'd be happy to help.

Thanks, hopefully the next time my periodic review comes around I'll catch that one.
Looks like I own the docs. Builds are apparently failing - should I
add you to the project?
-glyph
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Glyph Lefkowitz

2017-01-12 08:29:09 UTC

Permalink

Done.
-g

Sure, I'd be happy to help (but won't get to it until the weekend). Username is twm.
Thanks,
Tom

Post by Glyph Lefkowitz

Post by steven meiers
hi,
sending a simple post request with a body (cookies) results in a error
here.
im using python 3.x with the latest twisted and treq installed via pip
in a virtualenv.

I submitted a PR to fix this a while ago: <https://github.com/twisted/treq/pull/152>https://github.com/twisted/treq/pull/152 <https://github.com/twisted/treq/pull/152>
Also, the docs on RTD are still out of date. Who owns them? I'd be happy to help.

Thanks, hopefully the next time my periodic review comes around I'll catch that one.
Looks like I own the docs. Builds are apparently failing - should I add you to the project?
-glyph
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python <http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>

_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python