[Twisted-Python] Porting twisted.spread.banana to py3

[Jørn Lomax]

I'm a gsoc student working on the buildbot project, porting it to py3.
Part of this is looking at porting the dependencies buildbot has in
twisted and try to port them.

twisted.spread is the most used twisted dependency, so I have started
working on this. I started working on banana.py and I have encountered a
question. In dataRecieved (banana.py:174). A chunk is added to the
buffer. On python3, how should the distinction be made between string
and bytes. Buffer seems to be a string, the chunk seems to be bytes, and
sometimes opposite.

Any help would be appreciated

regards,
Jørn Lomax

[Wolfgang Rohdewald]

Post by Jørn Lomax
I'm a gsoc student working on the buildbot project, porting it to py3.
Part of this is looking at porting the dependencies buildbot has in
twisted and try to port them.
twisted.spread is the most used twisted dependency, so I have started
working on this.

I would much appreciate seeing spread ported.

Last year I actually already did (all tests passed) but then I did not find
the time to make separate tickets to be reviewed.

Actually porting spread is easier than getting the source into something
acceptable by the core developers.

Post by Jørn Lomax

there now is https://github.com/wrohdewald/twisted.git
branch spread-py3-7598

If I would continue with this now, I would probably first rebase that branch
onto master (not sure if the twisted project still uses svn or if the
migration to git has already been done).

You could also look for tickets opened by myself (wolfgang61), most
of them should be about porting spread.

Post by Jørn Lomax
I started working on banana.py and I have encountered a
question. In dataRecieved (banana.py:174). A chunk is added to the
buffer. On python3, how should the distinction be made between string
and bytes. Buffer seems to be a string, the chunk seems to be bytes, and
sometimes opposite.

I believe that chunk is a banana-encoded string, so there should be no
need to treat it as bytes.

[Itamar Turner-Trauring]

I'm a gsoc student working on the buildbot project, porting it to py3. Part of this is looking at porting the dependencies buildbot has in twisted and try to port them.
twisted.spread is the most used twisted dependency, so I have started working on this. I started working on banana.py and I have encountered a question. In dataRecieved (banana.py:174). A chunk is added to the buffer. On python3, how should the distinction be made between string and bytes. Buffer seems to be a string, the chunk seems to be bytes, and sometimes opposite.

Banana calls bytes "string". So if you're using banana there is only
bytes support. Python 3 vs 2 doesn't matter, it's always going to be
bytes.

[Jørn Lomax]

Thank you very much. That is the exact answer I wanted :)

I'm guessing maybe the tests need updating, because they seem to send
use unicode data, which of course didn't make any difference in py2, but
now it does.

Post by Itamar Turner-Trauring

Post by Jørn Lomax
I'm a gsoc student working on the buildbot project, porting it to
py3. Part of this is looking at porting the dependencies buildbot has
in twisted and try to port them.
twisted.spread is the most used twisted dependency, so I have started
working on this. I started working on banana.py and I have
encountered a question. In dataRecieved (banana.py:174). A chunk is
added to the buffer. On python3, how should the distinction be made
between string and bytes. Buffer seems to be a string, the chunk
seems to be bytes, and sometimes opposite.

Banana calls bytes "string". So if you're using banana there is only
bytes support. Python 3 vs 2 doesn't matter, it's always going to be
bytes.
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Glyph]

Post by Jørn Lomax
Thank you very much. That is the exact answer I wanted :)
I'm guessing maybe the tests need updating, because they seem to send use unicode data, which of course didn't make any difference in py2, but now it does.

We may actually need to change the wire protocol to allow some applications to continue to work on py3, to allow for a 'text' type that corresponds to Python unicode. This should probably be done after the initial port though.

-glyph

[Itamar Turner-Trauring]

We may actually need to change the wire protocol to allow some applications to continue to work on py3, to allow for a 'text' type that corresponds to Python unicode. This should probably be done after the initial port though.

Really? That seems like Jelly's job, not Banana's.

[Glyph]

Post by Itamar Turner-Trauring

Post by Glyph
We may actually need to change the wire protocol to allow some applications to continue to work on py3, to allow for a 'text' type that corresponds to Python unicode. This should probably be done after the initial port though.

Really? That seems like Jelly's job, not Banana's.

Yes, you're right.

-glyph

[Glyph Lefkowitz]

Post by Glyph

Post by Itamar Turner-Trauring

Post by Glyph
We may actually need to change the wire protocol to allow some applications to continue to work on py3, to allow for a 'text' type that corresponds to Python unicode. This should probably be done after the initial port though.

Really? That seems like Jelly's job, not Banana's.

Yes, you're right.

Furthermore (I thought I remembered this!) Jelly already has support for unicode, so nothing will need to change :).

-glyph

[Jørn Lomax]

I have found another issue I'm wondering a little how to handle. There
are places in the bana_test file that sys.maxint is used. This no longer
exists in python3, so how should it be handled. Should we make an
artificial maxint to use for python3. I might want to note that it is
not used for comparison, but to create an integer over a given size:

foo = [1, 2, [3, 4], [30.5, 40.2], 5,
[b"six", b"seven", [b"eight", 9]], [10],
# TODO: currently the C implementation's a bit buggy...
sys.maxint * long(3), sys.maxint * long(2), sys.maxint *
long(-2)]

regards,
JÞrn lomax

Post by Glyph Lefkowitz

Post by Glyph

On Jul 2, 2015, at 6:13 AM, Itamar Turner-Trauring

Post by Glyph
We may actually need to change the wire protocol to allow some
applications to continue to work on py3, to allow for a 'text' type
that corresponds to Python unicode. This should probably be done
after the initial port though.

Really? That seems like Jelly's job, not Banana's.

Yes, you're right.

Furthermore (I thought I remembered this!) Jelly already has support
for unicode, so nothing will need to change :).
-glyph
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Glyph]

Post by Jørn Lomax
foo = [1, 2, [3, 4], [30.5, 40.2], 5,
[b"six", b"seven", [b"eight", 9]], [10],
# TODO: currently the C implementation's a bit buggy...
sys.maxint * long(3), sys.maxint * long(2), sys.maxint * long(-2)]
regards,
Jørn lomax

These tests are just making sure that it's capable of serializing things that don't fit into a machine integer; sys.maxsize should be an acceptable substitute.

-glyph

[Wolfgang Rohdewald]

Post by Jørn Lomax
I have found another issue I'm wondering a little how to handle. There
are places in the bana_test file that sys.maxint is used. This no longer
exists in python3, so how should it be handled. Should we make an
artificial maxint to use for python3. I might want to note that it is

i5:~/tmp/twisted/twisted/test (spread-py3-7598) git show a0d35fa
commit a0d35fa912b055bd65fb76b597de1d64675711d8
Author: Wolfgang Rohdewald <***@rohdewald.de>
Date: Sun Nov 2 18:50:54 2014 +0100

twisted.test.test_banana: PY3 has no sys.maxint

diff --git a/twisted/test/test_banana.py b/twisted/test/test_banana.py
index e0504d0..c43bdc8 100644
--- a/twisted/test/test_banana.py
+++ b/twisted/test/test_banana.py
@@ -455,10 +455,11 @@ class BananaTestCase(BananaTestBase):
Test feeding the data byte per byte to the receiver. Normally
data is not split.
"""
+ maxint = sys.maxsize if _PY3 else sys.maxint
foo = [1, 2, [3, 4], [30.5, 40.2], 5,
[b"six", b"seven", [b"eight", 9]], [10],
# TODO: currently the C implementation's a bit buggy...
- sys.maxint * 3l, sys.maxint * 2l, sys.maxint * long(-2)]
+ maxint * long(3), maxint * long(2), maxint * long(-2)]
self.enc.sendEncoded(foo)
self.feed(self.io.getvalue())
assert self.result == foo, "%s!=%s" % (repr(self.result), repr(foo))