[Twisted-Python] Can't identify protocol

[Matilda Bernard]

Hi,

I am using Twisted 15.2.1 have been seeing a lot of half open TCP/IP
connections (lsof can't identify protocol) when I get a lot of traffic . Is
this something you have seen with twisted? I'm not really dealing with
socket connections at my code level so thought I'd ask if there is a known
issue as I am trying to figure out where the connections are being left
half open.

Thanks
Matilda

[Glyph Lefkowitz]

Hi,
I am using Twisted 15.2.1 have been seeing a lot of half open TCP/IP connections (lsof can't identify protocol) when I get a lot of traffic . Is this something you have seen with twisted? I'm not really dealing with socket connections at my code level so thought I'd ask if there is a known issue as I am trying to figure out where the connections are being left half open.

What do you mean by "half open"? What state is e.g. netstat showing them in?

-glyph

[Matilda Bernard]

So netstat doesn't show them but lsof does and our allocated file
descriptors keep increasing and the FDs have the description "can't
identify protocol"

Post by Matilda Bernard

Post by Matilda Bernard
Hi,
I am using Twisted 15.2.1 have been seeing a lot of half open TCP/IP

connections (lsof can't identify protocol) when I get a lot of traffic . Is
this something you have seen with twisted? I'm not really dealing with
socket connections at my code level so thought I'd ask if there is a known
issue as I am trying to figure out where the connections are being left
half open.
What do you mean by "half open"? What state is e.g. netstat showing them in?
-glyph
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Glyph Lefkowitz]

So netstat doesn't show them but lsof does and our allocated file descriptors keep increasing and the FDs have the description "can't identify protocol"

My guess would be that that means the file descriptor is open, but not connected to anything; which would mean that it's been left open so long that the OS has cleaned up the underlying network resources (perhaps via shutdown(2)?).

I've never seen this, personally, and my understanding of Twisted's transport mechanism is that it would be impossible unless the reactor were completely frozen for a very long period of time (4+ minutes, at least). What operating system and version are you running?

-glyph

[Matilda Bernard]

"I've never seen this, personally, and my understanding of Twisted's
transport mechanism is that it would be impossible unless the reactor were
completely frozen for a very long period of time (4+ minutes, at least)."

How can I detect that state?

We're using OEL 6.5 and PyPy 2.6.0

Regards
Matilda

Post by Matilda Bernard
So netstat doesn't show them but lsof does and our allocated file
descriptors keep increasing and the FDs have the description "can't
identify protocol"
My *guess* would be that that means the file descriptor is open, but not
connected to anything; which would mean that it's been left open so long
that the OS has cleaned up the underlying network resources (perhaps via
shutdown(2)?).
I've never seen this, personally, and my understanding of Twisted's
transport mechanism is that it would be impossible unless the reactor were
completely frozen for a very long period of time (4+ minutes, at least).
What operating system and version are you running?
-glyph
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Glyph Lefkowitz]

"I've never seen this, personally, and my understanding of Twisted's transport mechanism is that it would be impossible unless the reactor were completely frozen for a very long period of time (4+ minutes, at least)."
How can I detect that state?

Is the server completely unresponsive for multiple minutes at a time?

We're using OEL 6.5 and PyPy 2.6.0

You should upgrade to PyPy 4.0.1. (Generally speaking you should always run the latest available PyPy, since they only really do bugfix and performance releases; they don't break compatibility) I don't remember what version it was in, but somewhere in the 2.x series there was a nasty file descriptor leak whose symptoms might match what you're seeing.

-glyph

[Matilda Bernard]

Thanks Glyph for the possible causes of this problem.

Post by Matilda Bernard
"I've never seen this, personally, and my understanding of Twisted's
transport mechanism is that it would be impossible unless the reactor were
completely frozen for a very long period of time (4+ minutes, at least)."
How can I detect that state?
Is the server completely unresponsive for multiple minutes at a time?
We're using OEL 6.5 and PyPy 2.6.0
You should upgrade to PyPy 4.0.1. (Generally speaking you should always
run the latest available PyPy, since they only really do bugfix and
performance releases; they don't break compatibility) I don't remember
what version it was in, but somewhere in the 2.x series there was a nasty
file descriptor leak whose symptoms *might* match what you're seeing.
-glyph
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Glyph Lefkowitz]

Post by Matilda Bernard
Thanks Glyph for the possible causes of this problem.

I'm definitely curious if you manage to reach a resolution; please follow up with the list if you discover more information.

-glyph

[Matilda Bernard]

Will do.

Post by Matilda Bernard
Thanks Glyph for the possible causes of this problem.
I'm definitely curious if you manage to reach a resolution; please follow
up with the list if you discover more information.
-glyph