[Twisted-Python] Stdout from child of ProcessProtocol not propogating

[Jim Carroll]

We've been experimenting with creating an SSH virtual server for Windows
based on conch. We've made good process, but we've run into a problem
running child processes (children of spawnProcess()), in that stdout does
not make it back to the client.



I can demonstrate the problem using the example ptyserv.py script with a
slight change for Windows. Here's the code to demonstrate the problem:



from twisted.internet import reactor, protocol



class FakeTelnet(protocol.Protocol):



commandToRun = ['c:\\Windows\\System32\\cmd.exe']



def connectionMade(self):

print 'connection made'

self.propro = ProcessProtocol(self)

reactor.spawnProcess(self.propro, self.commandToRun[0],

self.commandToRun, os.environ)



def dataReceived(self, data):

self.propro.transport.write(data)



def conectionLost(self, reason):

print 'connection lost'

self.propro.tranport.loseConnection()



class ProcessProtocol(protocol.ProcessProtocol):



def __init__(self, pr):

self.pr = pr



def outReceived(self, data):

self.pr.transport.write(data)



def processEnded(self, reason):

print 'protocol connection lost'

self.pr.transport.loseConnection()



f = protocol.Factory()

f.protocol = FakeTelnet

reactor.listenTCP(5823, f)

reactor.run()





Run the above code on Windows, and telnet to localhost 5823.



After you get the CMD.exe prompt, try to launch python. This is what you'll

see:



Microsoft Windows [Version 6.3.9600]

(c) 2013 Microsoft Corporation. All rights reserved.



C:\Temp>python

python





That's it. Hitting enter just drops down to another next blank line.



Now what's interesting is that writes to stdin of the spawned CMD.exe get
there. You can see this by pressing <CTRL>-D. The python subprocess will
exit and you'll still be connected to your telnet session, now back at the
CMD.exe prompt.



I've experimented with modifying twisted.internet._dumbwin32proc.py so that
the DuplicateHandle() calls (lines 147 - 163) are not called, thinking that
inheritance was the cause, but no-go. I've also tried adding calls to
msvcrt.set_mode() to os.O_BINARY to make sure windows isn't somehow cooking
the child processes output, but that had no effect either.



I've been banging my head against this for a week and I've reached a stuck
point. I was wondering if someone could point me towards where I'm going
wrong. Any advice would be appreciated....



FYI - once we get this worked out, we'd be happy to share the code.



Thanks

Jim C.

[David Bolen]

Post by Jim Carroll
After you get the CMD.exe prompt, try to launch python. This is what you'll
Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.
C:\Temp>python
python
That's it. Hitting enter just drops down to another next blank line.

I suspect it's not a Twisted issue as much as a common risk running
programs in a child process under Windows, especially executables using
the standard C RTL with its default stdout buffering behavior.

Do the built-in commands like DIR work ok? If so, then it's likely that
it isn't that stdout isn't making it back to the client; rather it isn't
even making it to your parent Twisted process. Usually it's a buffering
or tty detection issue since a Windows child shell won't appear to be a
tty. I'll note that python behaves the same way you describe when run
from within a cmd shell under Cygwin's ssh server for example.

In python's case, the C RTL is going into fully buffered mode. You
should eventually get output once enough is generated (~8K) or the
process exits. You can't control how the child executable will buffer
from the parent unless the executable provides a way. So how you work
around the issue depends on the actual command being run.

Assuming this is the issue you're running into, using either "python -u"
(if you just need output as it occurs) or "python -i" (for interactive
use) should be a workaround. Or if you want to avoid clients having to
remember the options, you can install appropriate environment variables
(PYTHONUNBUFFERED or PYTHONINSPECT respectively) into the child process.
But that's Python specific - there's no general solution for all
executables, and there will be some for which you simply can't disable
the buffering.

The other thing you have to watch out for are executables you may run
that actually use the Win32 console API rather than interact with
stdin/stdout. Or certain methods of character inputs (like Python's
msvcrt.getch). Again, not a Twisted issue, but you can't interact with
that (unless you jump through some significant hoops). This shows up
often with password prompts (which want to prevent echoing), curses like
applications, or even in unexpected places for Y/N prompts in some
utilities (like SC's default help, or NET TIME /SET's confirmation).

Of course if you're just implementing the ssh server portion, you should
be able to pretty much punt on it and leave it up to the user to
determine how to best run whatever commands they are using. The
behavior with your shell should be similar to other remote shell
approaches.

-- David