Discussion:
[Twisted-Python] How to restart Twisted service in-process
Роман Мещеряков
2017-02-02 09:17:37 UTC
Permalink
_______________________________________________
Twisted-Python mailing list
Twisted-***@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Glyph Lefkowitz
2017-02-18 19:25:47 UTC
Permalink
Hello everyone! I wrote Twisted-based TCP server which is capable of running in several relatively different modes. When mode switch is needed, I would like the server to restart itself by some means, for it to read new mode from configuration file and create corresponding implementation. Also I wanted the restart to occur without re-running twistd process, because it seems to me more appropriate for Linux daemon.
Hi Roman,

Sorry it took a while to get to this one :).

If your solution is working for you, then it's fine. You've correctly understood the API associated with IServiceCollection and your implementation (at least at first glance, I don't have time to test it exhaustively :)) is correct.

However, there's a more philosophical question as well: if you really want to re-start from scratch, what exactly are you shutting down and starting up again? Wouldn't you want to be able to load new code? Simply stopping and starting objects won't re-initialize the process from the beginning, only from some indeterminate middle state where parts of the program are already set up; this doesn't have the usual desired effect of a "restart" where potentially unknown or buggy state is cleared away.

You may want to consider simply calling execv https://docs.python.org/2.7/library/os.html#os.execv <https://docs.python.org/2.7/library/os.html#os.execv> with [sys.executable] + sys.argv rather than trying to do this in terms of Twisted's APIs.

-glyph
Jean-Paul Calderone
2017-02-18 23:19:33 UTC
Permalink
*Hello everyone! I wrote Twisted-based TCP server which is capable of
running in several relatively different modes. When mode switch is needed,
I would like the server to restart itself by some means, for it to read new
mode from configuration file and create corresponding implementation. Also
I wanted the restart to occur without re-running twistd process, because it
seems to me more appropriate for Linux daemon.*
Hi Roman,
Sorry it took a while to get to this one :).
If your solution is working for you, then it's fine. You've correctly
understood the API associated with IServiceCollection and your
implementation (at least at first glance, I don't have time to test it
exhaustively :)) is correct.
However, there's a more philosophical question as well: if you really want
to *re-start* from scratch, what exactly are you shutting down and
starting up again? Wouldn't you want to be able to load new code? Simply
stopping and starting objects won't re-initialize the process from the
beginning, only from some indeterminate middle state where parts of the
program are already set up; this doesn't have the usual desired effect of a
"restart" where potentially unknown or buggy state is cleared away.
You may want to consider simply calling execv https://docs.python.org/
2.7/library/os.html#os.execv with [sys.executable] + sys.argv rather than
trying to do this in terms of Twisted's APIs.
Or better yet. Don't be afraid to just exit. Your process is being
managed by a supervisor with a rich feature set - like easily controlled
re-restart behavior, right (If not, what do you do when your daemon simply
*crashes*)?

Jean-Paul
Glyph Lefkowitz
2017-02-18 23:32:12 UTC
Permalink
(If not, what do you do when your daemon simply crashes)?
Twisted programs can crash!??!

-g
Jean-Paul Calderone
2017-02-18 23:33:41 UTC
Permalink
On Feb 18, 2017, at 3:19 PM, Jean-Paul Calderone <
(If not, what do you do when your daemon simply *crashes*)?
Twisted programs can crash!??!
The Linux OOM killer is pretty crazy, man.

Jean-Paul
-g
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Роман Мещеряков
2017-02-20 08:58:29 UTC
Permalink
_______________________________________________
Twisted-Python mailing list
Twisted-***@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Glyph Lefkowitz
2017-02-20 09:02:14 UTC
Permalink
Hi everyone,
The answers are below
--
Kind regards, Roman Mescheryakov
Post by Glyph Lefkowitz
Hello everyone! I wrote Twisted-based TCP server which is capable of running in several relatively different modes. When mode switch is needed, I would like the server to restart itself by some means, for it to read new mode from configuration file and create corresponding implementation. Also I wanted the restart to occur without re-running twistd process, because it seems to me more appropriate for Linux daemon.
Hi Roman,
Sorry it took a while to get to this one :).
If your solution is working for you, then it's fine. You've correctly understood the API associated with IServiceCollection and your implementation (at least at first glance, I don't have time to test it exhaustively :)) is correct.
However, there's a more philosophical question as well: if you really want to re-start from scratch, what exactly are you shutting down and starting up again? Wouldn't you want to be able to load new code? Simply stopping and starting objects won't re-initialize the process from the beginning, only from some indeterminate middle state where parts of the program are already set up; this doesn't have the usual desired effect of a "restart" where potentially unknown or buggy state is cleared away.
You may want to consider simply calling execv https://docs.python.org/2.7/library/os.html#os.execv <https://docs.python.org/2.7/library/os.html#os.execv> with [sys.executable] + sys.argv rather than trying to do this in terms of Twisted's APIs.
You are right, Glyph, using os.execv would be more simple, reliable and straightforward way of achieving my goal. The only thing that bothers me due to lack of Linux experience: is it OK to os.execv inside daemon process? Because I run my process using twistd, not python directly.
If you're actually daemonizing ('twistd' without '-n') then there might be a small issue. However, a modern best practice is to never daemonize; this is why the new `twist` runner (which you should probably switch to) doesn't daemonize and expects your supervisor to handle backgrounding.
Post by Glyph Lefkowitz
Or better yet. Don't be afraid to just exit. Your process is being managed by a supervisor with a rich feature set - like easily controlled re-restart behavior, right (If not, what do you do when your daemon simply crashes)?
Well, in fact I'm in the development stage now and did't have time to deal with this question yet :) As of now I understand very little about daemons and their supervisors (I belive it is systemd for Raspbian...).
Yes, systemd will do this.
Jean-Paul and Glyph, thank you for guiding me in the right direction! :)
Happy to help!

-g
Post by Glyph Lefkowitz
Jean-Paul
,
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python <http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python <http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
Роман Мещеряков
2017-02-28 16:11:16 UTC
Permalink
_______________________________________________
Twisted-Python mailing list
Twisted-***@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Glyph Lefkowitz
2017-03-01 02:52:16 UTC
Permalink
Hello everyone,
2017-02-28T18:39:47+0300 [ready_to_setup#debug] Before os.execv: sys.executable is /usr/bin/python and sys.argv are ['/usr/local/bin/twistd', '-ny', 'master_player.tac']
Another twistd server is running, PID 1939
This could either be a previously started instance of your application or a
different application entirely. To start a new one, either run it in some other
directory, or use the --pidfile and --logfile parameters to avoid clashes.
PYTHONPATH=. twistd -ny master_player.tac
(i.e. without daemonization)
d = task.deferLater(self._reactor, seconds,
shared_code.restart_script, self._log, self._reactor)
# _restart_script_done should never be called
d.addCallback(self._restart_script_done).addErrback(self._restart_script_failure)
logger.info("Restarting script...")
reactor.stop()
sys.stdout.flush()
#TODO: force logging system to flush all log files
logger.debug("Before os.execv: sys.executable is {sysexec} and sys.argv are {argv}",
sysexec = sys.executable, argv = sys.argv)
os.execv(sys.executable, [sys.executable] + sys.argv)
Could someone explain to me why does the "Another twistd server is running" error occur and how can I avoid it?
twistd writes a .pid file and then attempts to check whether another process on the system has the same PID contained in that file while starting up, to avoid conflicts.

Yours is an odd case, because the process with that process-ID is the _current_ process, and perhaps the check ought to even be modified to account for this.

Two other options you have are:

switch to `twist`, which doesn't write a pidfile
use the option "--pidfile ''" to disable writing a pidfile.

I strongly suggest the first option :).

-glyph
Роман Мещеряков
2017-03-01 17:46:29 UTC
Permalink
_______________________________________________
Twisted-Python mailing list
Twisted-***@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Glyph Lefkowitz
2017-03-01 21:09:23 UTC
Permalink
Then the next question is: how to run my .tac file with twist? I don't see option like '-y' in it's --help output.
We haven't implemented .tac files for `twist` yet; you can either add that feature :) or you can write a twist plugin. (It's the same as a twistd plugin.)

-glyph
Роман Мещеряков
2017-03-02 09:08:12 UTC
Permalink
_______________________________________________
Twisted-Python mailing list
Twisted-***@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Loading...