[Twisted-Python] Looking for help dealing with ClientService reconnections

[Daniel Sutcliffe]

Hello all, my first post here - only been using Twisted for about a
month and am also a relative newcomer to Python but have been coding
professionally for 20+ years. I was attracted to Twisted and Python
for a particular project purely because after research it seemed to be
the best tool for the job, and have actually been enjoying both Python
and Twisted much more than I ever thought I would.

The project I am coding towards is creating a sensor data collection
gateway. First iteration needs are simply pulling data from ModBus TCP
slave PLCs and writing it to a MySQL database, but goals beyond that
are making the source of the data and its destination(s) very
flexible(pluggable). Therefore I am trying to create a good clean
architecture from the outset so as I iterate forwards I don't finish
up having to take too many steps backwards before heading forwards.

I am using pymodbus to pull the data which works well for my devices,
has a twisted async API, and have created more than a few prototypes
that demonstrate all works as I expect. Where I am a bit stalled is
getting to grips with a good architecture that fulfills my needs - my
intention is that the application that meets my first goal will be a
twistd plugin.

The new ClientService class seems like it will fit my needs very
closely but I am struggling with how to handle the reconnections... I
have been using the whenConnected() method to grab the Protocol for
the initial connection and then use a method of this to poll the
connected slave. When the connection is lost I get an errback from
this method's deferred which I use as a signal to abandon the Protocol
and call whenConnected() again... at this point I have an issue though
as the returned deferred immediately gives me a callback with the same
Protocol which has just lost its connection, and thus loop...

Before I got on this mailing list I posted this Q to stackoverflow
with some example code:
http://stackoverflow.com/q/37061807/3448214
but no solution or much attention there yet.

As I say there, I realize I have probably just made a bad pattern
choice for how to use this API, but I have not been able to work out a
better choice which seems clean and fits my needs/understanding well.
I have tried deriving my own Protocol/Factory and handling the polling
there but this seems to get really messy once I start to add code to
get the collected data to a destination at that level, involving
giving the Protocol too much knowledge of how the data is to be
handled.

Any advice, good patterns, or pointers to other projects which do
something similar is appreciated,
Cheers
/dan

[Kevin Conway]

I'm still working through your code example and trying to get a better
grasp of what, exactly, you're trying to implement as far as client
behavior is concerned. In the meantime, it sounds, on the surface, like you
are trying to implement a form of the ReconnectingClientFactory:
http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.ReconnectingClientFactory.html
.

Your client protocol must be factory aware and call the resetDelay method
on the factory when a connection is made, but the factory will handle
reconnecting and generating a new protocol instance. Is that similar to
what you are trying to accomplish? Maybe see also
http://twistedmatrix.com/documents/current/core/howto/clients.html#reconnection
for a quick example of using the reconnecting factory.

Post by Daniel Sutcliffe
Hello all, my first post here - only been using Twisted for about a
month and am also a relative newcomer to Python but have been coding
professionally for 20+ years. I was attracted to Twisted and Python
for a particular project purely because after research it seemed to be
the best tool for the job, and have actually been enjoying both Python
and Twisted much more than I ever thought I would.
The project I am coding towards is creating a sensor data collection
gateway. First iteration needs are simply pulling data from ModBus TCP
slave PLCs and writing it to a MySQL database, but goals beyond that
are making the source of the data and its destination(s) very
flexible(pluggable). Therefore I am trying to create a good clean
architecture from the outset so as I iterate forwards I don't finish
up having to take too many steps backwards before heading forwards.
I am using pymodbus to pull the data which works well for my devices,
has a twisted async API, and have created more than a few prototypes
that demonstrate all works as I expect. Where I am a bit stalled is
getting to grips with a good architecture that fulfills my needs - my
intention is that the application that meets my first goal will be a
twistd plugin.
The new ClientService class seems like it will fit my needs very
closely but I am struggling with how to handle the reconnections... I
have been using the whenConnected() method to grab the Protocol for
the initial connection and then use a method of this to poll the
connected slave. When the connection is lost I get an errback from
this method's deferred which I use as a signal to abandon the Protocol
and call whenConnected() again... at this point I have an issue though
as the returned deferred immediately gives me a callback with the same
Protocol which has just lost its connection, and thus loop...
Before I got on this mailing list I posted this Q to stackoverflow
http://stackoverflow.com/q/37061807/3448214
but no solution or much attention there yet.
As I say there, I realize I have probably just made a bad pattern
choice for how to use this API, but I have not been able to work out a
better choice which seems clean and fits my needs/understanding well.
I have tried deriving my own Protocol/Factory and handling the polling
there but this seems to get really messy once I start to add code to
get the collected data to a destination at that level, involving
giving the Protocol too much knowledge of how the data is to be
handled.
Any advice, good patterns, or pointers to other projects which do
something similar is appreciated,
Cheers
/dan
--
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Glyph]

I'm still working through your code example and trying to get a better grasp of what, exactly, you're trying to implement as far as client behavior is concerned. In the meantime, it sounds, on the surface, like you are trying to implement a form of the ReconnectingClientFactory: http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.ReconnectingClientFactory.html <http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.ReconnectingClientFactory.html>.

Just to clear up this point: ReconnectingClientFactory is the old, bad way of doing things, that only works with connectTCP/connectSSL/connectUNIX et. al., and not with endpoints. ClientService <https://twistedmatrix.com/documents/16.1.1/api/twisted.application.internet.ClientService.html> is the new, good way, only available since the latest release.

-glyph

[Kevin Conway]

Thanks, glyph, for the correction. Also, thanks for reminding me that I'm
still living in twisted's past...

Post by Kevin Conway
I'm still working through your code example and trying to get a better
grasp of what, exactly, you're trying to implement as far as client
behavior is concerned. In the meantime, it sounds, on the surface, like you
http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.ReconnectingClientFactory.html
.
Just to clear up this point: ReconnectingClientFactory is the old, bad way
of doing things, that only works with connectTCP/connectSSL/connectUNIX et.
al., and not with endpoints. ClientService <
https://twistedmatrix.com/documents/16.1.1/api/twisted.application.internet.ClientService.html>
is the new, good way, only available since the latest release.
-glyph
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Glyph]

Thanks, glyph, for the correction. Also, thanks for reminding me that I'm still living in twisted's past...

Just a reminder that there's always cool new stuff on the horizon :).

In addition to just working with endpoints, the interface to ClientService is also a bit more flexible and testable, since the retry policy is implemented by a completely separate object from the service itself.

-glyph

[Daniel Sutcliffe]

I'm still working through your code example and trying to get a better grasp
of what, exactly, you're trying to implement as far as client behavior is
concerned. In the meantime, it sounds, on the surface, like you are trying
http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.ReconnectingClientFactory.html.

pymodbus actually has a pymodbus.client.async.ModbusClientFactory
which is derived from
twisted.internet.protocol.ReconnectingClientFactory
http://pymodbus.readthedocs.io/en/latest/library/async-client.html

Your client protocol must be factory aware and call the resetDelay method on
the factory when a connection is made, but the factory will handle
reconnecting and generating a new protocol instance. Is that similar to what
you are trying to accomplish? Maybe see also
http://twistedmatrix.com/documents/current/core/howto/clients.html#reconnection
for a quick example of using the reconnecting factory.

I had built working tests using the docs you link to and understand
the way it works, but when Twisted 16.1 came out I decided that as I
was writing fresh code I probably ought to use the suggested new API.
After reading up and checking the source for ClientService it seemed
just what I was after; offering the advantages that Glyph mentioned,
and simplifying the amount of code I had to write (and maintain),
especially as my intention was to use twistd and thus Services.

It was only when I started to implement this that I felt my
inexperience with Twisted's ways of doing things was holding me back
and couldn't see a clear direction of how to build my architecture
around it.

Initially I used my own Protocol class derived from
pymodbus.client.async.ModbusClientProtocol that handled the polling
internally with a Factory that was aware of this class based on
ClientFactory (not ReconnectingClientFactory) that stored the
persistent info (poll rate, etc). This worked well with ClientService
and reconnections happened as expected but once I started to try and
evolve my code to consolidate (from many ClientServices polling
various PLCs) and write out the data (to MySQL) the model felt like
the focus was in the wrong place and that maybe I should be focussing
more on making a derivation of the ClientService have more control of
the polling and just using the Protocol provided by pymodbus as-is.

After hitting the problem I described below, I decided it was time to
ask those with more Twisted experience what the most appropriate way
to approach this problem was. Am I better off going back to doing
polling in Protocol with a Factory that provides ways to pump the data
back upstream to where it needs to be consolidated, and not rolling my
own ClientService to control it? or do I just need a better way of
getting my ClientService to be aware of the latest connection? Maybe
there's a pathway in between, or even something I haven't yet thought
of?

If anyone does have the time and desire to help me with this but feels
they need more info from me or background then please just ask, and I
will provide and be most appreciative :)

Cheers
/dan

Post by Daniel Sutcliffe
Hello all, my first post here - only been using Twisted for about a
month and am also a relative newcomer to Python but have been coding
professionally for 20+ years. I was attracted to Twisted and Python
for a particular project purely because after research it seemed to be
the best tool for the job, and have actually been enjoying both Python
and Twisted much more than I ever thought I would.
The project I am coding towards is creating a sensor data collection
gateway. First iteration needs are simply pulling data from ModBus TCP
slave PLCs and writing it to a MySQL database, but goals beyond that
are making the source of the data and its destination(s) very
flexible(pluggable). Therefore I am trying to create a good clean
architecture from the outset so as I iterate forwards I don't finish
up having to take too many steps backwards before heading forwards.
I am using pymodbus to pull the data which works well for my devices,
has a twisted async API, and have created more than a few prototypes
that demonstrate all works as I expect. Where I am a bit stalled is
getting to grips with a good architecture that fulfills my needs - my
intention is that the application that meets my first goal will be a
twistd plugin.
The new ClientService class seems like it will fit my needs very
closely but I am struggling with how to handle the reconnections... I
have been using the whenConnected() method to grab the Protocol for
the initial connection and then use a method of this to poll the
connected slave. When the connection is lost I get an errback from
this method's deferred which I use as a signal to abandon the Protocol
and call whenConnected() again... at this point I have an issue though
as the returned deferred immediately gives me a callback with the same
Protocol which has just lost its connection, and thus loop...
Before I got on this mailing list I posted this Q to stackoverflow
http://stackoverflow.com/q/37061807/3448214
but no solution or much attention there yet.
As I say there, I realize I have probably just made a bad pattern
choice for how to use this API, but I have not been able to work out a
better choice which seems clean and fits my needs/understanding well.
I have tried deriving my own Protocol/Factory and handling the polling
there but this seems to get really messy once I start to add code to
get the collected data to a destination at that level, involving
giving the Protocol too much knowledge of how the data is to be
handled.
Any advice, good patterns, or pointers to other projects which do
something similar is appreciated,

[Glyph]

Post by Daniel Sutcliffe
Hello all, my first post here - only been using Twisted for about a
month and am also a relative newcomer to Python but have been coding
professionally for 20+ years. I was attracted to Twisted and Python
for a particular project purely because after research it seemed to be
the best tool for the job, and have actually been enjoying both Python
and Twisted much more than I ever thought I would.
The project I am coding towards is creating a sensor data collection
gateway. First iteration needs are simply pulling data from ModBus TCP
slave PLCs and writing it to a MySQL database, but goals beyond that
are making the source of the data and its destination(s) very
flexible(pluggable). Therefore I am trying to create a good clean
architecture from the outset so as I iterate forwards I don't finish
up having to take too many steps backwards before heading forwards.
I am using pymodbus to pull the data which works well for my devices,
has a twisted async API, and have created more than a few prototypes
that demonstrate all works as I expect. Where I am a bit stalled is
getting to grips with a good architecture that fulfills my needs - my
intention is that the application that meets my first goal will be a
twistd plugin.
The new ClientService class seems like it will fit my needs very
closely but I am struggling with how to handle the reconnections... I
have been using the whenConnected() method to grab the Protocol for
the initial connection and then use a method of this to poll the
connected slave. When the connection is lost I get an errback from
this method's deferred which I use as a signal to abandon the Protocol
and call whenConnected() again... at this point I have an issue though
as the returned deferred immediately gives me a callback with the same
Protocol which has just lost its connection, and thus loop...

If you want a hook each time a new protocol is created, you're probably better off writing a wrapper protocol factory, and passing that to your ClientService, then doing any set-up work you want to do in your buildProtocol implementation, which delegates to the real, pymodbus implementation.

Post by Daniel Sutcliffe
Before I got on this mailing list I posted this Q to stackoverflow
http://stackoverflow.com/q/37061807/3448214
but no solution or much attention there yet.
As I say there, I realize I have probably just made a bad pattern
choice for how to use this API, but I have not been able to work out a
better choice which seems clean and fits my needs/understanding well.
I have tried deriving my own Protocol/Factory and handling the polling
there but this seems to get really messy once I start to add code to
get the collected data to a destination at that level, involving
giving the Protocol too much knowledge of how the data is to be
handled.

I am curious as to why you say that this is "messy".

Post by Daniel Sutcliffe
Any advice, good patterns, or pointers to other projects which do
something similar is appreciated,

I spent a while thinking about your question, and I'm sorry that I can't give a more thorough answer, but I think you need to be a bit more specific about what it is you don't like about your potential solution. It seems to me that having a delegating Factory, especially if all you need to do is set up some state on each Protocol that gets produced, should be sufficient...

-glyph

[Daniel Sutcliffe]

Thanks Glyph, I think you have given me a push back in the 'right'
direction - more thoughts and commentary embedded below if you, or
anyone else, has the time.

On May 6, 2016, at 10:19 AM, Daniel Sutcliffe <***@gmail.com> wrote:
[...]

Post by Glyph

Post by Daniel Sutcliffe
The new ClientService class seems like it will fit my needs very
closely but I am struggling with how to handle the reconnections... I
have been using the whenConnected() method to grab the Protocol for
the initial connection and then use a method of this to poll the
connected slave. When the connection is lost I get an errback from
this method's deferred which I use as a signal to abandon the Protocol
and call whenConnected() again... at this point I have an issue though
as the returned deferred immediately gives me a callback with the same
Protocol which has just lost its connection, and thus loop...

If you want a hook each time a new protocol is created, you're probably
better off writing a wrapper protocol factory, and passing that to your
ClientService, then doing any set-up work you want to do in your
buildProtocol implementation, which delegates to the real, pymodbus
implementation.

Understood, if this is the way the framework is intended to be used I
realize doing anything else is going to be fighting against the flow.

However, just to probe the situation I found myself in further, for
the sake probing broken code to see how it might be fixed:

Given the ClientService.whenConnected() method is intended to provide
access to my connected Protocol through the deferred it returns, is it
not a little unfriendly that this Protocol may turn out to be
disconnected? OK occasionally due to timing but for this to be a
possible condition which can loop with the same disconnected Protocol
returned until the ClientService has its _currentConnection set to
None, suggests to me that I can't safely use my Protocol from
whenConnected() for much other than as a notification the first
connection has occurred... but how do I avoid this?

I have looked at the source and it seems to me the fact that the
connection has been lost should bubble up to the ClientService through
a t.a.i._DisconnectFactory and t.a.i._ReconnectingProtocolProxy once
my Protocol's connectionLost() is called. My issue seems to be that I
errback on a Protocol method's deferred returned to code at or above
the ClientService level which gives up on that Protocol and calls
whenConnected() to get the next one, only the Protocol's
connectionLost() has yet to be called and then doesn't have chance to
because my code is looping around calling whenConnected() and getting
the same Protocol back. I hope that makes sense :-/

My Q on this is if I should be internally calling my Protocol's
connectionLost() so it can bubble up to the ClientService before I
errback on the Protocol method - whose responsibility is it to call
this?

Post by Glyph

Post by Daniel Sutcliffe
Before I got on this mailing list I posted this Q to stackoverflow
http://stackoverflow.com/q/37061807/3448214
but no solution or much attention there yet.
As I say there, I realize I have probably just made a bad pattern
choice for how to use this API, but I have not been able to work out a
better choice which seems clean and fits my needs/understanding well.
I have tried deriving my own Protocol/Factory and handling the polling
there but this seems to get really messy once I start to add code to
get the collected data to a destination at that level, involving
giving the Protocol too much knowledge of how the data is to be
handled.

I am curious as to why you say that this is "messy".

Honestly, this was just a gut feeling at the time, probably more
sourced in my implementation from lack of experience in using Twisted;
after reading around the subject, looking at many more examples, and
your advice, I think I am convinced I need to back to looking at my
own Protocol derived from the pymodbus one with a Factory that
contains the persistent config and access to an interface to pump the
polled data upstream.

Post by Glyph

Post by Daniel Sutcliffe
Any advice, good patterns, or pointers to other projects which do
something similar is appreciated,

I spent a while thinking about your question, and I'm sorry that I can't give
a more thorough answer, but I think you need to be a bit more specific
about what it is you don't like about your potential solution. It seems to me
that having a delegating Factory, especially if all you need to do is set up
some state on each Protocol that gets produced, should be sufficient...

Looking at the code again I think it just seemed to make sense to me
at the time to have something that is (or has) a ClientService be the
object I am calling a DataSource have more control over when it polls
that data and what it does with it - and not to have to make the
Protocol and its Factory aware of this at all. Does that make it any
clearer? I am happy to push ahead with building this into my
Factory/Protocol if that is more normal usage for Twisted as I am sure
there will be benefits of encapsulating it here beyond the other side
I was seeing during that moment of confusion.

Thanks for the advice, I think it was enough to nudge me in a
direction that will work better to get me going, and if with more
experience other usage makes more sense I can always refactor,
refactor, refactor :)
Cheers
/dan

[Glyph]

Post by Daniel Sutcliffe
Thanks Glyph, I think you have given me a push back in the 'right'
direction - more thoughts and commentary embedded below if you, or
anyone else, has the time.
[...]

Post by Glyph

Post by Daniel Sutcliffe
The new ClientService class seems like it will fit my needs very
closely but I am struggling with how to handle the reconnections... I
have been using the whenConnected() method to grab the Protocol for
the initial connection and then use a method of this to poll the
connected slave. When the connection is lost I get an errback from
this method's deferred which I use as a signal to abandon the Protocol
and call whenConnected() again... at this point I have an issue though
as the returned deferred immediately gives me a callback with the same
Protocol which has just lost its connection, and thus loop...

If you want a hook each time a new protocol is created, you're probably
better off writing a wrapper protocol factory, and passing that to your
ClientService, then doing any set-up work you want to do in your
buildProtocol implementation, which delegates to the real, pymodbus
implementation.

Understood, if this is the way the framework is intended to be used I
realize doing anything else is going to be fighting against the flow.
However, just to probe the situation I found myself in further, for
Given the ClientService.whenConnected() method is intended to provide
access to my connected Protocol through the deferred it returns, is it
not a little unfriendly that this Protocol may turn out to be
disconnected? OK occasionally due to timing but for this to be a
possible condition which can loop with the same disconnected Protocol
returned until the ClientService has its _currentConnection set to
None, suggests to me that I can't safely use my Protocol from
whenConnected() for much other than as a notification the first
connection has occurred... but how do I avoid this?

whenConnected() is not intended to be used for "give me each Protocol as it is instantiated so that state can be set up", it is intended for API clients which want to send a message to the current connection to just retrieve the current connection so they can call a method on it.

I'm not sure what you mean by "turn out to be disconnected". The physical reality of networking is that you might always encounter a transport which has been disconnected but which you haven't received notification of its disconnection yet.

Post by Daniel Sutcliffe
I have looked at the source and it seems to me the fact that the
connection has been lost should bubble up to the ClientService through
a t.a.i._DisconnectFactory and t.a.i._ReconnectingProtocolProxy once
my Protocol's connectionLost() is called. My issue seems to be that I
errback on a Protocol method's deferred returned to code at or above
the ClientService level which gives up on that Protocol and calls
whenConnected() to get the next one, only the Protocol's
connectionLost() has yet to be called and then doesn't have chance to
because my code is looping around calling whenConnected() and getting
the same Protocol back. I hope that makes sense :-/

Let me try to rephrase: you call a protocol method which returns a Deferred; you add an errback to that Deferred which calls whenConnected() to re-try, but since the protocol hasn't disconnected yet, you get the same protocol instance back, which is useless to you.

Post by Daniel Sutcliffe
My Q on this is if I should be internally calling my Protocol's
connectionLost() so it can bubble up to the ClientService before I
errback on the Protocol method - whose responsibility is it to call
this?

It's the framework's responsibility to call it. You should not call it yourself. Your Protocol's connectionLost isn't going to bubble up to ClientService anyway; you'd have to call your wrapper's connectionLost, which would confuse its internal state, since the framework would call it again right afterwards, and we definitely don't have test coverage for that, since the framework will normally only call it once.

The right way to handle this would be to introduce a delay between re-tries. It's generally a good idea to have such a delay for lots of reasons; you don't want to overload your peer in the case of a transient failure. As a bonus, the fact that you've gone back up to the reactor loop to wait a while means that the transport will be properly disconnected and whenConnected() will do what you want.

Post by Daniel Sutcliffe

Post by Glyph

Post by Daniel Sutcliffe
Before I got on this mailing list I posted this Q to stackoverflow
http://stackoverflow.com/q/37061807/3448214
but no solution or much attention there yet.
As I say there, I realize I have probably just made a bad pattern
choice for how to use this API, but I have not been able to work out a
better choice which seems clean and fits my needs/understanding well.
I have tried deriving my own Protocol/Factory and handling the polling
there but this seems to get really messy once I start to add code to
get the collected data to a destination at that level, involving
giving the Protocol too much knowledge of how the data is to be
handled.

I am curious as to why you say that this is "messy".

Honestly, this was just a gut feeling at the time, probably more
sourced in my implementation from lack of experience in using Twisted;
after reading around the subject, looking at many more examples, and
your advice, I think I am convinced I need to back to looking at my
own Protocol derived from the pymodbus one with a Factory that
contains the persistent config and access to an interface to pump the
polled data upstream.

Post by Glyph

Post by Daniel Sutcliffe
Any advice, good patterns, or pointers to other projects which do
something similar is appreciated,

I spent a while thinking about your question, and I'm sorry that I can't give
a more thorough answer, but I think you need to be a bit more specific
about what it is you don't like about your potential solution. It seems to me
that having a delegating Factory, especially if all you need to do is set up
some state on each Protocol that gets produced, should be sufficient...

Looking at the code again I think it just seemed to make sense to me
at the time to have something that is (or has) a ClientService be the
object I am calling a DataSource have more control over when it polls
that data and what it does with it - and not to have to make the
Protocol and its Factory aware of this at all. Does that make it any
clearer? I am happy to push ahead with building this into my
Factory/Protocol if that is more normal usage for Twisted as I am sure
there will be benefits of encapsulating it here beyond the other side
I was seeing during that moment of confusion.
Thanks for the advice, I think it was enough to nudge me in a
direction that will work better to get me going, and if with more
experience other usage makes more sense I can always refactor,
refactor, refactor :)

Yup!

Happy to help, sorry for the super long lag time on this reply, but my email queue has been pretty full lately :)

-glyph

[Daniel Sutcliffe]

Delayed response here but hopefully it is still seen as worthwhile
discussion, sorry if it is seems to just be beating a dead horse ;)

Post by Glyph

Post by Daniel Sutcliffe
Given the ClientService.whenConnected() method is intended to provide
access to my connected Protocol through the deferred it returns, is it
not a little unfriendly that this Protocol may turn out to be
disconnected? OK occasionally due to timing but for this to be a
possible condition which can loop with the same disconnected Protocol
returned until the ClientService has its _currentConnection set to
None, suggests to me that I can't safely use my Protocol from
whenConnected() for much other than as a notification the first
connection has occurred... but how do I avoid this?

whenConnected() is not intended to be used for "give me each Protocol as it
is instantiated so that state can be set up", it is intended for API clients
which want to send a message to the current connection to just retrieve the
current connection so they can call a method on it.

I appreciate now this 'protocol/connection init' isn't what the
whenConnected() method was intended for, and it wasn't what I was
using it for when I came across my issue. However there seems to me to
be no documentary discouragement for using it this way... at least for
someone who is not as familiar with Twisted's common usage patterns

Post by Glyph
I'm not sure what you mean by "turn out to be disconnected". The physical
reality of networking is that you might always encounter a transport which
has been disconnected but which you haven't received notification of its
disconnection yet.

The 'turns out to be disconnected' actually came from my acceptance
that network connections drop/fail and I have little control of how or
when this happens. Thus I may call a method on a Protocol at any time
only to find that it is not in a good state to handle my request.

My (misguided) goal was to call a method of the Protocol returned from
whenConnected() every so often to give it a task to do; my expectation
being that *when* the connection failed I would find out from the task
method's returned deferred and could then just fire off another
ClientService whenConnected() to get the next useful Protocol once it
is established, and then use this to continue giving the Protocol
tasks until it also ultimately fails...

Post by Glyph

Post by Daniel Sutcliffe
I have looked at the source and it seems to me the fact that the
connection has been lost should bubble up to the ClientService through
a t.a.i._DisconnectFactory and t.a.i._ReconnectingProtocolProxy once
my Protocol's connectionLost() is called. My issue seems to be that I
errback on a Protocol method's deferred returned to code at or above
the ClientService level which gives up on that Protocol and calls
whenConnected() to get the next one, only the Protocol's
connectionLost() has yet to be called and then doesn't have chance to
because my code is looping around calling whenConnected() and getting
the same Protocol back. I hope that makes sense :-/

Let me try to rephrase: you call a protocol method which returns a Deferred;
you add an errback to that Deferred which calls whenConnected() to re-try,
but since the protocol hasn't disconnected yet, you get the same protocol
instance back, which is useless to you.

Thanks Glyph, that seems like exactly what I was trying to say.

Post by Glyph

Post by Daniel Sutcliffe
My Q on this is if I should be internally calling my Protocol's
connectionLost() so it can bubble up to the ClientService before I
errback on the Protocol method - whose responsibility is it to call
this?

It's the framework's responsibility to call it. You should not call it
yourself. Your Protocol's connectionLost isn't going to bubble up to
ClientService anyway; you'd have to call your wrapper's connectionLost,
which would confuse its internal state, since the framework would call it
again right afterwards, and we definitely don't have test coverage for that,
since the framework will normally only call it once.

I'm glad that I got this mostly right in my head - it didn't feel
right for me to be calling this, or any similar method I could find.

Post by Glyph
The right way to handle this would be to introduce a delay between re-tries.
It's generally a good idea to have such a delay for lots of reasons; you
don't want to overload your peer in the case of a transient failure. As a
bonus, the fact that you've gone back up to the reactor loop to wait a while
means that the transport will be properly disconnected and whenConnected()
will do what you want.

I didn't even consider this as my thinking was along the lines of the
ClientService having the ability to handle the connection re-tries and
delays to avoid any overloading of the service it wraps.

It would be nice if there was something I could do in my Protocol task
method such that it didn't fire the errback on its deferred until its
Factory (and ultimately its Service in this case) had been made aware
the connection had failed. Does this not have to be done elsewhere in
Twisted or other projects using Twisted, or is the common pattern just
to introduce a delay to avoid any possible race condition?

My current intention is to have the Protocol's Factory handle the
assigning of tasks to its Protocol which seems the 'more normal'
direction to take, so any discussions above are purely for my
education and intellectual curiosity - I'm glad of any and all
feedback.

Cheers
/dan