[Twisted-Python] Writing a low-level network debugging tool

[Jonathan Ballet]

Hi all,

I'm trying to toubleshoot network connectivity issues we have in one of
our office and I would like to monitor some metrics which seems to be
relevant for us, especially when trying to open TCP connections towards
external endpoints.

In particular, I'm looking for a way to get the following information
(let's say I want to monitor the connectivity towards the swisscom.com,
port 80 using TCP):

* how long does it take to resolv the domain name to (at least) one of
its IP address
- against a specified name server or using the system configured
servers
- how many tries did it require
* if there were several tries, the timing of each ones

* how long does it take to get the first bytes of the endpoint
- how long does it take to complete the TCP connection handshake
- the status of the packets exchanged (how many retries, how many
packets lost, etc.)

It's not exactly the same, but the curl option --write-out allows to get
this kind of values (especially time_namelookup, time_connect,
time_pretransfer, time_starttransfer and time_total) but I would like to
have more flexibility and more in-depth informations (like the state of
the packets exchanged, etc.)

How far can I do this kind of things with Twisted? I know I can somewhat
easily get the timings of the name resolution, the TCP connection
handshake also and the time to first byte(s), but what about the
packets? I haven't look at the code of Twisted Names yet, but if it's
doing the DNS request by itself, I may be able to plug-in somewhere and
have my request counter and the timers associated, but I'm not sure if
the underlying details of the TCP protocol are exposed to the upper
layer such as Twisted?

Thanks for the help!

Jonathan

[Phil Mayers]

Post by Jonathan Ballet
* how long does it take to resolv the domain name to (at least) one of
its IP address
- against a specified name server or using the system configured servers

That is relatively straightforward.

Post by Jonathan Ballet
- how many tries did it require
* if there were several tries, the timing of each ones

Typically, application-layer code doesn't retry a DNS lookup; rather the
c or other runtime will handle this, for example getaddrinfo() in glibc,
according to settings read from /etc/resolv.conf or compiled-in defaults.

So it depends on whether you want to emulate "typical" application code,
a specific application stack that may or may not do it's own resolution
(e.g. modern browsers) or something else.

Post by Jonathan Ballet
* how long does it take to get the first bytes of the endpoint
- how long does it take to complete the TCP connection handshake
- the status of the packets exchanged (how many retries, how many
packets lost, etc.)

Some of this is available in platform-specific APIs e.g. SIOCGSTAMP and
TCP_INFO socket options available on Linux.

In general, any timings you make based on return of control from kernel
will include error relating to system/scheduling issues. If you're
concerned about getting raw, on-the-wire timings, this is extremely
difficult without being in-kernel, and even then various issues - TCP
offload for example - can end up hiding data from you.

Post by Jonathan Ballet
How far can I do this kind of things with Twisted? I know I can somewhat
easily get the timings of the name resolution, the TCP connection
handshake also and the time to first byte(s), but what about the
packets? I haven't look at the code of Twisted Names yet, but if it's
doing the DNS request by itself, I may be able to plug-in somewhere and
have my request counter and the timers associated, but I'm not sure if
the underlying details of the TCP protocol are exposed to the upper
layer such as Twisted?

Only via platform-specific options.

To do this kind of thing "reliably", you'd need to reimplement TCP in
user-space.

But the info above may be a helpful start.

[Phil Mayers]

Post by Phil Mayers
But the info above may be a helpful start.

You may also want to look at TCP_CC_INFO. See the kernel source:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/net/ipv4/tcp.c?id=refs/tags/v4.4-rc2

...for an idea of what else is available. Be aware some of these are
relatively new.

[Jonathan Ballet]

Post by Phil Mayers

Post by Jonathan Ballet
- how many tries did it require
* if there were several tries, the timing of each ones

Typically, application-layer code doesn't retry a DNS lookup; rather the
c or other runtime will handle this, for example getaddrinfo() in glibc,
according to settings read from /etc/resolv.conf or compiled-in defaults.
So it depends on whether you want to emulate "typical" application code,
a specific application stack that may or may not do it's own resolution
(e.g. modern browsers) or something else.

That's a fair point, and I would like, as best as it can, to be as close
as possible as a "typical" application; the goal really would be to
measure the network conditions the applications are facing.

Although I understand I won't be able to get the retries number if the
underlying code is using getaddrinfo() or something like this, I was
thinking that twisted.names was maybe offering a "hand-made" resolver,
which was producing the UDP packets itself and offered a way to plug
some code there to measure these retries; I haven't checked yet.

But in any case, I guess it's going a little bit against my previous
point which was to try to measure things "as used by a 'typical'
application" (which isn't going to use twisted.names custom resolver.)

(Actually, I was surprised to discover that, although it can be
configured from the command-line, `dig` doesn't report the number of
tries/retries it makes, you can only deduce it by looking at the overall
command execution time.)

Post by Phil Mayers

Post by Jonathan Ballet
* how long does it take to get the first bytes of the endpoint
- how long does it take to complete the TCP connection handshake
- the status of the packets exchanged (how many retries, how many
packets lost, etc.)

Some of this is available in platform-specific APIs e.g. SIOCGSTAMP and
TCP_INFO socket options available on Linux.
In general, any timings you make based on return of control from kernel
will include error relating to system/scheduling issues. If you're
concerned about getting raw, on-the-wire timings, this is extremely
difficult without being in-kernel, and even then various issues - TCP
offload for example - can end up hiding data from you.

I will have a look at these options, as I will be running my tests under
Linux anyway.
It's probably out of scope of Twisted anyway, but I could also retrieve
the packets sent on the wire by listening on the related network
interface set in promiscuous mode and correlate packets together. It's
... "slightly" more work though...

Post by Phil Mayers

Post by Jonathan Ballet
How far can I do this kind of things with Twisted? I know I can somewhat
easily get the timings of the name resolution, the TCP connection
handshake also and the time to first byte(s), but what about the
packets? I haven't look at the code of Twisted Names yet, but if it's
doing the DNS request by itself, I may be able to plug-in somewhere and
have my request counter and the timers associated, but I'm not sure if
the underlying details of the TCP protocol are exposed to the upper
layer such as Twisted?

Only via platform-specific options.
To do this kind of thing "reliably", you'd need to reimplement TCP in
user-space.
But the info above may be a helpful start.

Thanks for your answer Phil, I'll see what I can come up with!

Jonathan