[Twisted-Python] ANN: Eliot 0.9, the logging system with causality - now with journald support

[Itamar Turner-Trauring]

Eliot 0.9 is out, with a new utility for pretty-printing log messages
and native journald support [1]. You can now route Eliot logs to
journald and when an error occurs easily find all logged actions that
led up to that particular error, as opposed to most logging systems
where this would involve reading all the logs and figuring out which
ones apply and which to ignore.

Most logging systems can tell you what happened; Eliot tells you _why_
it happened:

$ python linkcheck.py | eliot-tree
4c42a789-76f5-4f0b-b154-3dd0e3041445
+-- ***@1/started
`-- urls: [u'http://google.com', u'http://nosuchurl']
+-- ***@2,1/started
`-- url: http://google.com
+-- ***@2,2/succeeded
+-- ***@3,1/started
`-- url: http://nosuchurl
+-- ***@3,2/failed
|-- exception: requests.exceptions.ConnectionError
|-- reason: ('Conn aborted', gaierror(-2, 'Name unknown'))
+-- ***@4/failed
|-- exception: exceptions.ValueError
|-- reason: ('Conn aborted.', gaierror(-2, 'Name unknown'))

And here's the code that generated these logs (eliot-tree [2] was used
to render the output):

import sys
from eliot import start_action, to_file
import requests
to_file(sys.stdout)

def check_links(urls):
with start_action(action_type="check_links", urls=urls):
for url in urls:
try:
with start_action(action_type="download", url=url):
response = requests.get(url)
response.raise_for_status()
except Exception as e:
raise ValueError(str(e))

check_links(["http://google.com"], ["http://nosuchurl"])

Interested? Read more at https://eliot.readthedocs.org/.

Eliot is released under the Apache License 2 by ClusterHQ [3], the
Container Data People. We're hiring! [4]


Links:
------
[1] http://eliot.readthedocs.org/en/0.9.0/journald.html
[2] https://warehouse.python.org/project/eliot-tree/
[3] https://clusterhq.com
[4] https://clusterhq.com/careers/

[Glyph Lefkowitz]

Eliot 0.9 is out, with a new utility for pretty-printing log messages and native journald support <http://eliot.readthedocs.org/en/0.9.0/journald.html>. You can now route Eliot logs to journald and when an error occurs easily find all logged actions that led up to that particular error, as opposed to most logging systems where this would involve reading all the logs and figuring out which ones apply and which to ignore.

Cool, this looks awesome Itamar, thanks for letting us know!

Does Eliot make use of the new logging system at all? If not, is there anything that might make Eliot's Twisted backend easier to maintain if we added it? Just kind of curious about adoption of those new APIs in the world of logging...

-glyph

[e***@twistedmatrix.com]

Post by Glyph Lefkowitz

On Oct 8, 2015, at 6:04 AM, Itamar Turner-Trauring
Eliot 0.9 is out, with a new utility for pretty-printing log messages
and native journald support
<http://eliot.readthedocs.org/en/0.9.0/journald.html>. You can now
route Eliot logs to journald and when an error occurs easily find all
logged actions that led up to that particular error, as opposed to
most logging systems where this would involve reading all the logs and
figuring out which ones apply and which to ignore.

Cool, this looks awesome Itamar, thanks for letting us know!
Does Eliot make use of the new logging system at all? If not, is there
anything that might make Eliot's Twisted backend easier to maintain if
we added it? Just kind of curious about adoption of those new APIs in
the world of logging...

As a user of Eliot and Twisted, an unpleasantness I commonly encounter
is that the logs written by trial when running tests for code that uses
Eliot for logging are a mish-mash of text messages and mangled Eliot
logs. The Eliot bits often get in the way of me reading the text part
and the text parts and mangling ensure I can't use any Eliot tools to
interpret the Eliot bits.

My understanding is that there aren't any existing hooks in Twisted to
customize trial's logging which would allow this to be improved.

There is some related discussion (maybe more oriented towards a
particular solution than a full description of the problem, though) on a
ticket soon to be a couple years old:

https://twistedmatrix.com/trac/ticket/6939

Jean-Paul

[Glyph Lefkowitz]

Post by Glyph Lefkowitz

Eliot 0.9 is out, with a new utility for pretty-printing log messages and native journald support <http://eliot.readthedocs.org/en/0.9.0/journald.html>. You can now route Eliot logs to journald and when an error occurs easily find all logged actions that led up to that particular error, as opposed to most logging systems where this would involve reading all the logs and figuring out which ones apply and which to ignore.

Cool, this looks awesome Itamar, thanks for letting us know!
Does Eliot make use of the new logging system at all? If not, is there anything that might make Eliot's Twisted backend easier to maintain if we added it? Just kind of curious about adoption of those new APIs in the world of logging...

As a user of Eliot and Twisted, an unpleasantness I commonly encounter is that the logs written by trial when running tests for code that uses Eliot for logging are a mish-mash of text messages and mangled Eliot logs. The Eliot bits often get in the way of me reading the text part and the text parts and mangling ensure I can't use any Eliot tools to interpret the Eliot bits.
My understanding is that there aren't any existing hooks in Twisted to customize trial's logging which would allow this to be improved.
https://twistedmatrix.com/trac/ticket/6939 <https://twistedmatrix.com/trac/ticket/6939>

I actually ran into a variant of this exact same issue recently; thank you for directing my attention to this ticket.

I agree that the log output should be customizable, but I'm curious: if the logs were just serialized as JSON by default everywhere, would that make it easier to make the Eliot tooling work well?

-glyph

[Itamar Turner-Trauring]

Post by Glyph Lefkowitz
Does Eliot make use of the new logging system at all? If not, is there anything that might make Eliot's Twisted backend easier to maintain if we added it? Just kind of curious about adoption of those new APIs in the world of logging...

For Flocker we have a Twisted->Eliot gateway based on the old logging
API. Eventually that'll get moved into Eliot and I guess at that point
it'd make sense to switch to new API but haven't really looked at it.
Might be able to get structured info at that point which'd be nice.

I should steal some ideas from the new system, but at this point I
consider anything that doesn't log actions as legacy logging so I don't
personally write anything that *emits* logs with the new Twisted API.