[Twisted-Python] dropping old pyOpenSSL versions

[Glyph Lefkowitz]

In the past, we've been very conservative about updating to require new versions of pyOpenSSL and cryptography.

Right now we have a patch, <https://github.com/twisted/twisted/pull/146> (<https://twistedmatrix.com/trac/ticket/8441#comment:1>), that I'd like to just land. However, it establishes a dependency on a new version of pyOpenSSL, which transitively establishes a dependency on a new version of Cryptography.

Generally, my thinking has evolved over the last few years to think that security dependencies like this should move fast, especially on projects (like pyOpenSSL and cryptography specifically) that don't maintain "stable" branches which do security patch-releases.

In this specific case, the fix is not urgent; as it turns out, the netscape SPKI APIs actually do do the desired thing, which is just hashing the DER bytes of the key. (At the time I made the change to use Netscape SPKI, I thought it might be including somet other junk in the hash; we just lucked out here.) It's just a gross API for doing it which we should stop using now that better APIs have been exposed to do the same thing.

However, it bears discussing - what are the things that hold us to older versions of pyOpenSSL and cryptography? Is there any good reason not to move our version pins forward whenever there's a new API or feature that we'd like, even for something simple like this cleanup?

My default position is "upgrade upgrade upgrade" so if there's not a lot of interest in this discussion I'll probably just land the PR in question as-is.

Thanks all,

-glyph

[Clayton Daley]

I don't object to this specific change (we're on shiny new code), but want
to offer some food-for-thought:

1) Is newer really better in cryptography? Heartbleed affected 1.0.1, but
not 1.0.0 and there are a bunch of vulnerabilities that only affect the
newer libraries (https://www.openssl.org/news/vulnerabilities.html). It
even makes sense that the older libraries have been more-thoroughly
tested... so new code may just mean new vulnerabilities.

2) How does this impact regulated industries. In healthcare (my current
industry), changing a library (especially cryptography) could mean:

- An internal review to select a new version of the library
- An internal change management process
- Technical testing (perhaps a 3rd party audit)
- A notification to clients of the change
- Secondary reviews/testing at clients

The intensity of this process depends on the risk level of the system and
this could be a long and complicated process for some organizations. Seems
like a more deliberate deprecation policy would make it easier to plan
ahead.

Clayton Daley

Post by Glyph Lefkowitz
In the past, we've been very conservative about updating to require new
versions of pyOpenSSL and cryptography.
Right now we have a patch, <https://github.com/twisted/twisted/pull/146

Post by Glyph Lefkowitz
(<https://twistedmatrix.com/trac/ticket/8441#comment:1>), that I'd like

to just land. However, it establishes a dependency on a new version of
pyOpenSSL, which transitively establishes a dependency on a new version of
Cryptography.
Generally, my thinking has evolved over the last few years to think that
security dependencies like this should move fast, especially on projects
(like pyOpenSSL and cryptography specifically) that don't maintain "stable"
branches which do security patch-releases.
In this specific case, the fix is not urgent; as it turns out, the
netscape SPKI APIs actually *do* do the desired thing, which is just
hashing the DER bytes of the key. (At the time I made the change to use
Netscape SPKI, I thought it might be including somet other junk in the
hash; we just lucked out here.) It's just a gross API for doing it which
we should stop using now that better APIs have been exposed to do the same
thing.
However, it bears discussing - what are the things that hold us to older
versions of pyOpenSSL and cryptography? Is there any good reason *not* to
move our version pins forward whenever there's a new API or feature that
we'd like, even for something simple like this cleanup?
My default position is "upgrade upgrade upgrade" so if there's not a lot
of interest in this discussion I'll probably just land the PR in question
as-is.
Thanks all,
-glyph
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Tristan Seligmann]

Post by Clayton Daley
I don't object to this specific change (we're on shiny new code), but want
1) Is newer really better in cryptography? Heartbleed affected 1.0.1, but
not 1.0.0 and there are a bunch of vulnerabilities that only affect the
newer libraries (https://www.openssl.org/news/vulnerabilities.html). It
even makes sense that the older libraries have been more-thoroughly
tested... so new code may just mean new vulnerabilities.

First of all, newer cryptography and newer OpenSSL are different things.
Given that cryptography itself is mostly made of Python and cffi, not C
code, I think it's unlikely that a newer version of cryptography is likely
to be worse than an older one. Older libraries being "more thoroughly
tested" only really applies where a library has a plethora of simultaneous
release channels; for most libraries, using older versions just means
missing out on any fixes for issues that were found more recently than the
release was released.

Even regarding OpenSSL, which is a horrible pile of C, it's unlikely that
the potential of another *Heartbleed*-like issue is more dangerous than the
lack of actual known improvements.

2) How does this impact regulated industries. In healthcare (my current

Post by Clayton Daley
- An internal review to select a new version of the library
- An internal change management process
- Technical testing (perhaps a 3rd party audit)
- A notification to clients of the change
- Secondary reviews/testing at clients
The intensity of this process depends on the risk level of the system and
this could be a long and complicated process for some organizations. Seems
like a more deliberate deprecation policy would make it easier to plan
ahead.

Wouldn't all of the above apply equally to the new version of Twisted? I
would imagine you could upgrade Twisted and cryptography at the same time,
thus only doing one round of testing/review/etc. for both. (Perhaps I'm
missing something?)

[Clayton Daley]

Post by Tristan Seligmann
First of all, newer cryptography and newer OpenSSL are different things.

The proposal was a change to pyOpenSSL. If newer is better in all
(potentially) affected layers, then you've answered my question in the
affirmative.

Post by Tristan Seligmann
2) How does this impact regulated industries. In healthcare (my current

Post by Clayton Daley
- An internal review to select a new version of the library
- An internal change management process
- Technical testing (perhaps a 3rd party audit)
- A notification to clients of the change
- Secondary reviews/testing at clients
The intensity of this process depends on the risk level of the system and
this could be a long and complicated process for some organizations. Seems
like a more deliberate deprecation policy would make it easier to plan
ahead.

Wouldn't all of the above apply equally to the new version of Twisted? I
would imagine you could upgrade Twisted and cryptography at the same time,
thus only doing one round of testing/review/etc. for both. (Perhaps I'm
missing something?)

Regulations require an amount of scrutiny proportionate to the risk. We're
small so it's not as obvious in our policies and procedures, but the
potential for big differences in process are exemplified by this paraphrase
from a vendor's document (since I'm not sure I can outright quote it):

Changes with a high rank are escalated to the individual in Role X and

Post by Tristan Seligmann
Committee Y for approval

Changes with a medium risk are escalated to the individual in Role X for

Post by Tristan Seligmann
approval

[presumably low risk changes aren't escalated]
So your question comes down to... is a point release in Twisted as risky as
a change to the cryptography stack? You'd certainly know better than I.

Clayton Daley

[Glyph Lefkowitz]

1) Is newer really better in cryptography? Heartbleed affected 1.0.1, but not 1.0.0 and there are a bunch of vulnerabilities that only affect the newer libraries (https://www.openssl.org/news/vulnerabilities.html <https://www.openssl.org/news/vulnerabilities.html>). It even makes sense that the older libraries have been more-thoroughly tested... so new code may just mean new vulnerabilities.

I can understand the point here - that upgrades are not zero-risk - but this is a very dangerous line of reasoning. (I think "in cryptography" you really mean the general english word "cryptography" and not like "PyCA™ Cryptography™".) It's dangerous because, as application developers, you and I aren't really _qualified_ to evaluate whether newer versions of security-critical libraries are more or less secure; the best we can do is always, always, always upgrade. The solution to heartbleed was to upgrade to a newer version of 1.0.1, not to roll back to 0.9.8.

More importantly, heartbleed itself is not a general category of defect, in the sense that heartbleed itself shone a bright light directly on OpenSSL and exposed the real problem, which was massive underinvestment in critical security infrastructure. The issue isn't that upgrading is dangerous, it's that letting critical infrastructure decay is dangerous.

One way that you can promote the decay of critical infrastructure is to defer upgrading out of vague fears about the upgrade going poorly rather than specific technical issues.

I should take the opportunity to point out that if you're a professional network software developer, you should really be giving a couple of bucks to the OpenSSL foundation: <https://www.openssl.org/support/donations.html>. The most direct way to fight the decay of critical infrastructure is to fund it with cash money. (And, for that matter, <https://twistedmatrix.com/trac/#DonatetoTwisted>...)

An internal review to select a new version of the library
An internal change management process
Technical testing (perhaps a 3rd party audit)
A notification to clients of the change
Secondary reviews/testing at clients
The intensity of this process depends on the risk level of the system and this could be a long and complicated process for some organizations. Seems like a more deliberate deprecation policy would make it easier to plan ahead.

The problem with a lot of the regulatory standards that require this sort of laborious change-control is that during the entire period where all this redundant analysis and re-analysis of the change is happening, customers are still vulnerable (and in a health care context, this may mean even that lives remain at risk!).

My (entirely secondhand) understanding is that recognition is dawning in many compliance fields (HIPPA, PCI-DSS, SOX) that there is a mismatch between the realities of software development (delay in making changes == risk) and industrial change control (every change == risk), and auditors and regulators are beginning to take this into account. This means that the ability to make changes quickly to ensure safe operation is slowly gaining ground over the ability to delay changes until sufficient evaluation has taken place.

This recognition is dawning because many of these reviews are, in fact, nonsense. For example: "an internal review to select a new version of the library" - does every healthcare project have a qualified cryptographer and penetration tester to each independently spend the 6 months of careful code audits required to actually evaluate the relative security of new versions of OpenSSL, or does this internal review consist mainly of unqualified people pontificating about hypothetical risks, divorced from the technical realities of the upgrade in question?

My own experience has taught me that fear of changes like this is mostly developers being afraid of audit or compliance, rather than audit or compliance actually requiring it. If you tell your auditor "we absolutely have to upgrade OpenSSL every week if an update is available", they'll usually figure out a way to do it.

I don't mean to jump down your throat here; the tone is definitely harsher than I would like, but I want it to be very clear why I have such strong feelings about upgrading security-critical dependencies.

Additionally, the Netscape SPKI APIs are disappearing from OpenSSL itself eventually, so this specific issue isn't just about upgrading - it's about preventing it from being difficult to upgrade in the future. If we wait until the APIs are actually gone, rather than just deprecated, this may create friction around a critical security update.

-glyph

[Clayton Daley]

Post by Glyph Lefkowitz
I don't mean to jump down your throat here; the tone is definitely harsher
than I would like, but I want it to be very clear why I have such strong
feelings about upgrading security-critical dependencies.

I don't take it personally. I do a little coding (hello startup) but I'm
actually the guy who:

- Had to develop our Policies and Procedures (P&P) in conjunction with
our Compliance consultant
- Has to work with our lawyers to negotiate Information Security and
Business Associate agreements with customers
- Has to provide implementation details in request to our customers'
security groups (I spent all day working through a 160 item self-assessment
so it was top-of-mind for me)

When there's a vulnerability, you can fast track an upgrade because there's
a non-theoretical risk to doing nothing. The problem is an "optional"
version bump. It's all CYA. If I don't follow my P&P, the federal
government, state government, and customers all have (extra) grounds to sue
my company (cofounder so literally *mine*). If the consequence of waiting
are a transient Twisted bug or a delayed feature depending on a feature in
a blocked version, it's an easy choice.

Clayton Daley

Post by Glyph Lefkowitz
I don't object to this specific change (we're on shiny new code), but want
1) Is newer really better in cryptography? Heartbleed affected 1.0.1, but
not 1.0.0 and there are a bunch of vulnerabilities that only affect the
newer libraries (https://www.openssl.org/news/vulnerabilities.html). It
even makes sense that the older libraries have been more-thoroughly
tested... so new code may just mean new vulnerabilities.
I can understand the point here - that upgrades are not zero-risk - but
this is a very dangerous line of reasoning. (I think "in cryptography" you
really mean the general english word "cryptography" and not like "PyCA™
Cryptography™".) It's dangerous because, as application developers, you
and I aren't really _qualified_ to evaluate whether newer versions of
security-critical libraries are more or less secure; the best we can do is
always, always, always upgrade. The solution to heartbleed was to upgrade
to a newer version of 1.0.1, not to roll back to 0.9.8.
More importantly, heartbleed itself is not a general category of defect,
in the sense that heartbleed itself shone a bright light directly on
OpenSSL and exposed the real problem, which was massive underinvestment in
critical security infrastructure. The issue isn't that upgrading is
dangerous, it's that letting critical infrastructure decay is dangerous.
One way that you can *promote* the decay of critical infrastructure is to
defer upgrading out of vague fears about the upgrade going poorly rather
than specific technical issues.
I should take the opportunity to point out that if you're a professional
network software developer, you should really be giving a couple of bucks
to the OpenSSL foundation: <https://www.openssl.org/support/donations.html>.
The most direct way to fight the decay of critical infrastructure is to
fund it with cash money. (And, for that matter, <
https://twistedmatrix.com/trac/#DonatetoTwisted>...)
2) How does this impact regulated industries. In healthcare (my current
- An internal review to select a new version of the library
- An internal change management process
- Technical testing (perhaps a 3rd party audit)
- A notification to clients of the change
- Secondary reviews/testing at clients
The intensity of this process depends on the risk level of the system and
this could be a long and complicated process for some organizations. Seems
like a more deliberate deprecation policy would make it easier to plan
ahead.
The problem with a lot of the regulatory standards that require this sort
of laborious change-control is that during the entire period where all this
redundant analysis and re-analysis of the change is happening, customers
are still vulnerable (and in a health care context, this may mean even that
lives remain at risk!).
My (entirely secondhand) understanding is that recognition is dawning in
many compliance fields (HIPPA, PCI-DSS, SOX) that there is a mismatch
between the realities of software development (delay in making changes ==
risk) and industrial change control (every change == risk), and auditors
and regulators are beginning to take this into account. This means that
the ability to make changes *quickly* to ensure safe operation is slowly
gaining ground over the ability to delay changes until sufficient
evaluation has taken place.
This recognition is dawning because many of these reviews are, in fact,
nonsense. For example: "an internal review to select a new version of the
library" - does every healthcare project have a qualified cryptographer and
penetration tester to each independently spend the 6 months of careful code
audits required to actually evaluate the relative security of new versions
of OpenSSL, or does this internal review consist mainly of unqualified
people pontificating about hypothetical risks, divorced from the technical
realities of the upgrade in question?
My own experience has taught me that fear of changes like this is mostly
developers being afraid of audit or compliance, rather than audit or
compliance actually requiring it. If you tell your auditor "we absolutely
have to upgrade OpenSSL every week if an update is available", they'll
usually figure out a way to do it.
I don't mean to jump down your throat here; the tone is definitely harsher
than I would like, but I want it to be very clear why I have such strong
feelings about upgrading security-critical dependencies.
Additionally, the Netscape SPKI APIs are disappearing from OpenSSL itself
eventually, so this specific issue isn't just about upgrading - it's about
preventing it from being difficult to upgrade in the future. If we wait
until the APIs are actually gone, rather than just deprecated, this may
create friction around a critical security update.
-glyph
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Glyph Lefkowitz]

Post by Glyph Lefkowitz
I don't mean to jump down your throat here; the tone is definitely harsher than I would like, but I want it to be very clear why I have such strong feelings about upgrading security-critical dependencies.
I don't take it personally.

Whew :).

Post by Glyph Lefkowitz
Had to develop our Policies and Procedures (P&P) in conjunction with our Compliance consultant
Has to work with our lawyers to negotiate Information Security and Business Associate agreements with customers
Has to provide implementation details in request to our customers' security groups (I spent all day working through a 160 item self-assessment so it was top-of-mind for me)
When there's a vulnerability, you can fast track an upgrade because there's a non-theoretical risk to doing nothing. The problem is an "optional" version bump. It's all CYA. If I don't follow my P&P, the federal government, state government, and customers all have (extra) grounds to sue my company (cofounder so literally *mine*). If the consequence of waiting are a transient Twisted bug or a delayed feature depending on a feature in a blocked version, it's an easy choice.

The problem with this perspective is that it inappropriately assigns risk by default to upgrading but no risk by default to not-upgrading. For example, it is well known that various adversaries stockpile 0-day vulnerabilities in popular libraries. Of course new releases don't ever empty this stockpile, but they quite often reduce it.

Often fixes to this type of secret vulnerability are not identified as "high severity" because severity classifications are often incorrect, almost always in the direction of having a lower severity than they ought to. See for example this paper: https://www.usenix.org/legacy/event/hotos09/tech/full_papers/arnold/arnold_html/ It shows that (under the range of data collected, 2006-2008) there is _always_ a non-zero number of misclassified bugs impacting "stable" kernel versions' security. The same is almost certainly true of OpenSSL. Not to mention the fact that being stuck on old OpenSSL means being stuck without fundamental improvements such as TLS 1.3.

In other words, it may be possible to show that there is absolutely always a vulnerability being fixed by new versions, even when you are pretty sure there isn't.

I understand that certain regulatory regimes do still give a huge financial incentive to bias your change management decisions towards "status quo"; my comment earlier indicated that this is starting to change, not that that process is complete. Even if you were to agree with me completely it might not be reasonable to risk the entire future of your business on a fast upgrade cadence if you are liable for the risks of upgrading but not liable for the risks of not-upgrading.

However, in a situation with perverse incentives like that, an equally significant risk is building a process that punishes even preparing to make a change. Inasmuch as it's feasible should always have a codebase which is ready to roll out upgraded versions of every dependency, as if the regulators were to allow the upgrades, because when security researchers identify that a vulnerability is high-impact after the fact, you don't want to have to make big changes or retrofit your tooling or your codebase in the moment of that impact. Presumably all the governments and the customers could still sue you if you hadn't managed to fix e.g. Heartbleed after <some span of time that a layperson would think is unreasonable>.

Another great example of why you want to be ready to upgrade: if you can run your tests against a new Twisted in its pre-release week and report a regression (or better yet, run continuously against trunk and report regressions that affect your code as they occur) then you can offload the work of actually keeping your application running onto us, and force us to avoid ever releasing a version that breaks you. But if you only identify bugs years after the fact, there's no longer anything we can do except fix them with the same priority as everything else.

-glyph

[Clayton Daley]

We're very close on the theory. My point wasn't to discourage upgrades.
Even regulated entities can and should upgrade their security libraries as
part of their annual audit cycle. My point was to promote a more
deliberate depreciation cycle with better visibility for regulated entities.

I'm not clear the age of the library we're talking about, but for the sake
of argument consider something like:

- On Jan 1 every year, we mark the "current" version of the library
- On Jan 1 two years later, we deprecate everything older than that
version
- On Jan 1 three years later, we kill off support for everything before
that version

The worst case scenario is a library from Dec 31, 2015 is killed on Jan 1,
2018, a two year (plus one day) window. That would mean a minimum of two
annual cycles to make the upgrade. It would be easy to plan upgrades. It
would be clear which versions were dying when.

On the flip side, it would strongly discourage use of a library older than
3 years. Most folks would go from a 1.5 yr old library to a 0.5 yr old
library each year. Even a luddite (who wanted to guarantee support) would
be replacing a 2.5yr old library with a 1.5yr old library.

Clayton Daley

Post by Glyph Lefkowitz
I don't mean to jump down your throat here; the tone is definitely harsher

Post by Glyph Lefkowitz
than I would like, but I want it to be very clear why I have such strong
feelings about upgrading security-critical dependencies.

I don't take it personally.
Whew :).
- Had to develop our Policies and Procedures (P&P) in conjunction with
our Compliance consultant
- Has to work with our lawyers to negotiate Information Security and
Business Associate agreements with customers
- Has to provide implementation details in request to our customers'
security groups (I spent all day working through a 160 item self-assessment
so it was top-of-mind for me)
When there's a vulnerability, you can fast track an upgrade because
there's a non-theoretical risk to doing nothing. The problem is an
"optional" version bump. It's all CYA. If I don't follow my P&P, the
federal government, state government, and customers all have (extra)
grounds to sue my company (cofounder so literally *mine*). If the
consequence of waiting are a transient Twisted bug or a delayed feature
depending on a feature in a blocked version, it's an easy choice.
The problem with this perspective is that it inappropriately assigns risk
by default to upgrading but no risk by default to not-upgrading. For
example, it is well known that various adversaries stockpile 0-day
vulnerabilities in popular libraries. Of course new releases don't ever
*empty* this stockpile, but they quite often reduce it.
Often fixes to this type of secret vulnerability are not identified as
"high severity" because severity classifications are often incorrect,
almost always in the direction of having a lower severity than they ought
https://www.usenix.org/legacy/event/hotos09/tech/full_papers/arnold/arnold_html/
It shows that (under the range of data collected, 2006-2008) there is
_always_ a non-zero number of misclassified bugs impacting "stable" kernel
versions' security. The same is almost certainly true of OpenSSL. Not to
mention the fact that being stuck on old OpenSSL means being stuck without
fundamental improvements such as TLS 1.3.
In other words, it may be possible to show that there is absolutely always
a vulnerability being fixed by new versions, even when you are pretty sure
there isn't.
I understand that certain regulatory regimes do still give a huge
financial incentive to bias your change management decisions towards
"status quo"; my comment earlier indicated that this is *starting *to
change, not that that process is complete. Even if you were to agree with
me completely it might not be reasonable to risk the entire future of your
business on a fast upgrade cadence if you are liable for the risks of
upgrading but not liable for the risks of not-upgrading.
However, in a situation with perverse incentives like that, an equally
significant risk is building a process that punishes even *preparing* to
make a change. Inasmuch as it's feasible should always have a codebase
which is ready to roll out upgraded versions of every dependency, *as if* the
regulators were to allow the upgrades, because when security researchers
identify that a vulnerability is high-impact after the fact, you don't want
to have to make big changes or retrofit your tooling or your codebase in
the moment of that impact. Presumably all the governments and the
customers could still sue you if you hadn't managed to fix e.g. Heartbleed
after <some span of time that a layperson would think is unreasonable>.
Another great example of why you want to be ready to upgrade: if you can
run your tests against a new Twisted in its pre-release week and report a
regression (or better yet, run continuously against trunk and report
regressions that affect your code as they occur) then you can offload the
work of actually keeping your application running onto us, and force us to
avoid ever releasing a version that breaks you. But if you only identify
bugs years after the fact, there's no longer anything we can do except fix
them with the same priority as everything else.
-glyph
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Ray Cote]

Post by Clayton Daley
2) How does this impact regulated industries. In healthcare (my current
- An internal review to select a new version of the library
- An internal change management process
- Technical testing (perhaps a 3rd party audit)
- A notification to clients of the change
- Secondary reviews/testing at clients
The intensity of this process depends on the risk level of the system and
this could be a long and complicated process for some organizations. Seems
like a more deliberate deprecation policy would make it easier to plan
ahead.

Contrast this with a standard such as PCI (credit card security compliance)
where the assessors are required to check if you’re running the latest
version of all components.
Need to be up to date if you want to be compliant.