“20 Ways to Make AppSec Move at the Speed of DevOps” is in CSO. It’s a good collection, and I’m quoted.
Emergynt has created the Emergynt Risk Deck, a set of 51 cards, representing actors, vulnerabilities, targets, consequences and risks. It’s more a discussion tool than a game, but I have a weakness for the word “emergent,” and I’ve added it to my list of security games
Also, Lancaster University has created an Agile Security Game.
When I think about how to threat model well, one of the elements that is most important is how much people need to keep in their heads, the cognitive load if you will.
In reading Charlie Stross’s blog post, “Writer, Interrupted” this paragraph really jumped out at me:
One thing that coding and writing fiction have in common is that both tasks require the participant to hold huge amounts of information in their head, in working memory. In the case of the programmer, they may be tracing a variable or function call through the context of a project distributed across many source files, and simultaneously maintaining awareness of whatever complex APIs the object of their attention is interacting with. In the case of the author, they may be holding a substantial chunk of the plot of a novel (or worse, an entire series) in their head, along with a model of the mental state of the character they’re focussing on, and a list of secondary protagonists, while attempting to ensure that the individual sentence they’re currently crafting is consistent with the rest of the body of work.
One of the reasons that I’m fond of diagrams is that they allow the threat modelers to migrate information out of their heads into a diagram, making room for thinking about threats.
Lately, I’ve been thinking a lot about threat modeling tools, including some pretty interesting tools for automated discovery of existing architecture from code. That’s pretty neat, and it dramatically cuts the cost of getting started. Reducing effort, or cost, is inherently good. Sometimes, the reduction in effort is an unalloyed good, that is, any tradeoffs are so dwarfed by benefits as to be unarguable. Sometimes, you lose things that might be worth keeping, either as a hobby like knitting or in the careful chef preparing a fine meal.
I think a lot about where drawing diagrams on a whiteboard falls. It has a cost, and that cost can be high. “Assemble a team of architect, developer, test lead, business analyst, operations and networking” reads one bit of advice. That’s a lot of people for a cross-functional meeting.
That meeting can be a great way to find disconnects in what people conceive of building. And there’s a difference between drawing a diagram and being handed a diagram. I want to draw that out a little bit and ask for your help in understanding the tradeoffs and when they might and might not be appropriate. (Gary McGraw is fond of saying that getting these people in a room and letting them argue is the most important step in “architectural risk analysis.” I think it’s tremendously valuable, and having structures, tools and methods to help them avoid ratholes and path dependency is a big win.)
So what are the advantages and disadvantages of each?
- Collaboration. Walking to the whiteboard and picking up a marker is far less intrusive than taking someone’s computer, or starting to edit a document in a shared tool.
- Ease of use. A whiteboard is still easier than just about any other drawing tool.
- Discovery of different perspective/belief. This is a little subtle. If I’m handed a diagram, I’m less likely to object. An objection may contain a critique of someone else’s work, it may be a conflict. As something is being drawn on a whiteboard, it seems easier to say “what about the debug interface?” (This ties back to Gary McGraw’s point.)
- Storytelling. It is easier to tell a story standing next to a whiteboard than any tech I’ve used. A large whiteboard diagram is easy to point at. You’re not blocking the projector. You can easily edit as you’re talking.
- Messy writing/what does that mean? We’ve all been there? Someone writes something in shorthand as a conversation is happening, and either you can’t read it or you can’t understand what was meant. Structured systems encourage writing a few more words, making things more tedious for everyone around.
- Automatic analysis. Tools like the Microsoft Threat Modeling tool can give you a baseline set of threats to which you add detail. Structure is a tremendous aid to getting things done, and in threat modeling, it helps in answering “what could go wrong?”
- Authority/decidedness/fixedness. This is the other side of the discovery coin. Sometimes, there are architectural answers, and those answers are reasonably fixed. For example, hardware accesses are mediated by the kernel, and filesystem and network are abstracted there. (More recent kernels offer filesystems in userland, but that change was discussed in detail.) Similarly, I’ve seen large, complex systems with overall architecture diagrams, and a change to these diagrams had to be discussed and approved in advance. If this is the case, then a fixed diagram, printed poster size and affixed to walls, can also be used in threat modeling meetings as a context diagram. No need to re-draw it as a DFD.
- Photographs of whiteboards are hard to archive and search without further processing.
- Photographs of whiteboards may imply that ‘this isn’t very important.” If you have a really strong culture of “just barely good enough” than this might not be the case, but if other documents are more structured or cared for, then photos of a whiteboard may carry a message.
- Threat modeling only late. If you’re going to get architecture from code, then you may not think about it until the code is written. If you weren’t going to threat model anyway, then this is a win, but if there was a reasonable chance you were going to do the architectural analysis while there was a chance to change the architecture, software tools may take that away.
(Of course, there are apps that help you take images from a whiteboard and improve them, for example, Best iOS OCR Scanning Apps, which I’m ignoring for purposes of teasing things out a bit. Operationally, probably worth digging into.)
I’d love your thoughts: are there other advantages or disadvantages of a whiteboard or software?
No, seriously. Articles like “Microsoft Secure Boot key debacle causes security panic” and “Bungling Microsoft singlehandedly proves that golden backdoor keys are a terrible idea” draw on words in an advisory to say that this is all about golden keys and secure boot. This post is not intended to attack anyone; researchers, journalists or Microsoft, but to address a rather inflammatory claim that’s being repeated.
Based on my read of a advisory copy (which I made because I cannot read words on an animated background (yes, I’m a grumpy old man (who uses too many parentheticals (especially when I’m sick)))), this is a nice discovery of an authorization failure.
What they found is:
The “supplemental” policy contains new elements, for the merging conditions. These conditions are (well, at one time) unchecked by bootmgr when loading a legacy policy. And bootmgr of win10 v1511 and earlier certainly doesn’t know about them. To those bootmgrs, it has just loaded in a perfectly valid, signed policy. The “supplemental” policy does NOT contain a DeviceID. And, because they were meant to be merged into a base policy, they don’t contain any BCD rules either, which means that if they are loaded, you can enable testsigning.
That’s a fine discovery and a nice vuln. There are ways Microsoft might have designed this better, I’m going to leave those for another day.
Where the post goes off the rails, in my view, is this:
About the FBI: are you reading this? If you are, then this is a perfect real world example about why your idea of backdooring cryptosystems with a “secure golden key” is very bad! Smarter people than me have been telling this to you for so long, it seems you have your fingers in your ears. You seriously don’t understand still? Microsoft implemented a “secure golden key” system. And the golden keys got released from MS own stupidity. Now, what happens if you tell everyone to make a “secure golden key” system?  (Bracketed numbers added – Adam)
So, , no they did not.  No it didn’t.  Even a stopped clock …
You could design a system in which there’s a master key, and accidentally release that key. Based on the advisory, Microsoft has not done that. (I have not talked to anyone at MS about this issue; I might have talked to people about the overall design, but don’t recall having done so.) What this is is an authorization system with a design flaw. As far as I can tell, no keys have been released.
Look, there are excellent reasons to not design a “golden key” system. I talked about them at a fundamental engineering level in my threat modeling book, and posted the excerpt in “Threat Modeling Crypto Back Doors.”
The typical way the phrase “golden key” is used (albiet fuzzily) is that there is a golden key which unlocks communications. That is a bad idea. This is not that, and we as engineers or advocates should not undercut our position on that bad idea by referring to this research as if it really impacts on that “debate.”
I always get a little frisson of engineering joy when I drive over the Tacoma Narrows bridge. For the non-engineers in the audience, the first Tacoma Narrows bridge famously twisted itself to destruction in a 42-mph wind.
The bridge was obviously unstable even during initial construction (as documented in “Catastrophe to Triumph: Bridges of the Tacoma Narrows.”) And so when it started to collapse, several movie cameras were there to document the event, which is still studied and analyzed today.
Today, people are tired of hearing about bridges collapsing. These stories undercut confidence, and bridge professionals are on top of things (ahem). When a bridge collapses, there’s a risk of a lawsuit, and if that was happening, no company could deliver bridges at a reasonable price. We cannot account for the way that wind behaves in the complex fiords of the Puget Sound.
Of course, these are not the excuses of bridge builders, but of security professionals.
I always get a little frisson of engineering joy when I drive over the Tacoma Narrows bridge, and marvel at how we’ve learned from previous failures.
Bruce Schneier comments on “Apple’s Differential Privacy:”
So while I applaud Apple for trying to improve privacy within its business models, I would like some more transparency and some more public scrutiny.
Do we know enough about what’s being done? No, and my bet is that Apple doesn’t know precisely what they’ll ship, and aren’t answering deep technical questions so that they don’t mis-speak. I know that when I was at Microsoft, details like that got adjusted as we learned from a bigger pile of real data from real customer use informed things. I saw some really interesting shifts surprisingly late in the dev cycle of various products.
I also want to challenge the way Matthew Green closes: “If Apple is going to collect significant amounts of new data from the devices that we depend on so much, we should really make sure they’re doing it right — rather than cheering them for Using Such Cool Ideas.”
But that is a false dichotomy, and would be silly even if it were not. It’s silly because we can’t be sure if they’re doing it right until after they ship it, and we can see the details. (And perhaps not even then.)
But even more important, the dichotomy is not “are they going to collect substantial data or not?” They are. The value organizations get from being able to observe their users is enormous. As product managers observe what A/B testing in their web properties means to the speed of product improvement, they want to bring that same ability to other platforms. Those that learn fastest will win, for the same reasons that first to market used to win.
Next, are they going to get it right on the first try? No. Almost guaranteed. Software, as we learned a long time ago, has bugs. As I discussed in “The Evolution of Secure Things:”
Its a matter of the pressures brought to bear on the designs of even what (we now see) as the very simplest technologies. It’s about the constant imperfection of products, and how engineering is a response to perceived imperfections. It’s about the chaotic real world from which progress emerges. In a sense, products are never perfected, but express tradeoffs between many pressures, like manufacturing techniques, available materials, and fashion in both superficial and deep ways.
Green (and Schneier) are right to be skeptical, and may even be right to be cynical. We should not lose sight of the fact that Apple is spending rare privacy engineering resources to do better than Microsoft. Near as I can tell, this is an impressive delivery on the commitment to be the company that respects your privacy, and I say that believing that there will be both bugs and design flaws in the implementation. Green has an impressive record of finding and calling Apple (and others) on such, and I’m optimistic he’ll have happy hunting.
In the meantime, we can, and should, cheer Apple for trying.
I’ve repeatedly spoken out against “think like an attacker.”
Now I’m going to argue from authority. In this long article, “The Obama Doctrine,” the President of the United States says “The degree of tribal division in Libya was greater than our analysts had expected.”
So let’s think about that statement and what it means. First, it means that the multi-billion dollar analytic apparatus of the United States made a mistake, a serious one about which the President cares, because it impacted his foreign policy. Second, that mistake was about how people think. Third, that group of people was a society, and one that has interacted with the United States since, oh, I don’t know, someone wrote words like “From the halls of Montezuma to the shores of Tripoli.” (And dig the Marines, kickin’ it old skool with that video.) Fourth, it was not a group that attempts to practice operational security in any way.
So if we consider that the analytical capability of the US can get that wrong, do you really want to try to think like Anonymous, think like 61398, like 8200? Are you going to do this perfectly, or are there chances to make mistakes? Alternately, do you want to require everyone who threat models to know how attackers think? Understanding how other people think and prioritize requires a great deal of work. There are entire fields, like anthropology and sociology dedicated to doing it well. Should we start our defense by reading books on the motivational structures of the PLA or the IDF?
The simple fact is, you don’t need to. You can start from what people are building or deploying. (I wrote a book on how.) The second simple fact is repeating that phrase upsets people. When I first joined Microsoft, I used that phrase. One day, a developer grabbed me after a meeting, and politely told me that he didn’t understand it. Oh, wait, this was Microsoft in 2006. He told me I was a fucking idiot and I should give useful advice. After a bit more conversation, he also told me that he had no idea how the fuck an attacker thought, and if I thought he had time to read a book to learn about it, I could write the goddamned features customers pay for while he read.
Every time someone tells me to think like an attacker, I think about that conversation. I appreciate the honesty that the fellow showed, if not his manner. But (as Dave Weinstein pointed out) “A generalized form of this would be ‘Stop giving developers completely un-actionable “guidance”.’ Now, Dave and I worked together at Microsoft, so maybe there’s a similar experience in his past.
Now, this does not mean that we don’t need to pay attention to what real attackers do. It means that we don’t need to walk a mile in their shoes to defend effectively against it.
Previously, “Think Like An Attacker?,” “The Discipline of “think like an attacker”,” and “Think Like An Attacker? Flip that advice!.” [Edited, also previously, at the New School blog: “Modeling Attackers and Their Motives.”]
I was confused about why Dan Kaminsky would say CVE-2015-7547 (a bug in glbc’s DNS handling) creates network attack surface for sudo. Chris Rohlf kindly sorted me out by mentioning that there’s now a -host option to sudo, of which I was unaware.
I had not looked at sudo in depth for probably 20 years, and I’m shocked to discover that it has a -e option to invoke an editor, a -p option to process format string bugs, and a -a to allow the invoker to select authentication type(?!?!)
It’s now been a fully twenty years that I’ve been professionally involved in analyzing source code. (These Security Code Review Guidelines were obviously not started in August.) We know that all code has bugs, and more code is strongly correlated with more bugs. I first saw this in the intro to the first edition of Cheswick and Bellovin. I feel a little bit like yelling you kids get off my lawn, but really, the unix philosophy of “do one thing well” was successful for a reason. The goal of sudo is to let the user go through a privilege boundary. It should be insanely simple. [Updated to add, Justin Cormack mentions that OpenBSD went from sudo to doas on this basis.]
It’s not. Not that ssh is simple either, but it isolates complexity, and helps us model attack surface more simply.
Some of the new options make sense, and support security feature sets not present previously. Some are just dumb.
As I wrote this, Dan popped up to say that it also parses /etc/hostname to help it log. Again, do one thing well. Syslog should know what host it’s on, what host it’s transmitting from, and what host its receiving from.
It’s very, very hard to make code secure. When we add in insane options to code, we make it even harder. Sometimes, other people ask us to make the code less secure, and while I’ve already said what I want to say about the FBI asking Apple to fix their mistake by writing new code, this is another example of shooting ourselves in our feet.
Please stop making it harder.
[Update: related “Not-quite-so-broken TLS: lessons in re-engineering a security protocol specification and implementation,” abstracted by the morning paper” which examines an approach to re-implementing TLS, thanks to Steve Bellovin for the pointer.]
John Masserini has a set of “open letters to security vendors” on Security Current.
Everyone involved in product or sales at a security startup should read them. John provides insight into what it’s like to be pitched by too many startups, and provides a level of transparency that’s sadly hard to find. Personally, I learned a great deal about what happens when you’re pitched while I was at a large company, and I can vouch for the realities he puts forth. The sooner you understand those realities and incorporate them into your thinking, the more successful we’ll all be.
After meeting with dozens of startups at Black Hat a few weeks ago, I’ve realized that the vast majority of the leaders of these new companies struggle to articulate the value their solutions bring to the enterprise.
Why does John’s advice make us all more successful? Because each organization that follows it moves towards a more efficient state, for themselves and for the folks who they’re pitching.
Getting more efficient means you waste less time per prospect. When you focus on qualified leads who care about the problem you’re working on, you get more sales per unit of time. What’s more, by not wasting the time of those who won’t buy, you free up their time for talking to those who might have something to provide them. (One banker I know said “I could hire someone full-time to reject startup pitches.” Think about what that means for your sales cycle for a moment.)
One of the values of models is they can help us engage in areas where otherwise the detail is overwhelming. For example, C is a model of how a CPU works that allows engineers to defer certain details to the compiler, rather than writing in assembler. It empowers software developers to write for many CPU architectures at once. Many security flaws happen in areas the models simplify. For example, what if the stack grew away from the stack pointer, rather than towards it? The layout of the stack is a detail that is modeled away.
Information security is a broad industry, requiring and rewarding specialization. We often see intense specialization, which can naturally result in building expertise in silos, and tribal separation of knowledge create more gaps. At the same time, there is a stereotype that generalists end up focused on policy, or “risk management” where a lack of technical depth can hide. (That’s not to say that all risk managers are generalists or that there’s not real technical depth to some of them.)
If we want to enable more security generalists, and we want those generalists to remain grounded, we need to make it easier to learn about new areas. Part of that is good models, part of that is good exercises that appropriately balance challenge to skill level, part of that is the availability of mentoring, and I’m sure there are other parts I’m missing.
I enjoyed many things about Michael Zalewski’s book “The Tangled Web.” One thing I wanted was a better way to keep track of who attacks whom, to help me contextualize and remember the attacks. But such a model is not trivial to create. This morning, motivated by a conversation between Trey Ford and Chris Rohlf, I decided to take a stab at drafting a model for thinking about where the trust boundaries exist.
The words which open my threat modeling book are “all models are wrong, some models are useful.” I would appreciate feedback on this model. What’s missing, and why does it matter? What attacks require showing a new element in this software model?
[Update 1 — please leave comments here, not on Twitter]
- Fabio Cerullo suggests the layout engine. It’s not clear to me what additional threats can be seen if you add this explicitly, perhaps because I’m not an expert.
- Fernando Montenegro asks about network services such as DNS, which I’m adding and also about shared trust (CA Certs), which overlap with a question about supply chain from Mayer Sharma.
- Chris Rohlf points out the “web browser protection profile.
I could be convinced otherwise, but think that the supply chain is best addressed by a separate model. Having a secure installation and update mechanism is an important mitigation of many types of bugs, but this model is for thinking about the boundaries between the components.
In reviewing the protection profile, it mentions the following threats:
|Malicious updates||Out of scope (supply chain)|
|Malicious/flawed add on||Out of scope (supply chain)|
|Network eavesdropping/attack||Not showing all the data flows for simplicity (is this the right call?)|
|Data access||Local storage is shown|
Also, the protection profile is 88 pages long, and hard to engage with. While it provides far more detail and allows me to cross-check the software model, it doesn’t help me think about interactions between components.
End update 1]
A conversation with an old friend reminded me that there may be folks who follow this blog, but not the New School blog.
Over there, I’ve posted “Improving Security Effectiveness” about leaving Microsoft to work on my new company:
For the last few months, I’ve been working full time and talking with colleagues about a new way for security executives to measure the effectiveness of security programs. In very important ways, the ideas are new and non-obvious, and at the same time, they’re an evolution of the ideas that Andrew and I wrote about in the New School book that inspired this blog.
and about a job opening, “Seeking a technical leader for my new company:”
We have a new way to measure security effectiveness, and want someone who’ll drive to delivering the technology to customers, while building a great place for developers to ship and deploy important technology. We are very early in the building of the company. The right person will understand such a “green field” represents both opportunity and that we’ll have to build infrastructure as we grow.
This person might be a CTO, they might be a Chief Architect. They are certainly an experienced leader with strong references from peers, management and reports.
[There are broader critiques by Katie Moussouris of HackerOne at “Legally Blind and Deaf – How Computer Crime Laws Silence Helpful Hackers” and Halvar Flake at “Why changes to Wassenaar make oppression and surveillance easier, not harder.” This post addresses the free speech issue.]
During the first crypto wars, cryptography was regulated under the US ITAR regulations as a dual use item, and to export strong crypto (and thus, economically to include it in a generally available commercial or open source product) was effectively impossible.
A principle of our successful work to overcome those restrictions was that code is speech. Thus restrictions on code are restrictions on speech. The legal incoherence of the regulations was brought to an unavoidable crises by Phil Karn, who submitted both the book Applied Cryptography and a floppy disk with the source code from the book for an export license. The book received a license, the disk did not. This was obviously incoherent and Kafka-esque. At the time, American acceptance of incoherent, Kafka-esque rules was in much shorter supply.
Now, the new Wassenaar rules appear to contain restrictions on the export of a different type of code (page 209, category 4, see after the jump). (FX drew attention to this issue in this tweet. [Apparently, I wrote this in Jan, 2014, and forgot to hit post.])
A principle of our work was that code is speech. Thus restrictions on code are restrictions on speech. (Stop me if you’ve heard this one before.) I put forth several tweets that contain PoC I was able to type from memory, each of which, I believe, in principle, could violate the Wassenaar rules. For example:
- rlogin -froot $target
- echo wiz | nc $target 25
It would be nice if someone would file for the paperwork to export them on paper.
In this tweet, I’m not speaking for my employer or yours. I am speaking for poor, tired and hungry cryptographers, yearning to breathe free, and to not live on groundhog day.
For many years, I have been saying that “think like an attacker” is bad advice for most people. For example:
Here’s what’s wrong with think like an attacker: most people have no clue how to do it. They don’t know what matters to an attacker. They don’t know how an attacker spends their day. They don’t know how an attacker approaches a problem. Telling people to think like an attacker isn’t prescriptive or clear.
And I’ve been challenging people to think like a professional chef to help them understand why it’s not useful advice. But now, I’ve been one-upped, and, depending on audience, I have a new line to use.
Last week, on Veracode’s blog, Pete Chestna provides the perfect flip of “think like an attacker” to re-frame problems for security people. It’s “think like a developer.” If you, oh great security guru, cannot think like a developer, for heavens sake, stop asking developers to think like attackers.
There’s been a lot said in security circles about a talk on Tor being pulled from Blackhat. (Tor’s comments are also worth noting.) While that story is interesting, I think the bigger story is the lack of infrastructure for disclosure coordination.
Coordinating information about vulnerabilities is a socially important function. Coordination makes it possible for software creators to create patches and distribute them so that those with the software can most easily protect themselves.
In fact, the function is so important that it was part of why CERT was founded to: “coordinate response to internet security incidents.” Now, incidents has always been more than just vulnerabilities, but vulnerability coordination was a big part of what CERT did.
The trouble is, it’s not a big part anymore. [See below for a clarification added August 21.] Now “The CERT Division works closely with the Department of Homeland Security (DHS) to meet mutually set goals in areas such as data collection and mining, statistics and trend analysis, computer and network security, incident management, insider threat, software assurance, and more.” (Same “about” link as before.)
This isn’t the first time that I’ve heard about an issue where CERT wasn’t able to coordinate disclosure. I want to be clear, I’m not critiquing CERT or their funders here. They’ve set priorities and strategies in a way that makes sense to them, and as far as I know, there’s been precious little pressure to have a vuln coordination function.
It’s time we as a security community talk about the infrastructure, not as a flamewar over coordination/responsibility/don’t blow your 0day, but rather, for those who would like to coordinate, how should they do so?
Heartbleed is an example of what can happen with an interesting vulnerability and incomplete coordination. (Thanks to David Mortman for pointing that out in reviewing a draft.) Systems administrators woke up Monday morning to incomplete information, a marketing campaign, and a slew of packages that hadn’t been updated.
Disclosure coordination is hard to do. There’s a lot of project management and cross-organizational collaboration. Doing that work requires a special mix of patience and urgency, along with an unusual mix of technical skill with diplomatic communication. Those requirements mean that the people who do the work are rare and expensive. What’s more, it’s helpful to have these people seated at a relatively neutral party. (In the age of governments flooding money into cyberwar, it’s not clear if there are any truly neutral parties available. Some disclosure coordination is managed by big companies with a stake in the issue, which is helpful, but it’s hard for researchers to predict and depend apon.) These issues are magnified because those who are great at vulnerability research rarely spend time to develop those skills, and so an intermediary is even more valuable.
Even setting that aside, is there anyone who’s stepping up to the plate to help researchers effectively coordinate and manage disclosure?
[Update: I had a good conversation with some people at CERT/CC, in which I learned that they still do coordination work, especially in cases where the vendor is dealing with an issue for the first time, the issue is multi-vendor, or where there’s a conflict between vendor and researcher. The best way to do that is via their vulnerability reporting form, but if things don’t work, you can also email email@example.com or call their hotline (the number is on their contact us page.]
[Update 2, Dec 2014: Allen Householder has a really interesting model of “Vulnerability Coordination and Concurrency Modeling” on the CERT/CC blog, which shows some of the complexity involved, and touches on topics above. Oh, and some really neat Petri-dish state models.]