Diagrams in Threat Modeling

When I think about how to threat model well, one of the elements that is most important is how much people need to keep in their heads, the cognitive load if you will.

In reading Charlie Stross’s blog post, “Writer, Interrupted” this paragraph really jumped out at me:

One thing that coding and writing fiction have in common is that both tasks require the participant to hold huge amounts of information in their head, in working memory. In the case of the programmer, they may be tracing a variable or function call through the context of a project distributed across many source files, and simultaneously maintaining awareness of whatever complex APIs the object of their attention is interacting with. In the case of the author, they may be holding a substantial chunk of the plot of a novel (or worse, an entire series) in their head, along with a model of the mental state of the character they’re focussing on, and a list of secondary protagonists, while attempting to ensure that the individual sentence they’re currently crafting is consistent with the rest of the body of work.

One of the reasons that I’m fond of diagrams is that they allow the threat modelers to migrate information out of their heads into a diagram, making room for thinking about threats.

Lately, I’ve been thinking a lot about threat modeling tools, including some pretty interesting tools for automated discovery of existing architecture from code. That’s pretty neat, and it dramatically cuts the cost of getting started. Reducing effort, or cost, is inherently good. Sometimes, the reduction in effort is an unalloyed good, that is, any tradeoffs are so dwarfed by benefits as to be unarguable. Sometimes, you lose things that might be worth keeping, either as a hobby like knitting or in the careful chef preparing a fine meal.

I think a lot about where drawing diagrams on a whiteboard falls. It has a cost, and that cost can be high. “Assemble a team of architect, developer, test lead, business analyst, operations and networking” reads one bit of advice. That’s a lot of people for a cross-functional meeting.

That meeting can be a great way to find disconnects in what people conceive of building. And there’s a difference between drawing a diagram and being handed a diagram. I want to draw that out a little bit and ask for your help in understanding the tradeoffs and when they might and might not be appropriate. (Gary McGraw is fond of saying that getting these people in a room and letting them argue is the most important step in “architectural risk analysis.” I think it’s tremendously valuable, and having structures, tools and methods to help them avoid ratholes and path dependency is a big win.)

So what are the advantages and disadvantages of each?


  • Collaboration. Walking to the whiteboard and picking up a marker is far less intrusive than taking someone’s computer, or starting to edit a document in a shared tool.
  • Ease of use. A whiteboard is still easier than just about any other drawing tool.
  • Discovery of different perspective/belief. This is a little subtle. If I’m handed a diagram, I’m less likely to object. An objection may contain a critique of someone else’s work, it may be a conflict. As something is being drawn on a whiteboard, it seems easier to say “what about the debug interface?” (This ties back to Gary McGraw’s point.)
  • Storytelling. It is easier to tell a story standing next to a whiteboard than any tech I’ve used. A large whiteboard diagram is easy to point at. You’re not blocking the projector. You can easily edit as you’re talking.
  • Messy writing/what does that mean? We’ve all been there? Someone writes something in shorthand as a conversation is happening, and either you can’t read it or you can’t understand what was meant. Structured systems encourage writing a few more words, making things more tedious for everyone around.

Software Tools

  • Automatic analysis. Tools like the Microsoft Threat Modeling tool can give you a baseline set of threats to which you add detail. Structure is a tremendous aid to getting things done, and in threat modeling, it helps in answering “what could go wrong?”
  • Authority/decidedness/fixedness. This is the other side of the discovery coin. Sometimes, there are architectural answers, and those answers are reasonably fixed. For example, hardware accesses are mediated by the kernel, and filesystem and network are abstracted there. (More recent kernels offer filesystems in userland, but that change was discussed in detail.) Similarly, I’ve seen large, complex systems with overall architecture diagrams, and a change to these diagrams had to be discussed and approved in advance. If this is the case, then a fixed diagram, printed poster size and affixed to walls, can also be used in threat modeling meetings as a context diagram. No need to re-draw it as a DFD.
  • Photographs of whiteboards are hard to archive and search without further processing.
  • Photographs of whiteboards may imply that ‘this isn’t very important.” If you have a really strong culture of “just barely good enough” than this might not be the case, but if other documents are more structured or cared for, then photos of a whiteboard may carry a message.
  • Threat modeling only late. If you’re going to get architecture from code, then you may not think about it until the code is written. If you weren’t going to threat model anyway, then this is a win, but if there was a reasonable chance you were going to do the architectural analysis while there was a chance to change the architecture, software tools may take that away.

(Of course, there are apps that help you take images from a whiteboard and improve them, for example, Best iOS OCR Scanning Apps, which I’m ignoring for purposes of teasing things out a bit. Operationally, probably worth digging into.)

I’d love your thoughts: are there other advantages or disadvantages of a whiteboard or software?

Phishing and Clearances

Apparently, the CISO of US Homeland Security, a Paul Beckman, said that:

“Someone who fails every single phishing campaign in the world should not be holding a TS SCI [top secret, sensitive compartmentalized information—the highest level of security clearance] with the federal government” (Paul Beckman, quoted in Ars technica)

Now, I’m sure being in the government and trying to defend against phishing attacks is a hard problem, and I don’t want to ignore that real frustration. At the same time, GAO found that the government is having trouble hiring cybersecurity experts, and that was before the SF-86 leak.

Removing people’s clearances is one repsonse. It’s not clear from the text if these are phishing (strictly defined, an attempt to get usernames and passwords), or malware attached to the emails.

In each case, there are other fixes. The first would be multi-factor authentication for government logins. This was the subject of a push, and if agencies aren’t at 100%, maybe getting there is better than punitive action. Another fix could be to use an email client which makes seeing phishing emails easier. For example, an email client could display the RFC-822 sender address (eg, “<account.management@gmail.com>” for any email address that that email client hasn’t sent email to, rather than the friendly text. They could provide password management software with built-in anti-phishing (checking the domain before submitting the password. They could, I presume, do other things which minimize the request on the human being.

When Rob Reeder, Ellen Cram Kowalczyk and I created the “NEAT” guidance for usable security, we didn’t make “Necessary” first just because the acronym is neater that way, we put it first because the poor person is usually overwhelmed, and they deserve to have software make the decisions that software can make. Angela Sasse called this the ‘compliance budget,’ and it’s not a departmental budget, it’s a human one. My understanding is that those who work for the government already have enough things drawing on that budget. Making people anxious that they’ll lose their clearance and have to take a higher-paying private sector job should not be one of them.

Towards a model of web browser security

One of the values of models is they can help us engage in areas where otherwise the detail is overwhelming. For example, C is a model of how a CPU works that allows engineers to defer certain details to the compiler, rather than writing in assembler. It empowers software developers to write for many CPU architectures at once. Many security flaws happen in areas the models simplify. For example, what if the stack grew away from the stack pointer, rather than towards it? The layout of the stack is a detail that is modeled away.

Information security is a broad industry, requiring and rewarding specialization. We often see intense specialization, which can naturally result in building expertise in silos, and tribal separation of knowledge create more gaps. At the same time, there is a stereotype that generalists end up focused on policy, or “risk management” where a lack of technical depth can hide. (That’s not to say that all risk managers are generalists or that there’s not real technical depth to some of them.)

If we want to enable more security generalists, and we want those generalists to remain grounded, we need to make it easier to learn about new areas. Part of that is good models, part of that is good exercises that appropriately balance challenge to skill level, part of that is the availability of mentoring, and I’m sure there are other parts I’m missing.

I enjoyed many things about Michael Zalewski’s book “The Tangled Web.” One thing I wanted was a better way to keep track of who attacks whom, to help me contextualize and remember the attacks. But such a model is not trivial to create. This morning, motivated by a conversation between Trey Ford and Chris Rohlf, I decided to take a stab at drafting a model for thinking about where the trust boundaries exist.

The words which open my threat modeling book are “all models are wrong, some models are useful.” I would appreciate feedback on this model. What’s missing, and why does it matter? What attacks require showing a new element in this software model?

Browser security
[Update 1 — please leave comments here, not on Twitter]

  1. Fabio Cerullo suggests the layout engine. It’s not clear to me what additional threats can be seen if you add this explicitly, perhaps because I’m not an expert.
  2. Fernando Montenegro asks about network services such as DNS, which I’m adding and also about shared trust (CA Certs), which overlap with a question about supply chain from Mayer Sharma.
  3. Chris Rohlf points out the “web browser protection profile.

I could be convinced otherwise, but think that the supply chain is best addressed by a separate model. Having a secure installation and update mechanism is an important mitigation of many types of bugs, but this model is for thinking about the boundaries between the components.

In reviewing the protection profile, it mentions the following threats:

Threat Comment
Malicious updates Out of scope (supply chain)
Malicious/flawed add on Out of scope (supply chain)
Network eavesdropping/attack Not showing all the data flows for simplicity (is this the right call?)
Data access Local storage is shown

Also, the protection profile is 88 pages long, and hard to engage with. While it provides far more detail and allows me to cross-check the software model, it doesn’t help me think about interactions between components.

End update 1]

On Language

I was irked to see a tweet “Learned a new word! Pseudoarboricity: the number of pseudoforests needed to cover a graph. Yes, it is actually a word and so is pseudoforest.” The idea that some letter combinations are “actual words” implies that others are “not actual words,” and thus, that there is some authority who may tell me what letter combinations I am allowed to use or understand.

Balderdash. Adorkable balderdash, but balderdash nonetheless.

As any student of Orwell shall recall, the test of language is its comprehensibility, not its adhesion to some standard. As an author, I sometimes hear from people who believe themselves to be authorities, or who believe that they may select for me authorities as to the meanings of words, and who wish to tell me that my use of the word “threat” threatens their understanding, that the preface’s explicit discussion of the many plain meanings of the word is insufficient, or that my sentences are too long, comma-filled, dash deficient or otherwise Oxfordless in a way which seems to cause them to feel superior to me in a way they wish to, at some length, convey.

In fact, on occasion, they are irked. I recommend to them, and to you, “You Are What You Speak.”

I wish them the best, and fall back, if you’ll so allow, to a comment from another master of language, speaking through one of his characters:

‘When I use a word,’ Humpty Dumpty said, in rather a scornful tone, ‘it means just what I choose it to mean — neither more nor less.’
‘The question is,’ said Alice, ‘whether you can make words mean so many different things.’
‘The question is,’ said Humpty Dumpty, ‘which is to be master — that’s all.’

Conference Etiquette: What’s New?

So Bill Brenner has a great article on “How to survive security conferences: 4 tips for the socially anxious
.” I’d like to stand by my 2010 guide to “Black Hat Best Practices,” and augment it with something new: a word on etiquette.

Etiquette is not about what fork you use (start from the outside, work in), or an excuse to make you uncomfortable because you forgot to call the Duke “Your Grace.” It’s a system of tools to help otherwise awkward social interactions go more smoothly.

We all meet a lot of people at these conferences, and there’s some truth behind the stereotype that people in technology are bad at “the people skills.” Sometimes, when we see someone, there will be recognition, but the name and full context doesn’t come rushing back. That’s an awkward moment, and it’s worth thinking about the etiquette involved.

When you know you’ve met someone and can’t recall the details, it’s rude to say “remind me who you are,” and so people will do a bunch of things to politely encourage reminders. For example, they’ll say “what’s new” or “what have you been working on lately?” Answers like “nothing new” or “same old stuff” are not helpful to the person who asked. This is an invitation to talk about your work. Even if you haven’t done anything new that’s ready to talk about, you can say something like “I’m still exploring the implications of the work I did on X” or “I’ve wrapped up my project on Y, and I’m looking for a new thing to go frozzle.” If all your work is secret, you can say “Oh, still at DoD, doing stuff for Uncle Sam.”

Whatever your answer will be, it should include something to help people remember who you are.

Why not give it a try this RSA?

BTW, you can get the best list of RSA parties where you can yell your answers to such questions at “RSA Parties Calendar.”

IOS Subject Key Identifier?

I’m having a problem where the “key identifier” displayed on my ios device does not match the key fingerprint on my server. In particular, I run:

% openssl x509 -in keyfile.pem -fingerprint -sha1

and I get a 20 byte hash. I also have a 20 byte hash in my phone, but it is not that hash value. I am left wondering if this is a crypto usability fail, or an attack.

Should I expect the output of that openssl invocation to match certificate details on IOS, or is that a different hash? What options to openssl should produce the result I see on my phone?

[update: it also does not match the output or a trivial subset of the output of

% openssl x509 -in keyfile.pem -fingerprint -sha256

% openssl x509 -in keyfile.pem -fingerprint -sha512


[Update 2: iOS displays the “X509v3 Subject Key Identifier”, and you can ask openssl for that via -text, eg, openssl x509 -in pubkey.pem -text. Thanks to Ryan Sleevi for pointing me down that path.]

Think Like An Attacker? Flip that advice!

For many years, I have been saying that “think like an attacker” is bad advice for most people. For example:

Here’s what’s wrong with think like an attacker: most people have no clue how to do it. They don’t know what matters to an attacker. They don’t know how an attacker spends their day. They don’t know how an attacker approaches a problem. Telling people to think like an attacker isn’t prescriptive or clear.

And I’ve been challenging people to think like a professional chef to help them understand why it’s not useful advice. But now, I’ve been one-upped, and, depending on audience, I have a new line to use.

Last week, on Veracode’s blog, Pete Chestna provides the perfect flip of “think like an attacker” to re-frame problems for security people. It’s “think like a developer.” If you, oh great security guru, cannot think like a developer, for heavens sake, stop asking developers to think like attackers.

What to do for randomness today?

In light of recent news, such as “FreeBSD washing Intel-chip randomness” and “alleged NSA-RSA scheming,” what advice should we give engineers who want to use randomness in their designs?

My advice for software engineers building things used to be to rely on the OS to get it right. That defers the problem to a small number of smart people. Is that still the right advice, despite recent news? The right advice is pretty clearly not that a normal software engineer building in Ruby on Rails or asp.net should go and roll their own. It also cannot be that they spend days wading through debates. Experts ought to be providing guidance on what to do.

Is the right thing to hash together the OS and something else? If so, precisely what something else?

A Quintet of Facebook Privacy Stories

It’s common to hear that Facebook use means that privacy is over, or no longer matters. I think that perception is deeply wrong. It’s based in the superficial notion that people making different or perhaps surprising privacy tradeoffs are never aware of what they’re doing, or that they have no regrets.

Some recent stories that I think come together to tell a meta-story of privacy:

  • Steven Levy tweeted: “What surprised me most in my Zuck interview: he says the thing most on rise is ‘sharing with smaller groups.'” (Tweet edited from 140-speak). I think that sharing with smaller groups is a pretty clear expression that privacy matters to Facebook users, and that as Facebook becomes more a part of people’s lives, the way they use it will continue to mature. For example, it turns out:
  • 71% of Facebook Users Engage in ‘Self-Censorship’” did a study of people typing into the Facebook status box, and not hitting post. In part this may be because people are ‘internalizing the policeman’ that Facebook imposes:
  • Facebook’s Online Speech Rules Keep Users On A Tight Leash.” This isn’t directly a privacy story, but one important facet of privacy is our ability to explore unpopular ideas. If our ability to do so in the forum in which people talk to each other is inhibited by private contract and opaque rules, then our ability to explore and grow in the privacy which Facebook affords to conversations is inhibited.
  • Om Malik: “Why Facebook Home bothers me: It destroys any notion of privacy” An interesting perspective, but Facebook users still care about privacy, but will have trouble articulating how or taking action to preserve the values of privacy they care about.

The Psychology of Password Managers

As I think more about the way people are likely to use a password manager, I think there’s real problems with the way master passwords are set up. As I write this, I’m deeply aware that I’m risking going into a space of “it’s logical that” without proper evidence.

Let’s start from the way most people will likely come to a password manager. They’ll be in an exploratory mood, and while they may select a good password, they may also select a simple one that’s easy to remember. That password, initially, will not be protecting very much, and so people may be tempted to pick one that’s ‘appropriate’ for what’s being protected.

Over time, the danger is that they will not think to update that password and improve it, but their trust in the password manager will increase. As their trust increases, the number of passwords that they’re protecting with a weak master password may also increase.

Now we get to changing the master password. Assuming that people can find it, how often will someone wake up and say “hey, I should change my master password?” Changing a master password is also scary. Now that I’ve accumulated hundreds of passwords, what happens if I forget my new password? (As it turns out, 1Password makes weekly backups of my password file, but I wasn’t aware of that. Also, what happens to the old files if I change my master password? Am I now exposed for both? That’s ok in the case that I’m changing out of caution, less ok if I’m changing because I think my master was exposed.)

Perhaps there’s room for two features here: first, that on password change, people could choose to have either master password unlock things. (Encrypt the master key with keys derived from both the old & new masters. This is no less secure than having backups available, and may address a key element of psychological acceptability.) You’d have to communicate that this will work, and let people choose. User testing that text would be fascinating.

A second feature might be to let people know how long they’ve been using the same master password, and gently encourage them to change it. This one is tricky mostly because I have no idea if it’s a good idea. Should you pick one super-strong master and use it for decades? Is there value to changing it now and again? Where could we seek evidence with which to test our instincts? What happens to long term memory as people age? Does muscle memory cause people to revert their passwords? (I know I’ve done it.) We could use a pattern like the gold bar to unobtrusively prompt.

A last element that might improve the way people use master passwords would be better browser integration. Having just gone to check, I was surprised how many sites my browser is tracking. Almost all of them were low value, and all of them now are. But why do we have two places that can store this, especially when one is less secure than the other. A browser API that allows a password manager to say “I’ve got this one” would be a welcome improvement.

Studying these ideas and seeing which ones are invalidated by data gathering would be cool. Talking to people about how they use their password managers would also be interesting work. As Bonneau has show, the quest to replace passwords is going to be arduous. Learning how to better live with what we have seems useful.

1Password & Hashcat

The folks at Hashcat have some interesting observations about 1Password. The folks at 1Password have a response, and I think there’s all sorts of fascinating lessons here.

The crypto conversations are interesting, but at the end of the day, a lot of security is unavoidably contributed by the master password strength. I’d like to offer up a simple contribution. Agilebits should make two non-cryptographic changes in addition to any crypto changes.

These relate to the human end of the issue, and how real humans make decisions. That is, picking a master password is a one time event, and even if there’s a strength meter, factors of memorability, typability, etc all come into play when the user selects a password when first installing 1Password.

Those human factors are not good for security, but I think they’re addressable.

First, the master password entry screens should display the same password strength meter that’s displayed everywhere else. It’s all well and good to discuss in a blog post that people need strong master passwords, but the software should give regular feedback about the strength of that master password. Displaying a strength meter each time it’s entered creates some small risk of information disclosure via shoulder-surfing, and adds pressure to make it stronger.

Second, they should make it easier to change the master password. I looked around, couldn’t figure out how to do so in a few minutes. [Update: It’s in preferences, security. I thought I’d looked there, may have missed it.]


If master passwords are so important, then it’s important for the software to help its customers get them right.

There’s an interesting link here to “Why Johnny Can’t Encrypt.” In that 1999 paper, Whitten and Tygar made the point that all the great crypto in PGP couldn’t protect its users if they didn’t make the right decisions, and making those decisions is hard.

In this case, the security of password vaults depends not only on the crypto, but also on the user interface. Figuring out the mental models that people have around password storage tools, and how the interface choices those tools make develop those mental models is an important area, and deserves lots of careful attention.

Gamifying Driving

P90115441 highRes 640x419

…the new points system rates the driver’s ability to pilot the MINI with a sporty yet steady hand. Praise is given to particularly sprightly sprints, precise gear changes, controlled braking, smooth cornering and U-turns executed at well-judged speeds. For example, the system awards maximum Experience Points for upshifts carried out within the ideal rev range and in less than 1.2 seconds. Super-slick gear changes prompt a “Perfect change up” message on the on-board monitor, while a “Breathtaking U-turn” and a masterful touch with the anchors (“Well-balanced braking”) are similarly recognised with top marks and positive, MINI-style feedback.

For more, see “MINI Connected Adds Driving Excitement Analyser.”

Now, driving is the most dangerous thing most of us do on a regular basis. Most Americans don’t get any supplemental driving instruction after they turn 17. So maybe there’s actually something to be said for a system that incents people to drive better.

I can’t see any possible issues with a game pushing people towards things that are undesirable in the real world. I mean, I’m sure that before suggesting a U-turn, the game will use the car’s adaptive cruise control radar to see what’s around, even if the car doesn’t have one.

Happy Data Privacy Day! Go check out PrivacyFix

It’s Data Privacy Day, and there may be a profusion of platitudes. But I think what we need on data privacy day are more tools to let people take control of their privacy. One way to do that is to check your privacy settings. Of course, the way settings are arranged changes over time, and checking your settings regularly is a drain.

Enter PrivacyFix.

PrivacyFix is a Firefox & Chrome plugin that you might want to check out. It looks at your Facebook and G+ settings, and helps you fix things. It also helps you send opt-out email to web site privacy addresses, which is awesome.

Not having a Facebook or G+ account, I can’t really test it. I do find the model of a plugin that works when you’re on their site (versus local UI) to be confusing. But maybe I’m not their target audience. Anyway, I did want to refer back to my Lessons from Facebook’s Stock Slide, in which I talked about intent versus identity.

Facebook tracks
Google tracks

I don’t know if PrivacyFix’s estimates of revenue are accurate. But unless they’re off by 2 orders of magnitude for each of Facebook (under-estimating) and Google (over-estimating), then wow.

Can Science Improvise?

My friend Raquell Holmes is doing some really interesting work at using improv to unlock creativity. There’s some really interesting ties between the use of games and the use of improv to get people to approach problems in a new light, and I’m bummed that I won’t be able to make this event:

Monday Dec 17th – 7:15 to 9:15pm
835 Market Street, Rm. 619, Downtown San Francisco State University Campus

Register at http://www.acteva.com//booking.cfm?bevaid=234451
In advance- $15 At the Door- $20

What happens when you combine the playfulness of improvisation with
the rigor of science? The Life Performance Coaching Center which
leads people from all walks of life in a performance-based approach to
human development is pleased to host Dr. Raquell M. Holmes founder of
improvscience. Holmes has been bringing the discoveries in human
development and performance to researchers and educators in many areas
of science including biology and computing sciences.

In this exploration for scientists and those interested in creativity
and development, participants are introduced to what the
improvisational arts bring to science. Learning to build with the
contributions of others and see opportunities for improvisational
conversation helps us to take risks and discover new ways of seeing
each other and our work.

Come and play as we break down the social barriers that can inhibit
creativity, exploration and discovery.

Helen Abel, LCSW, has worked with people to develop their lives for
over 30 years as a social worker, therapist and coach. She is on the
staff of the Life Performance Coaching Center where she leads the
popular Playground series {link if available} where people learn how
to use their capacity to create, perform and play. As a life coach she
helps people access these same skills to develop creative and new
kinds of conversations with their friends, family and colleagues.

Dr. Raquell Holmes is Director of Outreach, Recruitment and Retention
at the Center for Cell Analysis and Modeling at University of
Connecticut Health Center. She helps biologists to incorporate
computing and computational resources into their teaching and
research. Community building and improvisational theater are explicit
components of the majority of her National Science Foundation funded
projects. She founded improvscience to provide scientists with
opportunities to develop skills in leadership, collaboration and
innovation. Since its inception improvscience has worked with over a
thousand professionals in Science, Technology, Engineering and
Mathematics education and research.

When an interrupt is important

So it’s cool that this “S.M.A.R.T” stuff tells the computer when the hard drive is failing. The next step in user interface is to take the message out of /Applications/Utilities/Disk Utility and into an interruptive UI, so that I don’t discover this problem when I happen to get an extra drive for backup.

I know Apple knows how to interrupt the user when it matters to them, because iTunes always gives me two chances to enter my password so it can auto-update things. Maybe they’re hoping I won’t notice this one and just figure I need a new machine:

Disk Utility