Boundary Objects and Threat Modeling

threat model dfd.jpg
Ethonomethodologists talk a lot about communities of practice. Groups of people who share some set of work that they do similarly, and where they’ll co-evolve ways of working and communicating.


When everyone is part of a given community, this works really well. When we talk aboutthink like an attacker” within a community of security practice, it works well. When we tell developers to do that, they look like a deer in the headlights. (Sorry, couldn’t resist.)

One of the tools which different communities of practice can use to communicate is a boundary object. Boundary objects include things like ISBNs. Books have ISBNs in large part to track payments. They differ from Library of Congress catalog numbers. 0321502787, HD30.2.S563 and “The New School of Information Security” all refer to the same book in different contexts.

In STRIDE/Element threat modeling, there are two accidental boundary objects. (I learned about the theory after developing the approach.) They are data flow diagrams (DFDs) and bugs. The picture is a DFD, showing the process of threat modeling, along with boundaries. The boundaries are doing double duty as trust boundaries, and bi-secting the boundary objects.

The DFD acts as a boundary object because it’s simple. It takes about 30 seconds to learn (except for trust boundaries). It looks a lot like most whiteboard diagrams. Developers can draw the diagram, and security experts can analyze it.

The second boundary object is the bug database. Everyone in software development understands bug databases. And though the practices which surround them differ pretty markedly, almost no one would ship a product without reviewing their bugs, which is why security people like putting the output of a threat modeling session into the database.

There are other possible boundaries, such as the interface between the business and the software. This is where assets come into some threat modeling approaches.

So what’s the takeaway here? If you’re watching groups of people frustratedly talk past each other — or wishing they’d be that communicative — look to see if you can find boundary objects which they can use to help organize conversation.

Identities are Created Through Relationships

identity.jpg
I’m listening to this really interesting podcast by Bob Blakley and Phil Windley. What really struck me was where Bob said “thinking of identity as an artifact all by itself is unsatisfactory because we can talk about an identity and the attributes of an identity leaves out important details about how identities are created and how they evolve…relationships are the landscapes in which identities exist.” I think this is interesting, but I’m reading a paper about ethnomethodology and information security. One of the claims it makes is that meaning is created through conversation, and that a history of conversation and shared reference points gives us an ability to converse in meaningful ways. When someone says we’re talking past each other, what they may mean is that the conversation lacks sufficient shared context to be meaning-full.

So I’d like to fuse these ideas, and propose that identity is created through relationships. That without relationships, identities actually don’t exist. In the pathological cases of solitary confinement or hermitage, identity is severely stressed or destroyed.

I think people understand this instinctively, although perhaps not formulated as I’ve said it. Who a child spends time with shapes them, for good or ill. What parent doesn’t ask to meet their children’s new friends? The relationships create identity. As people age, and intimate relationships end either by breakup or death, people say they feel like they’ve lost a part of themselves.

As regular readers know, I’m concerned about the impact of replacing personal relationships with dossiers, algorithms and their implementations, like background checks, the use of credit scores everywhere, etc. Dossiers and databases are fed by organizations with whom we have a relationship. But the relying parties often have no relationship with us. They start their relationship defining us by the contents of dossiers, and it impinges on our sense of self. Our identities are set aside. There’s no relationship, there’s no conversation, and we feel either elated — “they like my file me!” or dejected “what’s wrong with me?” This displacement also drives the emotional response to identity theft. We’re upset that the person or organization we’re talking to is confused about who we are. They’re confused because the dossier is confused, and the dossier is confused because of a web of relationships which are hard to see or understand. The relationship re-creates our identity.

The third place I’d like to look is the rise of new forms of ‘loosely coupled’ technological relationships, perhaps first created by usenet, and now visible in places like Tribe, Facebook or MySpace. Here, we see people presenting their identity — in part — by how many ‘friends’ they have. There’s also an element of restoration of older identities — reconnecting with a boy scout troop, high school friends — all relationships that contribute to identity.

In “The Presentation of Self in Everyday Life,” the idea is that we create personas to control relationships. From lawyers to doctors to waitstaff or auto mechanics, people present a view into their identity that makes sense. I would question if I want to give business to an auto mechanic who was reading the Harvard Law Review when I came in, or a lawyer who was reading a Chilton’s repair manual. People present themselves in certain ways to control the perception of ‘who they are,’ and so a professional relationship develops in the right way.

I also want to look at privacy in the sense of Schoeman’s “Privacy and Social Freedom.” Schoeman looks at privacy as essential to freedom because it allows us to explore ideas without having to ‘answer’ for them. If we have a conversation with a friend, we need to worry less about saying dumb things, because the conversation is private. We explore and shape our identity within relationships and through those we’ve chosen to trust.

So next time someone talks about identity or identity management, ask yourself, what are the assumptions about the relationship? And when you hear someone talking about ‘customer relationship management,’ as yourself what identity they seem to want to manage.

Photo: Which one, by BeViewed.

[Update: Corrected spelling errors, including someone’s name. I am the king of spelling errors today!]

The Presentation of Self in Everyday Tweeting

Chris Hoff pointed to an interesting blog post from Peter Shankman. Someone* tweeted “True confession but I’m in one of those towns where I scratch my head and say ‘I would die if I had to live here!'”

Well it turns out that…

Not only did an employee find it, they were totally offended by it and responded to the agency person. The kicker is that they copied the FedEx Coporate Vice President, Vice President, Directors and all management of FedEx’s communication department AND the chain of command at (his employer).

Now, the twit who tweeted was clearly a twit, having mixed business and personal in a way that offended a major client. But let’s step back.

First, it’s important to remember that we all have personal lives, and it’s a good thing to be able to separate them from our work lives. If you work in IT and want to blog about gardening, no one is going to confuse things. Where it gets a little grey is when we’re deeply enthused about our work. I blog under my real name about topics that impact my employer. Not all–there are posts that haven’t seen the light of day because they’re too close. Sometimes, I cover work here when I’m really excited about it. My co-workers at Microsoft and my colleagues at Waggener Edstrom also understand that Emergent Chaos is separate, and have never asked me to post anything here.

Second, I think it’s important to generate a zone of professionalism where we it is seen as reasonable for seasoned professionals to comment on things which impact their employers without a presumption that they speak for their employer. This is not without challenges. If we’re naive about it, we create a zone of shills where people are paid to speak for their employers, and lie. At the same time, there are people with a degree of experience, maturity, and wisdom where you want them to be free to speak. Similarly, Microsoft’s willingness to accept my continued posting here without a lot of oversight made me happier in accepting their job. There are lots of companies which would have said “no way.”

Third, I think you need to telegraph where difference is. Here, it’s very clear that we speak for the President of the United States, not our employers. When I mention Microsoft, I try to be clear, although in reviewing posts, I seem to have fallen down a little. A post like “SDL Announcements” is pretty clearly me speaking about work:

I’m in Barcelona, where my employer has made three announcements about our Security Development Lifecycle, which you can read about here…I’m most excited about the public availability of the SDL Threat Modeling Tool. I’ve been working on this for the last 18 months…

(Speaking of clear, not all of the posts in the category are by me.)

The title is of course, a reference to the classic work of sociology, in which Goffman explains that we all present different facets of ourselves in different contexts. In blurring these contexts, services like Twitter and Facebook present a serious challenge to how we conceptualize and present ourselves.

Chaos in the Airports! Baa! Baa!

TSA Badges.jpg
Some days the snark just writes itself:

The group that created Smokey Bear and McGruff the Crime Dog has a new potential icon: Stephanie the airport screener.

A $1.3 million ad campaign launched this month teams the Ad Council and the Transportation Security Administration trying to change behavior of passengers who no longer automatically accept post-Sept. 11 airport security procedures. The public relations push explains the terrorist threat and the reasons behind annoyances at checkpoints.

A passenger focus group conducted for TSA by New York City business consulting firm Blue Lime found that “unquestioning compliance has diminished.” Passengers say they are more afraid of missing their flight than they are of an airplane being attacked, the 73-page Blue Lime report found. (“TSA ads aim to get fliers on board with security measures,” USA Today)

Stephanie Naar has been in the news before, as part of the TSA’s wasting our money on jackboots badges. Not sure (yet) if the image is her. I’ll snap a pic if I see the ad.

PS to TSA: there’s a good reason McGruff and Smokey were animated.

Photo by Paul J. Richards / AFP / Getty. Races purely coincidental.

Public Policy and InfoSec

…Armed with my favorite govie (who is actually the lead on this, I’m just a straphanger), The New School of Information Security (Hi Adam and Andrew), some government policy directives, and the National Strategy to Secure Cyberspace, I am teaching an Information Security Management and Public Policy class for Carnegie Mellon’s Heinz School.

The more I work with the Masters of Science in Public Policy Management program, the more I’m sold on it. Basically the students do a year on-campus in Pittsburg, then they have the option of staying there or coming to DC. The students who come to DC work a 32-hour week (some do more), 2 night classes, and class for most of Friday. Our information security class fits in as a sector-specific deep-dive, the other one being healthcare (which needs smart public policy people, too).

Which is where we need some help. It’s a little behind the game, but we’re constantly looking for Government agencies, NGOs/NPOs, and contractors who are interested in taking on interns. Even better if you have jobs that don’t have a US citizenship requirement. If you want to be linked up, just drop me a line.

First, thank you! Andrew and I are both tremendously excited to see the New School being used at CMU. If anyone knows of internships to help their students find jobs, please visit “The Guerilla CISO” and let’s help them out?

Confirmation Bias and Newspaper Endorsements

We’ve been talking a lot lately about confirmation bias. It turns out that newspaper endorsements are more influential when they are unexpected.

The degree of this influence, however, depends upon the credibility of the endorsement. In this way, endorsements for the Democratic candidate from left-leaning newspapers are less influential than are endorsements from neutral or right-leaning newspapers…

Via the Economist Free Exchange blog, after the newspaper credibly endorsed Obama.

Previously on confirmation bias: “Things only an astrologist could believe,” “No evidence the data was misused,” and “More on confirmation bias.”

Thoughts about Democracy in America

There’s a place in de Tocqueville where he talks about America’s civic strength coming from the way we organize: those voluntary organizations which come together to solve a problem as a community. He pointed out that what we got from that was not merely that particular problem solved, but a sense of community and a willingness to solve problems without the heavy hand of government.

I am tremendously inspired by stories like “Daughter of slave votes for Obama.” There’s real progress for our country, within the course of a lifetime.

I’ve watched as a number of my friends have gone all out for Obama, some traveling on their own dime to knock on doors in states less blue than their own. I’m glad to see that level of enthusiasm: a politics of petty attacks is very likely to lose tomorrow, where a McCain who had been “the McCain of 2000” might well have won.

I worry about Obama’s views on national service, including his goal of 50 hours of community service from every middle and high school student, and his goal of federalizing non-profits. I think that the value of non-profits comes from their volunteer nature, and from their diverse goals. Federal dollars will be alluring for their sheer scale. They will also be distracting for many non-profits, forced, like many churches to strangely bifurcate their activity to allow for federal dollars to flow in. As de Tocqueville understood, much of the value of volunteerism — including volunteering for a political candidate — is that it brings us together as a civic society.

As I watch the outpouring of enthusiasm and of hope, I am hopeful that Obama is smart enough to understand that the real strength of our nation is not in Washington, and it’s not in directives from Washington. It’s from hundreds of millions of people pursuing their hopes and dreams. America is a diverse set of people with different hopes and different dreams, and the value of our democracy is that is has embraced and promoted the freedom of each of us to pursue our own dreams, chaotic though that may be.

CTOs, Product Management and Program Management

In “The product manager’s lament,” Eric Ries writes about his view of product managers:

Let’s start with what the product manager does. He’s supposed to be the person who specifies what the product will do. He writes detailed specs which lay out exactly what features the team should build in its next iteration. These specs are handed to a designer, who builds layouts and mockups of all the salient points. Then the designs are handed to a team of programmers with various specialties.

When I met this team, some acrimony had built up. The last few features came out pretty different from what was origianlly spec’d, and took far too long, to boot. The programmers keep asking for more say in the designs and direction that they work on.

I think Eric is almost right about what a product manager should do. I want to provide two disparate perspectives on what that almost entails, and why it’s important. First, I’d like to talk about the role of the program manager at Microsoft (my current day job) and then about the role of the startup CTO (my previous day job).

The program manager’s job is to understand the market and customer pain, shape consensus around what a solution looks like, spec that solution, then drive implementation and the inevitable tradeoffs and ship a solution which makes customers happy.* I do all of that in creating the SDL threat modeling tool.

Some people think the market approach is strange because inside Microsoft, the SDL requires threat modeling. But most markets are distorted in some way by legal requirements. I treat threat modeling as a market with pain that I need to address, and do my best to win in that market. I’m fairly pedantic about talking about our customers, rather than our users, because we give them better tools, and make them more successful when we treat them as valued customers.

Note that that is a super-set of Eric’s description of what a product manager does. He has some interesting suggestions, but the real fix is to get the guy who owns the spec deeply involved in the software process, from start to finish. Which brings me to the role of the CTO.

The role of a good CTO is to understand the market and customer pain, shape consensus around what a solution looks like, spec that solution, then drive implementation and the inevitable tradeoffs and ship a solution which makes customers happy. There’s also a responsibility to be a company leader, hiring, shaping the culture, and participating in the executive decisions the company makes. Sometimes, there’s a need to step in and build. But a large part of the CTO role is that of the program manager. I think this is why I’m able to succeed as a program manager—I’ve been at it for a while.

In Eric’s post last month, “What does a startup CTO actually do?,” he provided a different list: platform selection and technical design; seeing the big picture; providing options; finding the 80/20 and growing technical leaders. I think that’s a good list, but it’s missing a key piece, which is the vision to bits to customer experience scope that is at the core of the program management mindset.

[Update: The * was going to be a footnote citing an internal doc which I’m paraphrasing, but I decided to cut it, and forgot to remove the *. Oops!]

Discipline and Art

Stephan Bugaj has a fascinating article up, “Steve Kurtz: Tactical Art.” I wanted to tie this to my post “The Discipline of ‘think like an attacker’

Kurtz only briefly mentioned his four year ordeal with the Department of Justice (this is also a good article about it), and only as a single exemplar of his overall thesis that the role of art is to push back against the social mechanisms of what he’s termed “expression management.”

In staging this mock bioweapon release in front of the U.S. Embassy, what Kurtz found was that his own internal microfascisms were causing him to attempt to derail his own project by listing things he was sure they wouldn’t be allowed to do: march and then assemble in front of the embassy, then use a city tower to release the smoke with the (harmless) biological sample in it, and then bring skin samples from the participants to a lab for testing.

What he found instead was that the Leipzigers, despite Germany’s decades longer ordeal with terrorism (from not just Islamists, but also neo-Nazis and Communists), were quite willing to support the project. When the sponsoring Leipzig arts institution asked, the city gave them use of the tower, and permission to march to and in-front of the embassy, with no fuss. The biological laboratory in the city was equally obliging.

It’s a very interesting post about the intersection of art with ‘the policeman within.’ The lecturer in question has certainly had enough encounters with the policemen to have developed an interesting orientation towards their relationship with society.

In security engineering work, we often have to overcome internal filters, such as “why would anyone do that?” I think that powerful art, like that of Banksy or Wendy Richmond has an ability to transform the way we see the world for the better. It’s a shame when our artists need to contend with arrest for doing things which are not illegal, but merely confusing to our armed public servants.

Previously on Emergent Chaos: Banksy on anonymity, England, and Disneyland.

The Costs of Secrecy

Security continues to be crippled by a conspiracy of silence. The ongoing costs of not talking about what’s going wrong are absolutely huge, and today, we got insight into just how huge.


Richard Clayton and Tyler Moore of Cambridge University have a new paper on phishing, “The consequence of non-cooperation in the fight against phishing.” In it, they look at how phishing sites are taken down, and estimate how much faster it would be if there were better sharing of data. From their blogpost:

Since extended lifetimes equate to more unsuspecting visitors handing over their credentials and having their bank accounts cleaned out, these delays can also be expressed in monetary terms. Using the rough and ready model we developed last year, we estimate that an extra $326 million per annum is currently being put at risk by the lack of data sharing. This figure is from our analysis of just two companies’ feeds, and there are several more such companies in this business.

I haven’t had time to read the paper in depth, but I have a lot of respect for both Richard and Tyler. Have you read the paper? Impressions? (Here or on their blog.)

Security is an Empirical and Social Science

In reading Mordaxus’ post “Quantum Crypto Broken Again,” I was struck by his comment:

It is a serious flaw because one of the main arguments about quantum cryptography is that because it is “physics” based as opposed to “computer” based, that it is more secure than software cryptography.”

Firstly, security is almost always an outcome of the combination of science, engineering and the socio-legal context in which the engineering is deployed. Let’s assume that the science and engineering on the SUX-8000 Quantum Key Distributor are perfect, and the SUX has t three lights: power, carrier and tampering. When the tampering light starts blinking, one of two things can happen. First, Alice will continue to use the bits, because her operations manual doesn’t say what to do. Alternately, she’ll call Bob and say “Hey Bob, is your SUX blinking red?” At this point, we’re out of the realm of unobservable spin (or perhaps not–quantum crypto does seem to involve a tremendous of spin which is hard to interact with). But then we’re out of the realm of particle spins and into the realm of human activity which gives meaning and relevance to the physics.

I’m not going to delve into the physics of it. I know enough to know that I don’t play there. But I can listen and understand people who play at the engineering level. There are issues with the orientation or changes in orientation of the mirrors, or with bursts of unexpected photons down the fiber, and these lead to a whole slew of attack vectors which may or may not be practical. The quantum cryptographers call these cheating. I call them security engineering.

Finally, on the socio-legal level, what action Alice and Bob take is first determined by their personal relationship. If they’re husband and wife, they might have some spare bits available from last time they were in the same place. If they’re co-workers, perhaps they have a boss who can help them get secure bits. But maybe Alice works at a stock exchange, and Bob at a bank. There might be some urgency, and there might also be economic or legal consequences to shutting down the communication lines.

This is one of the key points Andrew and I made in the New School: that the technology is embedded in a human context, and we need to examine it as such. That idea is embodied in a paper by my friends Sarah Blankinship, Tomasz Ostwald and Jon Pincus, “Computer Science is a Social Science.” (Link points to a draft, a fuller version is forthcoming.)

Claims that a technology is secure absent the social and legal contexts which give security meaning are no longer just irksome: they actively detract from progress in the field.

The Discipline of “think like an attacker”

John Kelsey had some great things to say a comment on “Think Like An Attacker.” I’ve excerpted some key bits to respond to them here.

Perhaps the most important is to get the designer to stop looking for reasons attacks are impossible, and start looking for reasons they’re possible. That’s a pattern I’ve seen over and over again–smart people who really know their system also usually like their system, and want it to be secure. And so they spend a lot of time thinking about why their system is secure. “Nobody could steal our PIN because we encrypt it with triple-DES.”

So this is a great goal. I have two questions: first, is it reasonable? How many people can really step outside their design and regard it with a new perspective? How many people can then analyze the security of a system they’ve designed? (Is there a formal name for this problem? I call it ‘creator-blindness.’) I’m not sure exhorting people to think like an attacker helps. This problem isn’t unique to security, which brings me to my second question: is it effective? I was once taught to read my writing aloud as a way of finding mistakes. I teach people to diagram their system and then use a system we call “STRIDE per element” to help people look at it. By giving people a structure for analysis, we help them step outside of that creator frame.

A second goal of that “think like an attacker” exhortation is to get people to realize that, in order to know whether their system is secure, they need to learn something about what tools and resources an attacker is likely to have.

So, for a moment, let’s assume that this is a reasonable goal, and one we can expect every developer who hears the phrase to go pursue. Where do they go? How much time should they devote to it? Again, I’m not talking about the use of the phrase within the security engineering community, but in software engineering more generally. Secondly (again), there’s the question of “is this the most effective way to push people?”

Third, there’s a mindset of being an attacker. I don’t know how to teach that. It’s not just about intelligence–I’ve worked with stunningly brilliant people who don’t seem to have that mindset, and with people who are much less brilliant in that brute-force impressive brain sense, but who just seem to have the right kind of mind to break stuff.

Well, that I can’t argue with. All I’ll say is that we’ve been exhorting people to think like attackers for years, and it hasn’t helped.

I believe that security analysis is a skill which can be taught. The best have both talent and have worked to develop that talent. I hope and expect that we can figure out how to do so. Figuring that out will involve figuring out what pedagogic approaches have failed, so we can set them aside, and make room for experimentation, chaos, and — we hope — actual improvements. I believe that, when asked of non-security experts, the ‘think like an attacker’ is on that list of things we should set aside.

Finally, a side note on the title. If you’re indisciplined, feel free to skip to about 3:10.

Think Like An Attacker?

One of the problems with being quoted in the press is that even your mom writes to you with questions like “And what’s wrong with “think like an attacker?” I think it’s good advice!”

Thanks for the confidence, mom!

Here’s what’s wrong with think like an attacker: most people have no clue how to do it. They don’t know what matters to an attacker. They don’t know how an attacker spends their day. They don’t know how an attacker approaches a problem. Telling people to think like an attacker isn’t prescriptive or clear. Some smart folks like Yoshi Kohno are trying to teach it. (I haven’t seen a report on how it’s gone.)

Even if Yoshi is succeeding, it’s hard to teach a way of thinking. It takes a quarter or more at a university. I’m not claiming that ‘think like an attacker’ isn’t teachable, but I will claim that most people don’t know how. What’s worse, the way we say it, we sometimes imply that you should be embarrassed if you can’t think like an attacker.

Lately, I’ve been challenging people to think like a professional chef. Most people have no idea how a chef spends their days, or how they approach a problem. They have no idea how to plan a menu, or how to cook a hundred or more dinners in an hour.

We need to give advice that can be followed. We need to teach people how to think about security. Repeating the “think like an attacker” mantra may be useful to a small class of well-oriented experts. For everyone else, it’s like saying “just ride the bike!” rather than teaching them step-by-step. We can and should do better at understanding people’s capabilities, giving them advice to match, and training and education to improve.

Understanding people’s capabilities, giving them advice to match and helping them improve might not be a bad description of all the announcements we made yesterday.

In particular, the new threat modeling process is built on something we expect an engineer will know: their software design. It’s a better starting point than “think like a civil engineer.”

[Update: See also my follow-up post, “The Discipline of ‘think like an attacker’.”]

More on Confirmation Bias

Devan Desai has a really interesting post, Baffled By Community Organizing:

First, it appears that hardcore left-wing and hardcore right-wing folks don’t process new data. An fMRI study found that confirmation bias — “whereby we seek and find confirmatory evidence in support of already existing beliefs and ignore or reinterpret disconfirmatory evidence” — is real. The study explicitly looked at politics…

What can I say? Following up on my post, “Things Only An Astrologist Could Believe,” I’m inclined to believe this research.