Tag Archives: critical thinking

The HORROR (of intelligence analysis)

Yeah, there are plenty of places on the internet where you can find horror themed blog posts (especially today) and perhaps not so many (but still plenty) that talk about intelligence issues.  But where, gentle readers, other than right here will you find a blog that can seamlessly merge the two as Dr. Frankenstein sewed together the body parts of the dead?

If you’re a long time reader of TwShiloh (and have too much time on your hands) you may remember way back in October of 2007 (there must be something about the season that makes me return to this subject) when I talked about an idea I had for using the John Carpenter movie The Thing as a vehicle for teaching various aspects of critical thinking and structured analytical techniques.

Well, I’m very pleased to announce that I finally had a bit of time and put my money where my mouth was, developing such a course.  While it still needs a group (or two) of guinea pigs to test the concept I think my draft project exceeded even my original optimistic vision.

My list of learning objectives include:

  • Recognize how structuring benefits the work of an analyst
  • Explain how biases interfere with analysis
  • Discuss how structured analytic techniques assist in defeating biases and overcoming mind-sets
  • Discuss the utility of and demonstrate proficiency in structured analytic techniques to include:
  • Brief an audience about the results of a hypothesis generation/testing process that that disseminates results of research and analysis

As I watched the movie (again and again throughout the week) I became more and more convinced that various forms of media could serve as useful source material for teaching aspects of analysis.

Intelligence analysts in law enforcement – observations. UPDATED.

I just spent a few days with a bunch of civilian intelligence analysts and that plus a number of other recent events have prompted some observations on the state of the field.

This particular group came from a number of agencies spread across two states and I have to admit I was blown away by their motivation and drive.  While, generally, analysts remain an introverted bunch (even I, gentle reader am an INTJ) their work ethic seemed to overpower that trait and didn’t dawdle when there was work to be done, even in groups.  While they were much more quite than comparable groups of law enforcement officers I’ve worked with before they were more task focused by orders of magnitude.  You know you’ve got an engaged bunch when you have to tell them to go home three times!  The next time you hear someone talk about lazy government workers, tell them to suck it.

They also had a great deal of interest and pride both in their craft generally and in their specific fields.  These are people who are chomping at the bit to do their work.  It’s stuff like this that really keeps me going in this field.

That’s good because (yeah, thanks, bring on the buzzkill. eds), I’m convinced (based on observations and reports from others) there remain deep structural problems in the field of law enforcement intelligence.  Let me count them (well, some of them anyway):

  1. I keep hoping to come across analysts that actually have some sort of ‘seat at the table’ and to be quite honest at this point I’d even be happy if it was the kiddy table out in the kitchen with all the 4 year olds making jokes about poo.  Analysts are simply not brought in to discuss serious issues in which they should be central players.  I’m talking about issues of hiring, training, and career progression where (and for once I’m not exaggerating) I don’t think I can think of more than three or four analysts I’ve met over the past eight years that have been brought in to actually have a serious voice on these issues.  There are even fewer incidents of analysts being allowed to have a say in things like analytical focus or resource allocations.  And just forget about the idea of analysts actually supervising in analytical shops.  I’m not saying these things never happen, just that they’re so rare we might as well consider them mythical creatures.
    • Given that most state and local agencies are facing big budget problems with many departments considering laying off police officers, does it make sense to pay cops to not conduct investigations, enforce the law and arrest bad guys and, instead, assign them tasks for which they are rarely trained in and, usually, have little interest or capability to do?  Does it make sense to take those people with training in intelligence work and perpetually keep them at the bottom of the food chain?
    • This makes sense only if you consider the following:
      • These supervisory positions represent promotional opportunities for agencies and you always take care of your tribe first.  No matter how much they may like their civilian employees, they simply aren’t on the right side of that thin blue line and if push comes to shove, you take care of your tribe first.
      • Further riffing off the tribal theme…There’s an abundance of literature and anecdotal evidence that civilian analysts are secondary or peripheral players (or, as I recently heard to my annoyance, ‘cop-lite’).  Given the recent emphasis on things like ‘Intelligence Led Policing’ where intelligence is supposed to be the prime mover in operations, this attitude is simple cultural chauvinism.  There’s unease at letting ‘out-group’ members get access to information or resources.  Who knows what they’ll do with them?  Can they be counted on to further the ‘in-group’ interests?
        • (A brief vignette) I once sat in on a meeting, the only civilian in a room of law enforcement officers discussing some procedural issues.  When the possibility arose that civilian analysts might have access to investigative information, you would have thought wikileaks just published the nuclear launch codes.  Bizarre scenarios began floating around about analysts being sleepers for criminal groups, selling confidential information, etc.  I then pointed to the dearth of evidence of ‘dirty analysts’ and recommended we asked internal affairs to generate some numbers of investigations of analysts compared to cops.  For some reason, there didn’t seem to be much enthusiasm for following this up with hard numbers.  While the point was made, analysts still were barred from accessing important information.
        • Think I’m being too cynical?  Allow me to recommend part 2 of last week’s This American Life (ignore the first part which is about a jerk who can’t understand why publicly threatening to shoot people would possibly get him in trouble).  It’s laden with interesting items regarding what happens when you rely too much on poor metrics, question the status quo and if such problems are the result of a ‘few bad apples’ or a rotten barrel which spoils most of what you put into it, regardless of how nice the apples were when you started out.
  2. I still think Fusion Centers do more harm than good.  Since they were created without a great deal of forethought, their numerous flaws just continue to get bigger and more obvious.  Agencies which  run them (usually law enforcement entities) have limited experience dealing with ways to measure success meaningfully (whether that’s intentional or accidental, I’ll leave for you to ponder) and so nobody has really bothered to figure out what these centers should do and how you should measure their effectiveness.  As a result, they generally figure out what the easiest things are to measure and then reverse engineer metrics and mission statements around those.  Therefore, things like number of bulletins published, database checks made, events hosted, or agencies contacted are used, giving a priority to activity rather than progress or effectiveness.
  3. This deemphasizes analysis which is a tricky thing to measure or evaluate easily.  I dig, trust me, I do.  This sort of thing is hard to do but it doesn’t mean you don’t even try.  So, rather than bother with figuring out how to build a system where you ‘fuse’ intelligence into functional analytical products, fusion centers give information sharing primacy because it’s much easier to measure.  Now, don’t get me wrong.  Information sharing is very important but it’s not intelligence and it’s not going to prevent intelligence failures.
    • Many years ago, Richards Heuer wrote that intelligence failures were failures of analysis rather than collection.  I’m not sure I agree completely with that (unless you use the term ‘analysis’ broadly to include a leadership component that actively obstructs the work on analysts) but he was definitely on to something.  You wouldn’t know that however, by examining how fusion centers operate.  Rather than spending their efforts on enhancing their analytical capabilities, they spend all their time on the collection side.
    • There’s a manic frenzy to get access to more and more sources of information (what a shock, yet another simple metric you can throw around to demonstrate how ‘cutting edge’ you are) and give any Fusion Center shill a chance to pitch his/her agency and within the first two minutes you can bet you’ll hear them proclaim how many databases they have access to and advertise to anyone who will listen that they’re a ‘one stop shop’ for your agency when it comes to sorting through the numerous datasets out there.  That’s great and could be a really powerful tool but problem is the people they assign to answer all those queries, chained to those databases, are the analysts who are supposed to be doing, like, analysis.
    • To resurrect a tired cliché, everyone is so obsessed with making sure we’ve got all the dots, nobody is spending any time trying to connect them.
  4. I’m torn about intelligence professional organizations.  I know a number of people in them that I really respect and like but, as I’ve written before, I can’t help thinking they’re just enablers for bad practises and cover for agencies that engage in the trade of smoke and mirrors rather than intelligence analysis.  In short, they just don’t have any balls.  Now, I understand the argument that we’re talking about big cultural changes and we need to gently and gradually lead everyone by the hand, build consensus and we’ll eventually get to the promised land.  I don’t think it’s true but I understand it.  But if that’s our plan let’s be honest about it and tell the taxpayers ‘Yeah, you know that whole 9/11 thing?  Well, we’re going to fix our intelligence systems to prevent that sort of thing but we don’t want to be too pushy so we’re going to wait another decade or two until people who are uncomfortable with change can ride out their careers without too much stress and then we’ll start fixing things.’

All of these things occasionally make me swerve into the territory of crisis of faith and wonder if talking about intelligence analysis in law enforcement is futile.  I still don’t know the answer but suspect it lies in waiting for the analytical community to develop its ‘revolutionary consciousness’.

A las Barricadas!

Update:  Ask and ye shall receive!  I received an email from someone (who I was kind of hoping would take the bait since I knew he/she was an exception to what I was taking about) who said:

My analysis shop is run by analysts.  I’ve never worked in any other shop so I don’t know what it would be like for this not to be the case.  Our shop is set up with a “director” who basically just handles grants, supplies, HR stuff, etc., and then a managing analyst, with 3 civil service analysts, and then 6 contract analysts.  We’re actually hiring 2 contract analysts and the managing analyst is basically taking care of the whole thing, choosing who to interview and hire.

I think the key is that our shop basically started as one guy attached to homicide, and then another guy who worked with narcotics, and then it grew to 3, and then 4, until it exploded to 10.  And because it initially grew so slowly and was working with the most sensitive stuff, and also because the chiefs really liked what we were doing, they basically allow us to run our own stuff.  My sense is that this is not how most analytical shops develop.
Still, no idea how to measure our success.  Maybe in the # of proactive successful investigations?  Or crimes solved with analyst help?  The problem is that if we trumpet our success then people won’t want to work with us as much.  Part of our success is because we let the investigators and narcotics cops take all the credit (which is why they like us).

I’ve heard of success in analysis shops before but the times I’ve been able to delve into their details (and I can’t speak to the situation above) it’s been attributable to a particularly competent and influential person.  The concern (well, at least mine) is that there aren’t sufficient processes in place to maintain that success.  Time will tell…

And, thanks to my ghost contributor.  I don’t consider this endeavor futile.  If I did I wouldn’t stay in the field.

Critical thinking for everyone

I just stumbled upon a podcast/website by Kevin deLaplante who’s building a promising looking catalog of work revolving around critical thinking.  I’ve sampled the first three episodes of his video podcast in which he explains the value of critical thinking and the parameters of his project.  His goal is to produce 5-10 minutes lectures focusing on one idea at a time.  The podcast is now firmly ensconced in my queue.  Good sound quality, fair production values (thus far, essentially a PowerPoint presentation…no shiny objects or fluttering animation) but well worth your consideration.

He also offers other videos, some of which are free but the primary goal is to encourage you to purchase a membership to access a great deal more original content.  I haven’t had a chance to check these out yet, but I have high hopes.

I’ve felt that critical thinking courses for intelligence analysts can be a bit too subject focused and miss the underlying messages and concepts.  It becomes harder (in my opinion) when students feel they have an extensive understanding of the subject material that the instructor is using to demonstrate the practice of critical thinking.  In those cases it becomes easy to fall prey to cognitive biases and jump to conclusions, even when claiming use critical thinking techniques.  Therefore, a non-national security/law enforcement themed class on critical thinking is generally superior (although, you have to do more up front work when it comes to highlighting the relevance of what you’re teaching).

Anyway, give it a looksee…

Analysis of Competing Hypothesis

Kristan over at Sources and Methods has a post up about the efficacy of ACH.  I’m a huge fan of the system and actually used the freeware version to help Mrs. TwShiloh and me in deciding on a house to buy.

We were trying to juggle multiple factors on multiple properties.  As you know from reading Richards Heuer, you’ll know that the human mind can really only track 7-9 things at one time and attempting to do more just results in all sorts of cognitive train wrecks. We pllugged in the properties with a whole list of criteria (from the important to the trivial), weighted them and ranked them.  When it was all set and done one house that we thought we loved dropped to the bottom of the pack and we realized that we had gotten swept away by one or two really attractive features and blotted out the many, many problems such a house would give us.

So, I was pleased to read that ACH actually does have some utility.

The results were in favor of ACH in terms of both forecasting accuracy and bias.  In Drew’s words, “The findings of the experiment suggest ACH can improve estimative accuracy, is highly effective at mitigating some cognitive phenomena such as confirmation bias, and is almost certain to encourage analysts to use more information and apply it more appropriately.”

That’s doubly good since it looks like ACH is about to get another boost through another freeware release of a new, improved version.

For three years, Matthew Burton has been trying to get a simple, useful software tool into the hands of analysts at the Central Intelligence Agency. For three years, haggling over the code’s intellectual property rights has kept the software from going anywhere near Langley. So now, Burton’s releasing it — free to the public, and under an open source license.

This is really exciting and I’m anxious to see it when it’s finally released.  Burton’s improvement is that his ACH version will be collaborative, allowing multiple analysts to add evidence and share views. 

Kvick Tänkare

Ta-Nehisi has a great post about the case of the D.C. detective pulling a gun on a bunch of people throwing snowballs.  The detective is claiming that he thought an angry mob of  anarchists were threatening him and he feared for his life and so pulled his gun.  Yeah…cause anarchists attack a whole lot of cops…and snowball them to death.  Maybe…maybe, he’d have a case if the G20 was meeting at the time and he was in the area of protests.  But no, Baylor was in civilian clothes and a civilian vehicle.  There was no way he’d have been identified as a police officer.  That means Baylor had to be under the assumption that these ‘anarchists’ were on some sort of racial attack.  Yeah…cause as widespread as anarchists attacking police officers is, it’s even more common for them to lynch random black people.

As Coates sums up:

…it’s good to know that Detective Baylor won’t be, like, fired or anything. Wouldn’t want a cop who feels endangered by snowballs to be bounced off the force. The rough streets of D.C. need men with that kind of mettle.

If you are the sort of person that worries about the carbon footprint of food, don’t feel guilty about bananas.

Bananas are a great food for anyone who cares about their carbon footprint. For just 80g of CO2e you get a whole lot of nutrition: 140 calories as well as stacks of vitamin C, vitamin B6, potassium and dietary fibre. All in all, a fantastic component of a low-carbon diet.

• They are grown in natural sunlight, which means that no energy-intensive hot-housing is required.

• They keep well, so although they are often grown thousands of miles from the end consumer, they are transported by boats, which per kilo of freight transported emit only 1% as much CO2 as planes do.

• There is hardly any packaging, if any, because they provide their own. (You might sometimes see a bunch in a light plastic bag or wrapper, but this probably pays for itself carbon-wise by reducing the chance of customers ruining the fruit when they try to split a bunch.)

“You get to play with very large toys in the oil industry.”  A great explanation of the science behind the BP oil spill.  (h/t Phronesisaical)

I was never a big fan of Andrew Sullivan’s ‘View from your window’ (probably because he refused to publish my own, most excellent entry) but he’s now got a contest where you guess the location of the shot.  It’s quite amazing to read the logic people use to figure out where the picture was taken.  I only wish they published all the guesses.  Great critical thinking exercise.

Help an analyst out…

From Sources and Methods a fledgling analyst is working on his thesis and needs some help.  He’s put together an on line exercise together and needs as many people as possible to participate.  If you’re convinced that the goons in the CIA and FBI miss all the big clues and you’ve got a foolproof method of divination or you just want to help a guy out, give it a go.

Mulder Thesis Experiment.

Intelligence Analysis 101 (part 2)

(Here’s part 1 if you’re joining us late)

Ok…pens down.  So, you’ll remember this picture:And the questions associated with it were:  Do you agree with the interpretation of the ad?  Why or why not?  Do you agree with the analysis?  Why or why not?  If not, please provide an alternate explanation.  What recommendation(s) do you make to your boss?

The interpretation of the ad was totally off base.  In fact, the ‘analysis’ provided was so bad I’d argue it belongs in a hall of fame somewhere.  At least George W. Bush had ‘Curveball’.  He might have had no credibility but at least it was something.  The interpretations that came in the alert had nothing.  Now, let’s talk about how one arrives at that conclusion (that’s the good stuff).  As an aside, if you’re teaching entry level analysts (or anyone for that matter) this can be a pretty good way to introduce the workings of Analysis of Competing Hypothesis.

  1. The ad itself.  The cost of a full page ad in both the NY Times and the Washington Post will run into tens of thousands of dollars (perhaps with a total price tag into six figures).  I don’t know if you’ve seen animal rights activists but they aren’t exactly awash in money.  Assuming they pooled their money or had a rich benefactor though, there are still problems with the ad itself.
    1. Show me the money! You probably aren’t going to be able to pay for this ad space anonymously and with cash so the people taking it out are going to leave a paper trail.  Groups like ALF derive their strength from their anonymity and if they were going to give it up it would probably have to be for a pretty big payoff.
    2. The picture.  The iconography is all wrong.  Google animal liberation front images and you will indeed see pictures of guys in balaclavas but almost without fail they’re shown with animals.  The menacing garb is not the focus of the imagery.  The focus is on the act of ‘liberating’ those animals.  So, they balance the scary image of the dude in camo and face mask by having him coddle a cute bunny, usually holding it and looking at it in a way to trigger images of parents holding an infant.  Whoever made this ad would be the same people who argue that the Mona Lisa is a great Italian landscape picture…oh, and I think there’s some chick in it.  They’re clearly trying to raise the fear factor by putting a guy who looks like a terrorist in the image but I don’t think that’s not how the ALF would picture themselves.
    3. The jacket.  I’m torn on the jacket.  Either it was a sloppy mistake (you aren’t going to find many animal liberation advocates wearing leather – and that’s an understatement) or it was a subtle ‘fuck you’ to the activists.  I guess it could be pleather though…
  2. The message
    1. Who’s the target demographic?  If the point of the ad is to intimidate the business community (well talk about messaging problems below), why not place the ad in the Wall Street Journal or Business Week?  Who are these people targeting?
    2. What are they trying to get their demographic to do?  Are we to believe they’re going to spend a huge wad of cash to get people to go to a website?  That’s it?
    3. There’s NO WAY anyone sympathetic to the cause of ‘animal liberation’ would ever, EVER describe the activities of LSRI as ‘vital pharmaceutical research’. Really, you’re more likely to hear that the local B’nai B’rith is planning a lecture series on the valuable research into humanitarianism done at concentration camps.
    4. People tend not to refer to themselves in the third person.  Hence, if the ad was done by animal rights activists, this sentence wouldn’t read this way:  NYSE employees were reportedly threatened by animal rights activists whose campaigns had already targeted businesses connected to LSRI. But instead would read something like:  ‘In addition to targeting businesses connected to LSRI, we’ve also been targeting NYSE employees.’
    5. While not complementary to the NYSE, this isn’t exactly a revolutionary calling to free the animals or protest animal testing.  The message calls the NYSE cowards for bowing to pressure from animal liberation activists.  It doesn’t even work as gloating.  It only makes sense if you are unhappy with the fact that the NYSE caved into pressure, which one can assume, would not include SHAC or other animal liberation advocates.

I won’t get into a detailed tear down of the analysis since that part was fabricated by me to provide a composite example of the types of alerts and warnings commonly used.  I will say that there’s nothing in there that would have been out of place or unusual at that time in that ‘analysis’.  But allow to highlight some serious concerns with this sort of ‘analysis’ in general terms.

  1. You have no idea how the analyst came to their conclusion.  Blanket statements without supporting facts loaded with unexamined assumptions.
  2. Mealy-mouthed qualifications.  Ye old ‘we have no specific information…’ has become the nervous tic of the intelligence community.  It’s the refuge of people who want to cover their ass in the event something goes wrong and absolves them from having to do real intelligence work and follow through.  It’s especially maddening when seen (as it usually is) at the bottom of a warning threatening the end of civilization and life on earth.  “Warning:  If three dozen nuclear weapons were detonated in the largest U.S. cities simultaneously, there could be catastrophic social, economic and humanitarian consequences!!!  Terrorists have indicated their intent to acquire such weapons and all agencies should remain on high alert!!!  Oh..by the way, we have no indication that anyone is actually close to acquiring such a weapon.  We just don’t want to get blamed for not warning you in case some evil overlord has been planning something like this in his hidden cave complex with the mole people.”

This was not a difficult nut to crack.  Really, I would consider this something an entry level analyst should be able to work through on their own.  If you’ve got an intermediate level analyst (more than a couple years on the job) who has difficulty with this, they need to see a career counselor immediately.

I’ve been accused of being a bit cynical regarding the progress (or lack thereof) of the capabilities of intelligence analysis within the law enforcement community and this has probably more than a little to do with that.  As I stated in Part 1, I used this as a training exercise and was taken aback at how poorly people did at this.  Preconceived notions about who the ‘bad guys’ are and a slavish (really quite inexplicable) deference to everything produced by other law enforcement agencies meant that it was difficult to even get people to consider that there might be alternate explanations.  When pushed, there seemed to be a tendency to try to retain as much as possible of the original analysis, and only grudgingly working at the edges.

Finally, it also highlighted another serious flaw in these sorts of ‘alerts’.  There is little in the way of procedure and almost nothing in terms of incentives to get agencies to issue corrections.  Corrections are a permanent and official record of a screw up which can be used to undermine your reputation and authority.  My observation is that the preferred way to deal with errors like this is to just ignore them.  Usually they’ll fade away without a whisper and you still get to use the alert as ‘proof’ that you’re providing cutting edge intelligence (We issued 500 intelligence alerts to our partner agencies the past fiscal year!).  Of course, the problem is that once you release that bogus information it keeps bouncing around, getting recycled, absorbed into cognitive frameworks throughout the community.  Until you see as much effort into putting out credible, reliable analysis and correcting mistakes as you do in fussing over the quantity of information being put out, intelligence analysis will struggle for relevance and credibility.