Note: This post is freakin’ long. Longer than any blog post has a right to be but that’s too bad because I can’t think of a natural cut off point for it. So…if you don’t want to read the whole thing, here’s the executive summary:
Overview: The class remains quite promising and the research and work that’s gone into it so far is quite evident. It’s main drawback is that it tries to stuff too many subjects in too small a timeframe. As a result, some rather important elements of intelligence analysis get a rather superficial treatment. Ideally, there’d either be an extra day of class time or a bit fewer topics to provide additional depth.
On with the main text:
I just completed the second of four parts of the Intermediate Intelligence Analysis Training Program conducted by the Sacramento Regional Office of Homeland Security.
The administrative aspects of the class went very smoothly. The class designers say that their intent, once the program gets final approval, is to take this class on the road so many of these aspects will be irrelevant but it was nice to see that there were no hiccups in the basics. Accommodations, transportation, food, training facilities, instructional material and additional paperwork were handled quickly and efficiently giving students one less thing to worry about. Bonus points for getting me vegetarian meals at lunch. My only recommendation would be that if they are planning on providing food in the future, they might want to specifically ask if the students have any dietary restrictions up front.
I’d also recommend that the course begin with an agreement that the class be a ‘non-attribution zone’. In order to make attendees comfortable that they could speak openly and honestly about issues, their agency, etc. everyone should agree that nothing said in class would be quoted or attributed to the speaker. I don’t think it would have changed much in this course but it would be nice to establish it right at the beginning and set a standard for professionalism.
The course was given over a three and a half days that ran a total of approximately 28 hours (excluding lunch times). They were pretty long days, packed and demanding which I continue to think is particularly good in keeping out the riff-raff who are just looking to chill out away from the office for a couple of days. When I was finished I was ready for a couple days of shooting zombies and other mindless activity.
The three and a half days of classes were divided into the following sections:
- Introduction and Pre-test
- Problem Based Learning
- Personal Leadership
- Critical Thinking
- Analytic Techniques
- Critical Analysis
- Collaborative Risk Management
- Final Exam and Evaluation
Generally, the course required a higher level of participation than many other trainings I’ve attended and that was an intentional decision on the part of the course designers. Their position is that adults generally (and those in law enforcement specifically) don’t learn or retain information particularly well in a lecture format. Therefore, they tried to emphasize interactivity and practical exercises wherever they could. There were still those awkward moments of silence where you could tell either the class wanted to play a passive role but overall the class embraced the concept.
The pre-test and final exam were multiple choice questions which apparently were a requirement by our evil federal overlords and the requirement demonstrates how far there still is to go in getting people to understand what analysis is. In a course designed to teach critical thinking and problem solving, is regurgitation of lists and definitions really what we want to test for? I understand how difficult it is to find good, objective metrics to demonstrate competence for those skills but that it looks like the government just took the easy way out in an effort to find something to test. Needless to say, I don’t think the tests accurately demonstrated the extent of learning we did (or didn’t) do.
We were provided with a hefty course guide and three publications for the class:
Hey…who doesn’t love free stuff? The publications were only occasionally referenced during the class so it might have been nice to refer to them more frequently. Too many publications get thrown onto a shelf to collect dust if a compelling case for their being read and incorporated isn’t made.
Problem based learning seems to be a riff of the Socratic method (kinda-sorta) and the class took to it quickly. This set up the rationale for the participatory style of instruction versus the traditional lecture method. I don’t have much to say about this block of instruction since both the instruction and practical exercise met the stated objectives.
Personal leadership: I had some problems with this block in the first session and have to admit I still think this block misses some opportunities (of course, those who know my success, or lack thereof, in bureaucratic institutions may question my ability to speak with any sort of authority on this subject). The idea that there is a ‘right’ way to introduce organizational change seems a bit too simplistic and formula-loving to me and overlooks the important aspects of organizational and personal analysis. Since analysts (civilian ones, at least) are frequently considered junior partners in law enforcement circles (including fusion centers), an honest approach to organizational change should revolve around how analysts can best parley their limited influence into the most effective change. How can analysts evaluate and utilize the goals and motivators of the institution and senior personnel around them to bring about change (call it ‘Judo-style change’)? If you wanted to get really high-speed, you could recommend the works of Machiavelli or Sun-Tzu (with practical examples) as guides to thinking about change. Now this may seem a bit esoteric and/or academic for analysts at this level so anything like this would need to be introduced properly but my recommendations are built around two assumptions:
- The course is advertised as being for ‘intermediate’ analysts which means they should have already mastered the basic technical skills of analysis and should be ready for the cerebral stuff
- Analysis is all about pushing your intellectual boundaries and if you aren’t pushing analysts (at least a bit) out of their comfort zone, they’ll fall into the mental pitfalls of routine thinking processes.
This is also the place where it might be worthwhile to discuss the ethics of intelligence analysis. What should an intermediate level analyst do if they (or a subordinate) is instructed to create a product with a particular finding? How about if they witness a violation of federal regulations or civil liberties? I’m not talking ‘pie-in-the-sky’ discussions but rather a ‘real world’ discussion of the responsibilities of the analyst and the implications of any action.
Critical Thinking and Analytical Techniques: These are hard for me to judge fairly since I’ve taught a very similar course and I’m having difficulty in separating my personal instruction style from the material. That being said, here are my observations.
The discussion of ‘universal intellectual standards‘ could have benefited from a bit more exploration, ideally through example(s).
The practical exercise involved a scenario where 15 people were on the international space station when a massive meteor strikes the earth. There are only enough supplies to keep 6 people alive on the station long enough until earth becomes inhabitable again. There are brief biographies of the 15 and each group had to determine who should survive and who should get booted out of the airlock. The exercise went well although there was more room to question each group more specifically about how they developed their selection criteria and weighed their choices according to their criteria. That would have allowed for us to go beyond arguing over individual people and examine if we were even measuring people according to the same criteria (which has definite implications for analysts).
There was a brief foray into the world of 28CFR Part 23 which could have been done in an on-line portion of the class. Given the importance of the regulation, it’s incredibly boring nature, and our press for time I’d recommend moving it entirely on-line.
This portion drew heavily upon The Psychology of Intelligence Analysis by Richards Heuer which is often regarded as the foundation of systematic examination of analysis and any analyst worth his/her salt should at least know of it’s existence. It was a bit disappointing (even if not surprising) how many seemed to be unfamiliar with the work. I try to make the link between Heuer’s ‘parlor tricks’ and actual analysis a bit more explicit but that’s a personal preference.
The portion about hypothesis development and the use of the Analysis of Competing Hypothesis program was quite good and the instructors understood clearly that the ACH works best when demonstrated.
Discussions about other techniques were a bit uneven. Key assumptions check was good, devil’s advocacy could have used an example (to define it from simple contrarianism), and red cell analysis needed a bit more fleshing out (the temptation to mirror image or engage in fanciful movie plot scenarios becomes great if you don’t understand the technique clearly) and I would have liked to see a concrete example attached to inside-out/outside-in thinking.
The discussion of the Beltway Sniper was relevant and a great example for many of the points discussed. The supporting documentation allowed the class to raise all sorts of interesting questions that demonstrated a nice grasp of the concepts. We had two practical exercises to use the techniques described and they were complex enough to prevent jumping to conclusions yet simple enough to allow us to get to a point where we could say something intelligible about them in a reasonable amount of time.
Critical Analysis: For this block we watched the Battle of Algiers which I hadn’t yet gotten around to watching (despite repeated brow-beatings by my boss) so I was anxious to see it both as a film and to see how they’d integrate it into the curriculum. While I thought the movie was quite good (and yes, I should have taken the time to watch it earlier) after we were about half way through the film I couldn’t figure out how it fit into the larger course. We stopped the film for discussion and to answer some questions every 20-30 minutes but it mostly focused on the terrorism/insurgency angle and didn’t spend much time talking about analysis or critical thinking. The scene where the paratrooper colonel describes the hierarchy of the insurgency lends itself to a discussion about how law enforcement sees organized criminal networks and frequently assumes they use a similar model. They could also have looked at the film as a film to determine what message the creators attempted to deliver and how well they did. Analysts should be able to critically assess not only the images on the screen (or words on the page) but what biases or point of view the creator had and what they’re attempted to ‘sell’. Too often, analysts assume that if a product has a seal of an official agency on the title page the contents are beyond question and totally objective. I’d recommend shifting the focus from this portion of the instruction away from teaching about terrorism and insurgency (which really deserves a specific class) and towards the lessons it can teach about analysis and critical thinking.
Then there was a brief discussion of ‘White’s Continuum of Conflict’ which is an attempt by one author to place all of human conflict on one sliding scale. I wasn’t particularly impressed with it and reject the implications of a linear scale that conflict progresses along. Again, I wasn’t too sure what the point of this was and it could be dropped with no loss.
At some point in this block was my least favorite part of the whole week. Three individuals (I don’t remember if it was ever revealed where they work) gave their names and said they would be evaluating our writing in the next, on-line, portion of the class. I assume they were rushed and crammed in where ever there was time but they began listing various dos and don’ts in a none too friendly manner. There were some minor histrionics among members of the class during the break (I don’t think I could contain my eye-roll when one participant threatened to drop the class) but I do agree it could have been handled better.
Collaborative Risk Management: This block introduced the risk management tool CARVER+Shock (the link is to the FDA version of the tool but it’s a very similar system). This block took up the final 3 or 4 hours of the course and it felt very rushed. In that time we had to both get acquainted with the CARVER+Shock method and run through a practical exercise. The practical exercise involved us playing ‘red cell’ by identifying a terrorist group and planing an attack as if we were that group. I did a similar exercise at a training about ten years ago and it was very interesting but here it was too rushed.
In exercises like this there’s a temptation to try to outdo movie terrorists with elaborate plots that would cost millions and require a cast of thousands to have any chance of being pulled off. We were so rushed there wasn’t a lot of structure to this part of the exercise and so we frequently got into the realm of fairy tale. I’d recommend breaking this part of the exercise up into daily bits so that on the first day of the class each group would be told to create/identify a terrorist group (without much/any additional information about what they’d be used for), their ideology, resources, goals. Then, the next day, groups could be assigned to identify a target which would be in line with those parameters. Finally, they could plan the attack.
This block really needed more time to be effective given the numerous learning objectives they set out for us to achieve. The time allocated would probably be fine if it was just an introduction to the concept of risk management but then the idea of a practical exercise would be more problematic.
Next: Surprises, impressions, and recommendations