Metrics Metrics Metrics

From Tom Ricks’ blog:

Sam Damon, our thoughtful officer of last week, concludes by offering some different and unusual ways of measuring whether we are succeeding:

Stop counting IEDs. The enemy is a second order effect or consequence. Count how many times tribes use shar’ia law or tribal mediation to settle disputes rather than local government. Count how many days the Afghan elected governor is in his province or district and compare it to how many days the Taliban Commander or shadow governor is there among the population. This is the primary type of Intelligence Preparation of the Battlefield that is needed, not the old paradigm that we still teach Military Intel officers.

I like this approach, which strikes me as similar to some of the suggestions Exum and his posse made in their paper titled Triage, in the section about metrics to avoid and metrics to use.

So at least we’re talking about metrics in Afghanistan.  Why then, here in the U.S., when we’re looking at the effects of crime and criminal networks are we still obsessed with the same two metrics we’ve been using for decades:  arrests and quantity of illegal material seized?

Neither of those factors will tell you if all your efforts will do any more than create a temporary vacuum into which new criminal networks will rush in.

Back in 2004 I was convinced the military could learn some valuable lessons from law enforcement in its wars in Iraq and Afghanistan.  Now, I’d say the reverse is true but there’s simply very little interest in law enforcement absorbing anything from the military except ‘Rambo gear’.

Product versus process

As I slowly (oh, so slowly) try to climb out of this deep, deep hole otherwise known as pursuing a graduate degree I’m mildly surprised how much I miss blogging.  So, I’m up for a quick breath of air…

I’ve seen a lot of attempts to ‘reboot’ the intelligence process in organizations.  One reason why I think I’ve seen so many is that all of the previous attempts have failed.  It struck me as I was listening to a pitch for yet another attempt (Once more into the breach!) that these attempts have shared some common characteristics.

• Sense of urgency – It’s eight years after 9/11, and nobody wants to hear that our intelligence work isn’t up to snuff or that we need to go back to the drawing board and do all the foundational work that wasn’t done initially.  In short, no one wants to have to answer the questions “Well, what have you been doing for the past 8 years?”  So, once the determination has been made that the intelligence/analytical process needs help, there’s the follow on demand that change occur NOW.  The underlying assumption is that all of the foundational work must be correct and any shortfalls must be superficial in nature (software applications, training deficiencies, etc.)
• Product fetish - The demand to see immediate results leads to an expectation to see a ‘quick win’ (a phrase which should result in immediate flogging if ever uttered again) in the form of some sort of intelligence product.  The stated reason for such a product is that it will build momentum.  I suspect the real reason for such a product is that it allow the sucker who was roped into this new responsibility a way to get higher ups off his/her back with the appearance of progress and the ability to leap to the next higher wrung on the ladder before the edifice crumbles yet again.
• Process avoidance – What doesn’t happen is an examination of what processes are needed to build an effective intelligence system and the development of a strategy to implement such a system.  I am hard pressed to find an analyst below the federal level who can claim to have a working planning and direction process in their agency.  Collection plans?  Who has time for that.  Analytical standards?  Hey, this isn’t some ivory tower.  The pathological avoidance of processes means that intelligence efforts will remain ad hoc and require constant reinvention.  (I’ll admit the possibility that such a system may be in the best interest of the institution by creating an ever renewing opportunity for ‘rising stars’ to make their mark by delivering such ‘quick wins’.  Such a position would be valuable for a person to have within his/her pervue within a system which allows for the distribution of promotional opportunities based upon patronage rather than merit.)
• Lack of appropriate expertise – If you ask analysts about the people put in charge of them or their agencies and specifically inquire about the backgrounds of their leadership you’ll often find a lack of training or experience in intelligence and analytical work.  I attribute this to two primary factors:
  • patronage – ’nuff said.  These jobs tend to be high profile and often on the ‘fast track’.  Leaders like to put their favorites in those positions.
  • Tom Clancy – Well, maybe it’s not really his fault but since I don’t like the guy I’m going to blame him.  What I mean here is the increasing access of information over the past decade or so along with a reoccuring narrative that everyone’s opinion is just as valid and insightful as everyone else’s.  As a result intelligence and analysis are viewed as no big deal and something that can easily be picked up.  Just watched season 1 of ‘24′?  Sure you’re qualified to run a counter-terrorism center.  I exaggerate, to be sure.  But not by much.

This is all facilitated by the fact that analysts remain the red headed step children in virtually all (I’d say all but I haven’t spoken to people in every agency) sub-federal level agencies.  In part this is because analysts are relative newcomers to the field, encompass such a wide range of duties (from data entry to investigative support to ‘true’ analysis) to almost make the title ‘analyst’ useless, and have no real institutional power (control of resources or the ability to make policy).

If you can get your hands on an organizational chart for an agency with analysts in it, see how many levels of non-analyst leadership a product or idea would need to go through to get to the top of the food chain.  Then try to see how many of those intervening levels have a familiarity with intelligence issues.  In most cases you’ll see that a lot of talk about ‘breaking down information silos’, ‘intelligence fusion’ or ‘intelligence led’ whatever really boils down to security theater.

What has improved (marginally) over the past eight years has been in information collation and distribution.  Many more people get information they wouldn’t have otherwise received.  The problem is that information almost never is evaluated in a meaningful way, analyzed, placed into context or combined with other information to answer questions that analysts should be paid to answer.  We need to stop obsessing on the ‘who, what, and where’ questions (which investigators, law enforcement officers and other currently do very well on their own) and work on the ‘why, so what and what’s next’ questions (which nobody is doing).  After all, if we aren’t answering those, what exactly is the job of an analyst?

Setting ourselves up for tragedy

I’m not sure I actually have to say that I’m not too sympathetic towards sex offenders but I do think that we should do what we can to prevent them from re-offending.  Some ways to do that include not labeling people for life as dangerous deviants for engaging in consensual relationships while minors, ‘mooning’ or streaking.  For those who are legitimately guilty of crimes against minors, identify those who can be helped with treatment and give it to them.  For those who are the true, irredeemable predators, alter the laws to make sure they can’t endanger the public.

However, stupid laws which push all offenders outside of society are only going to make all of them more likely to commit more crimes.  When you say that offenders can’t live, work or be within some arbitrary distance from every school, bus stop, play ground, etc. you’re going to end up with a whole bunch of ex-convicts congregating together and interacting with no one else.  Take away every option from people other than committing crimes and should we be surprised when they re-offend?

“Conjecture without evidence of imminent harm…”

“…simply fails to meet the board’s burden of proof.”

That was part of a quote by a federal district judge rejecting the argument by the Federal Reserve that releasing the names of financial institutions which received assistance would cause harm and instability to the financial system.  (You can hear a little more on this Planet Money podcast)

I thought about this quote and how it could equally appeal to the over classification of information, particularly in law enforcement circles.  Don’t get me wrong, over classification is a big problem in the military (and I assume throughout the rest of the national security system) as well but that deserves it’s own treatment by someone more familiar with the subject than I.

Law enforcement, however, seems to fetishize the classification of information and do it for particularly poor reasons.  Now, there are some very good reasons for keeping some information generated by law enforcement and sub-national intelligence organizations (in which I’ll lump fusion centers and state or regional ‘homeland security’ agencies in addition to others).  Information about confidential informants and ongoing operations need to be kept secure lest investigations and the safety of people involved become compromised.  Likewise, some information about the understanding of the threat and vulnerabilities that can be exploited should be kept secret.

That does not mean, however, that the default position for all information generated or intended for consumption by these agencies needs to have a classification label attached to it.  Yet, far too often, information that is either publicly available or of an entirely trival nature is festooned with some many security caveats that you’d think that it contained the nuclear launch codes.

I think there are three basic reasons for this, two of which are never stated when defending the current practice but which, I believe, exert a considerable force in preventing any real reform to the system.

1. The criminals will know what we know. By far, this is most often cited as the reason by information has to be classified.  It assumes that criminals (and there is no distinction made about criminal groups with this argument, both the street level addict ripping off hubcaps for his next fix and the highly sophisticated international criminal network are viewed as equal here) are all conducting intense counter-intelligence operations with advanced collection plans and analysis of their own.  In short, agencies argue that criminals are doing a better job of intelligence work than they are.  The fact that such operations are exceedingly rare makes no matter.  The fact that anyone has done such work anywhere means that we need to act as if everyone is doing it everywhere.  This is where the judge’s quote comes into play.  Just because you can conceive of a possible threat doesn’t mean that threat is likely.  Life isn’t a Tom Clancy novel or an episode of ‘24′.
2. Look, ma!  I’m important.  Unfortunately, regional, state and local agencies have absorbed the lesson from the federal intelligence community that the only information that is worth looking at is information which is restricted in some way.  In many cases this whole thing is a bit of a game since caveats like ‘Law enforcement sensitive’ are useless in their lack of clear definition and sanctions if violated.  As a result, agencies put these restrictions on products no so much because of any harm which would result in the event of their unauthorized disclosure but as a badge designed to impart some authority on the product.  It’s not uncommon to see media reports, copies at length without additional commentary labeled law enforcement sensitive for no reason other than a lazy and pathetic attempt to generate an aura of authority and expertise.
3. Well, I actually forgot the third reason while writing this but I’m sure it was brilliant.  Until I can remember it and update this post, these first two reasons should be enough to support my argument.

The key in the above is that they are the victim (or, I guess, beneficiary) of unexamined assumptions.  And when there are questions about the validity of these arguments, where does everyone turn?  To the very people who propagate and operate under these systems.  Very frustrating.

We need to get away from fantastical scenarios of how criminals operate and begin to consider that, in fact, there are many benefits to making more information available to the general public.  We already release raw crime data to the public (in the form of UCR data).  Some states do a very good job of putting that data in formats that can searched in multiple ways and can be compared to other data sources.  Other states just publish a pdf document, which in todays information saturated world seems, to me at least, a lot like through a stack of papers at someone, giving them the middle finger salute and daring them to actually do anything with the data.

The public should not only know what sorts of criminal activities are occurring in their neighborhoods but they should also be able to get informative assessments of threats.  Why shouldn’t people be allowed to know if the authorities have determined a particular area is at low risk for terrorist attack but, let’s say, moderate risk for hate crimes, as well as how such a determination was made?  I fear the wrong lessons were learned earlier this summer when the DHS report on political extremism got leaked and created a PR firestorm.  Rather than caving into criticism, the DHS could have started a real dialog about their methodology and findings, using the incident to explain to the public how analysis works and what it is.  I’m sure there still would have been strong disagreements about the conclusions by many and even a bit of paranoid hysteria among the fringe but really, what a lost opportunity to educate the public.

State and local authorities have a lot to learn from this incident.  If they’re serious about conducting intelligence at all, they’d be much better off laying the groundwork and making the process as transparent as possible now, before they have to scramble to explain a potentially embarrassing conclusion or product.  The goal isn’t to get unanimous agreement with the conclusions but, as Prof. Roberto says, you need to achieve an understanding that the decision or analytical process is legitimate and fair.  Otherwise, you open yourself up to all sorts of criticism that you’re paving the way for a Socialist (or fascist, authoritarian, extraterrestrial, anti-christ, etc.) takeover of mom’s apple-pie business.

The Swedish response to organized crime

Last week, while listening to my Radio Sweden podcast, I heard this story about something called the ‘Alcatraz List’.  It seems it is a list developed by Swedish law enforcement of the ‘most dangerous’ criminals in the country and there are concerns that it might not have been worth the expense to create it given that only half of the people on the list have received any jail time in the three years the list has been in existence.  The story was pretty sparse on details so there wasn’t much to say.  I did a bit of poking around on this wonderful series of tubes that is the internets and found this summary of report evaluating 14 initiatives designed to fight organized crime.  Thank god the Swedes love to 1)  do research, 2)  write about it and 3)  translate it.

Just an observation, Swedes apparently also love their white space.  The amount of blank pages stuffed in the front of the report (you’ve got to get to page 6 of a 33 page report before you get to any content) and the margins in the report are so big (oh my god…it’s full of stars!) that I’m reminded of my school days as I tried desperately to pad my reports to get them to meet the minimum page requirements.

In 2006, the Swedish National Police Board decided that SEK 120 million (approx 15.5 million dollars. Eds.) would be released for police efforts aimed at fighting organised crime…It was hoped that the campaign would provide the Criminal Investigation Department and the police authorities with an incentive to commence
larger projects, which previously lacked resources.

That being said, even though the report is in typically restrained Swedish style there’s quite a bit here to raise an eyebrow.  First, let this soak in:

Brå was commissioned to make an evaluation of the special campaign against organised crime, i.e. a thorough investigation of outcome and results made upon completion…This entails not just describing what was done within the campaign, but also why it was done, and whether it was the most efficient way of doing it.

Whoa…an actual evaluation of anti-crime initiatives?  That is impressive all by itself.  While in the U.S. the Department of Justice may do evaluations of initiatives I suspect you’d be hard pressed to find such work being done at the state level.  While it may be basic management strategy, the idea that law enforcement initiatives should be clearly articulated with cost estimates and metrics to determine success or failure remains as elusive as bigfoot at sub-Federal levels in the U.S..  Given that 11 U.S. states have populations roughly equal or larger than Sweden’s 9 million and have significant amounts of personnel and resources devoted to law enforcement and anti-crime activities one would think there would be some interest to determine the effectiveness of their efforts.

Secondly they identified their broad targets as:

1. Groups involved in organised crime: Hells Angels, Bandidos, Outlaws, Original Gangsters, Angereds Tigrar, Brödraskapet and A.S.I.R.
2. The importation and distribution of cannabis in Sweden.
3. Particularly difficult and serious crime over time, for example attacks on security transports.

Now, the report isn’t clear if their goals were more specific than this (How exactly is ‘organized crime’ defined?  What in the world is ‘difficult and serious crime’?) and it might either have been included in the full report or be an issue with translation but such broad goals could lead to abuse of the program.

How?  Glad you asked.

Imagine offering departments a big bucket of money to address the problems above.  A good bureaucrat will look around at what they’re already doing and see if they can shoe-horn existing activities (via some creative writing) into this program resulting in extra money for no extra work.  I’m not saying that happened here and knowledge that there would be an evaluation of funding and results would certainly go a long way to mitigating that risk but clear definitions would be helpful.

Some of the findings of the report:

• Much more is needed in the form of preventive action.
• Cooperation with municipalities and the social authorities involved in crime prevention
  work has, however, not been developed within the project.
• Cooperation with private sectors has not been particularly developed either.
• What has been mentioned in the interviews as less successful, are those projects that have had a narrow focus, primarily those that have been directed towards a particular individual.
• Most projects within the campaign have been operational, and have been directed towards tactical analysis with short-term objectives, in the form of confiscation and legal
  proceedings. Internationally, it has been put forward that there is a need for more strategically directed analyses within the police
• One implication regarding the projects that have, so far, been concluded, is that most of them are based on methods used by the police in their day-to-day police work.

It’ll be interesting to see if these points resonate within the Swedish justice system and they devote some energy to prevention, strategic planning and targeting, and incorporating non-law enforcement partners.  They are, essentially, arguing to adopt an intelligence led policing model of which I’m a fan, although I haven’t seen it actually work as advertised in an ongoing basis (although I hear positive things out of Canada and the U.K).

I would really like to know what sort of intelligence and analytical techniques they found (un)helpful and incorporate in their operations.  They get hinted at several times throughout the report but I can’t get a feel if they’re being intentionally vague, don’t see it as central to their initiatives or don’t really incorporate intelligence and analysis in their activities.

Update:  Oh, and just for the record I do not approve of the new intro music for Radio Sweden.

Gangs and insurgencies – the final chapter

I figured I had said all there was to about this subject and then David Kilcullen had to go and appear on The Colbert Report.  Check it out (sorry, I can’t seem to embed it).

Here’s the relevant exchange (forgive any errors in the transcription but this is really close if not exact):

Colbert:  …[You say] people in poor communities turn to gangs for protection, for services when the government can’t do it for them…

Kilcullen:  Exactly right…There’s actually a huge amount of similarity between basic police work and the sort of stuff that happens with gangs and so on and what happens in this [insurgency] environment.

So, first, it’s nice to know that David obviously reads this blog but he really should credit me for these ideas [I'm just kidding Mr. Kilcullen.  Call me! ]

More seriously, I’d make a slight tweak to his observation.  There is a huge similarity to the challenges in police work and insurgencies but I’d argue that most American police departments have been trying to approach those problems from a mindset that more closely hews to traditional military thinking that counterinsurgency doctrine.

Current anti-crime measures are almost exclusively reactive and suppression based.  You wait for a crime to occur, you find the suspect and you arrest him/her.  There’s no identification or addressing of underlying factors.

This video really struck a cord with me since I had virtually the same conversation with my command which I was in Afghanistan in 2003.  My command could not get their heads around the idea of insurgency and could only conceive of conventional military threats.  Hence, I was tasked to do an Intelligence Preperation of the Battlefield so that they could plan what to do if the Taliban attempted to overrun Bagram airfield.  Now, Bagram had over 10,000 soldiers at the time, in addition to a sizable number of attack helicopters and aircraft and yet, the only threat scenario these guys could come up with was a Taliban motorized rifle division coming over the Koh-i-Safi mountains.  IEDs?  Rocket attacks?  Nah…”This isn’t Iraq” I was told or “C’mon…who does that?”

As a side note, my repeated attempts to convince them that such a scenario was highly unlikely and that other threats should have a higher priority went unheeded, setting off an unfortunate string of increasingly dysfunctional exchanges which ended with me telling the S-3 that he was full of bulls*it at a very full shift change brief.  (Not a particularly wise move for a mid-level NCO although, miraculously, I avoided any repercussions).

We had a number of soldiers with civilian law enforcement experience and I recommended dragooning them, on a part time basis, to assist in intelligence gathering and developing a decent view of our new operating environment.  Command couldn’t figure out why we couldn’t get all the information we needed from the internet.  Needless to say, that didn’t happen and 2003-2004 in Afghanistan (at least in the Bagram area) can best be thought of as a year of lethargy.

Still, it is encouraging to hear that the military was inspired in part by the academic/research work done in law enforcement and people who understood the issue made it to the top.   Hopefully the military can return the favor to the law enforcement community.

Beach reading

I continue to think about how intelligence analysis can integrate with law enforcement operations and remain incredibly frustrated about how little progress has been made in the past decade on that front.  It’s been sticking in my mind since I expressed my dissatisfaction with the ‘organizational change’ element of a recent training I attended.  The one thing I’m reasonable sure of is that this problem isn’t one of dysfunction in an individual agency but rather is a function of a number of inherent characteristics of those agencies.

To that end I strongly recommend Organizational Pathologies in Police Intelligence Systems by James Sheptycki (you’ll have to have access to ProQuest or some other journal database).  It really is worth the time you might have to spend hunting it down.

Here’s a mind map that riffs off of Sheptycki’s article and captures what I think are his core ideas when overlayed with the intelligence cycle (I use a slightly different version of the cycle than is mentioned by the link).

Click on image for a larger version

What you’ll notice is that there are serious issues at every point of the cycle which indicates an serious institutional problems.  Law enforcement agencies are among the longest lived and structured government agencies we have and so introducing significant change to how they operate really requires someone who can and chooses to wield an extraordinary amount of power to alter a system which most likely brought him/her to the lofty position they now hold.

To use a rough analogy, in the military the intelligence section reports directly to the commander as does the operations section.  While operations typically has more influence the two groups are (at least in theory) on the same level of staff organization.  I would argue that translating the current law enforcement/intelligence situation into military terms would be the equivalent of placing the intelligence element not just under the operational one but, in at least some cases, one or more layers even further down the organizational structure.   Think about how military operations might change if intelligence information had to pass through two, three or more additional filters (which had limited to no experience with intelligence) before getting to the commander.  It wouldn’t just be an issue of what information gets passed up but even what questions would be asked and sent back down.  Therefore, Sheptycki warns:

The organizational pathologies outlined in this paper show that the processing of criminal intelligence is not as rationally ordered as may appear in the pages of the strategic analyses that are its product.

In the contemporary period there is a concern to use the product of strategic criminal intelligence analysis to re-order priorities. Policy makers in the field of organized and serious crime in all of the member states of the European Union are consequently exercised with commissioning and interpreting
strategic assessments of the organized crime problem and making policy of the basis thereof.There is an expectation that strategic analyses of criminal intelligence provided by the policing sector will aid policy makers in making decisions about resource allocation in the management of a wide range of organized criminal activity…Given the catalogue of organizational pathologies that affect the policing intelligence system, it will be necessary to use the analytical products of that system with circumspection. [italics added]

That’s a big reason why I’m increasingly of the opinion that analytical capability (beyond rudimentary case/investigative support which is what most in the law enforcement community consider to be intelligence analysis) needs to be either placed in a separate organization or moved up the system and placed in the hands of a higher authority.

Of course, there’s been quite a bit of discussion on this issue but not, as far as I can see, not a lot of change which indicates that the powers that be don’t want to upset the apple cart.  While most of the discussion has focused on terrorist/espionage threats I would argue that the arguments for a domestic intelligence agency are just as applicable for significant criminal threats coming from networks.  Too often law enforcement agencies (both local and federal) have to follow the headlines and respond to public perceptions of threats rather than actual threats to public safety and well being.  As a result, criminal threats that don’t generate local news headlines (and corresponding outcry from voting constituents) don’t generate pressure from elected officials to ‘do something’.

As one crime researcher told me:  ‘Look around you.  On the internet, in the newspapers, even in the phone book there are all sorts of advertisements for ‘escort’ services.  Do you hear anyone complaining or demanding the authorities stop prostitution because of those ads.  But…put a streetwalker on your corner and the police switchboard will light up like a Christmas tree.’  Since most people are going to respond to the perception of threats, there should be someone out there attempting to see if those perceptions are grounded in reality.

Do we have anyone looking out over the next 6, 12, 18 months (or longer) to try to figure out what trends we can expect to see among criminal networks?  What vulnerabilities those networks are exploiting and which ones can be used to exploit them?  Is anyone looking at the effectiveness of various suppression/prevention strategies?  On an even more basic level, are we confident we even have a handle on the number, characteristics and activities of criminal networks out there so that we can have the confidence that resource allocation is being done in a (semi) rational way?

Attention Gang Researchers!

Effective today, all of the underlying data for the New Jersey Street Gang Surveys conducted in 2001, 2004 and 2007 are available to the public.

You can access the data here.

Enjoy!

Once more into the breach

I’m still thinking about that DHS report that’s caused such a stir among the right.  This post got me thinking about how the wording of the report got interpreted differently by different audiences.  I suspect Malkin accurately captured the assumption of the right today (and could have described the left over the past eight years) when she wrote that “there are no coincidences” when discussing the current administration.

In fact, there are coincidences everywhere, as this video demonstrates (h/t Daily Dish)

And so, when members of the right read in the report that

“right wing extremists are antagonistic toward the new presidential administration and its perceived stance on a range of issues, including immigration and citizenship, the expansion of social programs to minorities, and restrictions on firearms and use”,

I suspect they understand that to mean:

“Everyone who exhibits any of these characteristics will be classified as a ‘right-wing extremist’.”

I’m guessing the writers assumed their product would only be read by law enforcement officials and so didn’t think they needed to insert a disclaimer or more complete defintion that would be understood without explicitly being made.  While that is taken for granted by most in the field, readers not familiar with intelligence work in the law enforcement/homeland security field would have no context or familiarity with which to place the report.

Therefore, I’d argue that the intent of that sentence was suppossed to be something along the lines of:

The definition of right-wing extremism (to the extent that it is of interest to law enforcement/homeland security personnel) depends not only on the ideology (mentioned above) of individuals but also on criminal activity connected to or in furtherance of that ideology.

In any case, it’s more a case of poor writing/organization than a harbinger of a sinister plan to lock up ‘undesirables’ into some sort of FEMA created gulag.

Therefore, blogging about tax policy isn’t a problem but arguing that someone should bomb the Treasury and then illegally acquiring explosives might put you in the ‘right wing extremist’ category.

Intelligence Analysis Training Review – Part 5

Part 1

Part 2

Part 3

Part 4

I’ve just completed the final portion of the Intermediate Intelligence Analyst course that I’ve been writing about and after thinking about parts 3 and 4 am ready to disgorge my final judgment.

If you just want to cut to the chase: This is a great course.  Maybe the best general analytical course I’ve seen.  It’s got some teething problems but none that can’t be overcome.  If I were running an analytical shop, this course would be part of every analysts’ career development plan

Ok, onto the details

Module 3 was designed “to improve the writing skills of intelligence analysts. This will be accomplished by providing the learners with an understanding of the writing process and the strategies that drive that process. Learners will engage in numerous problem-solving exercises and activities, in addition to engaging in numerous practice writings. The instructor will provide the learner with meaningful feedback that will improve the learner’s writing skills.”

The same problems with the course management software I mentioned in the first module were back with a vengeance. They used the ANGEL software which seems to be fairly widely represented in the educational field so I’m hesitant to say the software was bad but I’m not sure how to account for the numerous log on problems, clunky and slow posting performance, limited formatting options and maddeningly absurd email system. If there was one point of agreement across all the students (and, I suspect, among the staff as well) it was the universal dislike of the course management software.

That being said, even with good software I think there were some problems with the distance learning portion of the course. Distance learning is difficult in the best of circumstances and given the limited classroom time (about 7 total days) and the need to establish and emphasize good teamwork and collaboration the class missed an opportunity to use the social capabilities of the software to begin that team building. The course web page became a vehicle for submitting assignments and not much else, encouraging students to hop on the course website, upload their assignment and get the hell out of there as quickly as possible.

I’ve taken a number of distance learning courses of widely varying quality and the ones that have succeeded have had a teacher who keeps students engaged and communicating on the discussion boards/email system. Ideally students will do this on their own and sometimes they do, but a facilitator can’t count on that and should be prepared to prod the class to get them to speak up.

My recommendation:

Make discussions on the bulletin boards a required part of the coursework. One way I’ve seen it work is where the teacher will ask one (or more) questions per week and all students are required to respond to the teacher as well as at least two comments to other students in a meaningful way (which means “I totally agree!” wouldn’t count). For this course specifically, this greatly increases the amount of writing available from each student for evaluation without placing an undo burden on the students. They don’t need to write ‘War and Peace’, they just need to submit a well thought out answer and use appropriate citation rules. I’ve seen quite good discussions go on long, long after grading was done between students because their interest buttons were pushed. Perhaps this would be a good place to discuss some ‘big picture’ issues and let the fur fly: Why haven’t we had a major attack on U.S. soil since 9/11? Is there a meaningful difference between crime analysts and intelligence analysts and what should their roles be in law enforcement analysis? Should analysts focus on tactical or strategic products? The list could go on and on.

The module had two ‘big’ (big for a professional development course, positively puny for a graduate level course) assignments. The first was to read a series of articles from The Economist about the War on Terror and designed to “enhance the learner’s cognitive ability to write an intelligence product that demonstrates comprehension of ” audience, purpose, form and the collection and collation steps of the intelligence process.

The instructions went on to say:

Applying the considerations described in the lecture material, craft the documents to be appropriate for the different readers. Be prepared to defend you decisions regarding type and level of content, degree of analysis, general organization, level of vocabulary, and word choice…This exercise will focus on the ability of analysts to collect and identify significant facts from an assigned source and communicate the results in a written product for later evaluation and analysis…You are not “interpreting” the article, nor are you “arguing” for or against its point of view. You are summarizing the article, and your summary should be accurate and free from personal bias.

Now, this assignment didn’t have anything to do with the collection phase of the intelligence process since we were told what information source to use and there was no room for bringing in outside sources of information.

I was torn on this assignment since I hated the articles (which were superficial, poorly imagined and gave the impression that some editor just had a bunch of stuff lying around and decided to cram it into a ’special report’) but enjoyed the assignment. The length of the source articles precluded (in my opinion) any attempt to summarize each component and rather forced the writer to look for underlying themes which connected the scattered ideas and concepts (in other words, we were asked to do the work that the writers should have done). In order to do that however, you do need to be able to place the source document in some context and need to have at least a rudimentary understanding of current events in Pakistan, Afghanistan, Iraq, as well as European approaches to terrorism and current theories of terrorist dynamics in general. If you don’t have that this assignment becomes much more difficult. Likewise, if you are fairly well versed in these areas you can write this pretty easily.

My recommendation:

Use different source material. Ideally, pick a subject that won’t allow students to ‘game’ the system if they come to the table with prior knowledge about a subject. The purpose of this block is writing to different audiences and so it shouldn’t really matter what students are writing about. A subject that isn’t law enforcement or terrorism related should put students on a level playing field and therefore make it a bit easier to see how they are at the skills of summarizing, audience analysis, and writing skills. This would require the course managers to explain this concept explicitly to students so they can understand why they’re being asked to write about…the dangers of Astro-turf (or whatever) instead of what they do for a living.

The second assignment was to create an intelligence assessment which would have real value for the students’ agency. This would both serve to provide the agency with something of value for sending their people to school.

This brought out one problem with the course that I regarded as only relatively minor until this point.  In the student guide, each block of instruction and assignment was given a time limit.  The final writing assignment was slated for four hours.  I assumed that those limits were created for a purpose, namely to encourage analysts to organize their thoughts and produce a product under some pretty serious time constraints.  So, I took four hours and researched, wrote and submitted my paper for evaluation on the day it was due.

And I waited.

Nine days later I received an email addressed to the class which said ‘If you haven’t finished your assessment yet, please try to have it done in the next seven days.’

Now, had I known I had two weeks to research, write, and revise the assessment I would have ended up with a very different product.  You can have a valuable training experience either way you go but they really should have been more clear about this issue.

My recommendation:

Determine what training value you want students to take away from the assessment process and tailor the assignment to address it.

Using the time constraint forces analysts to quickly conduct decide what information they think they can get, how to organize it and how to characterize their work to their audience.  In addition to the product itself, analysts should be asked/mentored in the process of how to compress the analytical/writing process.

An assessment with a longer time frame between assignment and submission could have analysts demonstrate the various steps of the analytical and writing process.   In this scenario, I’d recommend breaking down the project into multiple mini-assignments to verify understanding and competence.  Steps might include:

• Develop your hypothesis
• Create a collection plan
• Apply analytical techniques
• Write your outline
• Write draft
• Revise
• Finalize

Each one of those steps could be an assignment that needs to be completed before being allowed to move on.

Module 4 consisted of two main portions:  a capstone exercise designed to merge the variety of skills taught in the earlier modules and a briefing of our assessments that we completed in Module 3.

The Capstone exercise was quite fun and well done.  I don’t want to reveal any spoilers but it centered around potential terrorist activity and we played various fictional fusion centers trying to understand what, if anything, was going on.  It wasn’t totally realistic since at a fairly early stage the federal authorities would have taken over an investigation like this but I’m not sure that was a serious flaw.  It occurred to me that there were numerous points during the exercise that the class would have benefited from a brief pause to discuss how things were progressing and how might things work in the ‘real world’.

One specific example revolves around information sharing and collaboration.  Our four groups were in different rooms and could contact each other via email or (more commonly) by walking down the hall.  A lot of confusion was avoided by walking 10 meters.  It might have been nice to have the analysts really think about how things would have changed if our fusion centers were separated by hundreds of miles or if we had institutional and/or personal biases which would have discouraged information sharing.

Also, each group was given free reign in terms of how to organize themselves, their information and their workload.  It would have been nice to discuss the thinking that did (or didn’t) go into that decision making process.  Was it thought or or just happen?  Was there a leader or group consensus?  Which worked better in which situations?

The briefings went well and I don’t have much to say about them.  They set up a system for peers to evaluate each other that seemed to work OK.

So…that’s it.  The course administrators were top notch and professional throughout.  Their dedication to the course and the class were clearly evident and they frequently exhibited more patience than I could muster.

Kudos to the Sacramento Regional Office of Homeland Security for a job well done.