This contribution was published on The Chaos Group on June 29, 2020
There is far too much information to process: what is the analysts’ best way to wade through it all?
When I worked in security intelligence in Canada over a 32-year career the collection, processing and analysis of information was the bread and butter of what we did. We received intelligence requirements from our government agencies and figured out the best way to meet those requirements. Sometimes that was through human sources; at others it was through a ‘technical’ one (SIGINT, imagery, etc.).
Even back then we were swimming in data. The analogy we used was ‘drinking through a firehose’. That was then. I cannot imagine how many orders of magnitude the problem has grown today. The challenge is not to find information: that part is easy. The real task is to find the nuggets of truth and accuracy in an ocean of noise.
Intelligence agencies have ways of determining whether a source is lying or not. These include obtaining similar information from multiple sources, what we call ‘corroboration’. There is also an attempt to see whether the source is fabricating the intelligence by subjecting him/her to a polygraph.
Neither of these is perfect (many doubt the usefulness of the polygraph at all). Examples from the 2003 US invasion of Iraq serve as a painful reminder how difficult this is.
The US tried to make the case that the Iraqi regime of Saddam Hussein had weapons of mass destruction to justify the need to remove him. The main human source was named Curveball, a chemical engineer who lied about the Iraqi government’s biological weapons program (an accomplishment he is proud of as it helped to end a brutal dictatorship).
There was also intercepted communications that purported to back up the US conviction that there was a program. It turned out that facilities managers lied about their progress out of fear that a truthful rendering would end up badly for them. All in all, the US intelligence community came to the wrong conclusions, the invasion took place and Iraq is where it is now.
Determining how good the information is
As the vast majority of analysts do not work for intelligence agencies, how can they get better at determining how good the information is upon which they rely for their work? While there is no foolproof system (and never will be) there are steps that can be taken to make better decisions on what to use and what to reject:
- what is the source’s track record (if this can be determined)?
- what, if any, agenda does the source have?
- how plausible is the information? If something is too good to be true perhaps it is not true.
- what are other sources saying?
- does this piece of information fit with your working theory? If so, are you choosing it because you want your ideas to be seen as supported?
These questions are hard to answer at times, especially in an era of misinformation and disinformation. Furthermore, there are no guarantees that you will have the best, most accurate information available when you need it. Hence, you have to make do with what you have in light of probable time constraints and delivery pressures.
Related: The challenges of gathering intelligence during a pandemic
Analysis is tough and takes time to get good at. As long as you are open to new ideas and are willing to question your own conclusions you will get better at it. After all, the world needs people to wade through the maelstrom of data and make sense of it, providing good advice to those who require it.
2 replies on “What makes a good analyst?”
Part 2:
So what makes a good analyst, assuming the intelligence service has a culture that supports good analysis, as an activity, and good analysts, as a group of people.
Of course they must have good general knowledge, a capacity to write well, in clear, precise and engaging language that does not merely parrot the hackneyed and career safe phrases of their specific service and not fixated on brevity at the expense of accuracy and clarity.
They must be able to distinguish between the reliability of the source of information and the credibility and plausibility of the information itself. Of the former, they must be able to see the record of the source, know the nature of the source (sigint, or humint) and know whether, in the case of humint, the access the source to information, and at at , accurate information, has and also whether the humint source has a record of reporting the information accurately. All this is the skills that analysts develop to assess information and requires them to know, something of the theory and psychology of information, belief and knowledge.
They also need an eye for detail as well as a willingness to ask: Why is this happening? Don’t simply take things at face value – or because the Party Line says it is so. If it is the party line, then you need to be especially suspicious.
But what then? They need the following, in my experience:
– independence of thought: just because someone has concluded that X is the case, does not make it so. Identify group think – and avoid it;
– awareness of their own biases, beliefs and theories;
– humility in being prepared to jettison their beliefs and theories and remember, as Cromwell said: “I beseech you, in the bowels of Christ, think it possible that you may be mistaken.” No one is perfect.
– lack of ego: the shame is not in being wrong; the shame is not admitting you are and thinking you are infallible;
– diligence: working away, slowly, methodically. Making lists, keeping records of where you saw something, looking for coincidences – and especially seeking out anomalies and ontrary information and not dismissing it because it disagrees.
– courage: don’t just case in and go with the group. Don’t be intimidated by a SME: subject matter expert. Ask them to teach you, pick their brains.
– courtesy: say where you got an idea, give credit, and praise.
– curiosity: be interested in things and ask Why? Know that people and information does not just appear, be emerges from the soup of life. For instance, if people use specific language, then that tells us something about how they think, their world view – and the people and idea with whom they have had contact.
– mindful: know that your assessments make well involve the expenditure of public monies – or human lives.
– creative: look for and always be open to better ways to think about an issue, better ways to find information; do not default to the, “if it is classified then it’s better than open source”. Remember the anecdote from Soviet times. The President of the USSR, realizing that the Soviets no longer could attract ideologically motivated spies in the West said to the head of the KGB: “Tell me Comrade, what do you need? Cost is no object. Ask and you will be given”. To which the chairman of the KGB replied: “A subscription to the New York Times; and one to the Washington Post”.
Skeptical particularly of “analytical tools”: some agencies try to “formalise” analysis, by the use of “analysis of competing hypotheses” or “structured analytical judgement”. Apart from the fact that instituional bullies misused them to force a conclusion on unwilling analysts, they lead to the exclusion of countervailing information and often the exclusion of significant but seemingly trivial information. They make analysts lazy, risk averse and enable analysts to transfer responsibility: the system said Yes! Humans do not think in terms of hypotheses or structured judgments. It is much more like identifying a painting as being the work of a specific artist. Analysis is a skill and an art, not merely a clinical application of an algorithm or a machine, where you crank the handle and the assessment results. As Hippocrates says: Life is short; the Art long. And judgement difficult.
Logical thinkers: be able to identify reasons and construct arguments: Premise; Premise; Conclusion; or Conclusion: premise; premise.
Intelligence analysis is an exercise in “applied epistemology”. An analyst needs to know about the nature of knowledge and belief, but know how that is applied in the real world. A person can be all these things, but if the institutional culture in which they work is not supportive, then no amount of being a good analyst will produce “good analysis”.
And while a person can learn how to do analysis, and practice can make a person better, there is a lot, in my experience, to the belief that analysts are to some extent born, not made: you have to have a brain wired for it. This is why some subjects are over represented: history, classics, philosophy.
Anyway, that is my theory. This is informative: https://www.dailymotion.com/video/x2oh8ia.
Hi Phil
Two comments – in two parts.
Part 1:
About the US (and allies’) decision to invade Iraq. Your narrative is not, to my recollection, quite correct. According to a public narrative, the US and the UK via CIA and MI6 respectively, were told that Iraq had no WMD. Moreover, the US was told that CURVEBALL may not be reliable. See: The Spies Who Fooled The World [https://www.youtube.com/watch?v=UOsHLA1CMPI] This is more or less on the money, as per my recollection. There are also these press reports, which even taken with a grin of salt, indicate that the US and UK governments were intent on invading and “sexed up” the information they had or ignored those gainsayers within their analytical community:
A. Iraq: The spies who fooled the world: https://www.bbc.com/news/uk-21786506
B. The Spies Who Fooled The World: https://www.bbc.co.uk/mediacentre/proginfo/2013/12/panorama-spies.html
C. MI6 and CIA were told before invasion that Iraq had no active WMD: https://www.theguardian.com/world/2013/mar/18/panorama-iraq-fresh-wmd-claims
D. An interesting take from German media: How German Intelligence Helped Justify the US Invasion of Iraq: https://www.spiegel.de/international/world/the-real-story-of-curveball-how-german-intelligence-helped-justify-the-us-invasion-of-iraq-a-542840.html
So why did the intelligence communities in the US and UK appear to support the claim that Iraq had WMD? [This was also the case in one other Five Eyes country too, that the PM of that country spoke publicly of intelligence assessments supporting the view.]
The political class in those countries was determined to invade – and the heads of agencies fell into line. [I saw this in several cases.] In other words, they concocted judgments to support the course of action their political masters wanted to take. Not all did, however. At least one agency in a Five eyes country told their government that Iraq had no WMD, but that agency was ignored. And there were many analysts who raised objections, but were ignored. And in some cases, their careers suffered or were destroyed.
If ever there was an environment that punished dissent, it is the intelligence community – and for the same reason religious cults do: conformity is crucial to survival and there is no outside critic that has much, if any, effect. In a closed community, being part of the herd and being seen to be part of the herd is everything. This is also why the the British services managed so well during WWII. MI5 did manage a culture of dissent and discussion, owing largely to the leadership of Guy Liddell. As you no doubt know, they ran Double Cross. The Germans, in contrast, were heavily invested in “group think”, not disagreeing with superiors and so forth. [But why would you? You would be shot.] And the British services ran rings around them for the most part, because a number of the British analysts knew this about their German adversaries. [The main exception there is the Nordpole deception ran by Herman GISKES. And to make the point: Leo MARKS realised what the Germans were doing and when he tried to warn his superiors was told to shut up, however, there may be more to that than has thus far been revealed.]
The lure of “Group Think”, both for career and for personal reasons – no one wants to be a pariah – and compulsive need or just because it is easier – to be “one of the in-group” who have the secrets, is what Kim Philby exploited. And that is a problem is some services to this day.
But, some agencies are aware of this weakness and make efforts to encourage dissent, and encourage people to challenge the “Party Line” and the “Group Think” and work to suppress the “in-group” culture. But astute analysts know that if you want a career, you don’t act up. You toe the party line. But contrarians are what makes intelligence services secure and successful. It was, according to some sources, a bunch of contrarians, largely analysts, who rumbled Ames. See: “Circle of Treason”. But on the other side, also from the CIA, we can see that toeing the party line had benefits: What happened to those who were aware of the “Enhanced Interrogation Program” – i.e. torture program? Most of those who objected, left or were pushed out. The clearance re-validation system is a really effective way to do that. (I have seen that at first hand in the service of another country.) Those who were “on the program” remain, and are promoted, including one person who destroyed the tapes of the torture sessions, according to press reports: https://www.washingtonpost.com/news/fact-checker/wp/2018/05/11/cia-director-nominee-haspel-and-destruction-of-interrogation-tapes-contradictions-and-questions/
And that august source of record, wikipedia: https://en.wikipedia.org/wiki/2005_CIA_interrogation_videotapes_destruction#:~:text=Rodriguez%20sent%20a%20cable%20to,copying%20them%20on%20a%20decision. [Good place to start, in looking for sources, but best not to rely upon it.]
So, what can we conclude? A person can be a competent analyst but unless there is a culture to enable not only those talents to flourish but dissent to be encouraged, ideas to be tested and conclusions to be given based on the information, free of political interference, then the profession of analysis becomes somewhat pointless. And you end up with Normandy; or moving your forces to Greece, when the invasion is in Sicily.