What is Judgment?
Judgment is the expression of a mental activity that may be exercised with greater or less skill.
Information, whether raw or analyzed, enters the decision makers stream as part decision makers "appreciations." In his classic work, The art of Judgment, Sir Geoffrey Vickers defines "appreciation" as the product of interaction between reality Judgments-"What is going on?"-, value judgments-"what difference does it make?"-, and action judgments-what do we do about it?"-.
An appreciation comprises both an action decision and the rationale that underpins it. The quality of the decision depends on the thought devoted to each component and the thought applied to relating one component to another. Perfect knowledge of reality is not of much use without criteria for separating the important from the unimportant. Knowing clearly what is important is not much use in practical affairs without capacity for matching knowledge to action.
"innovative appreciation" describes a kind of analytical discovery of how to match up understanding of facts, assessment of their relative importance, and creative thought about what to do.
None of the elements of appreciation is fixed, they change rapidly and sometimes unpredictably. When one element changes, the others are likely also to change. The appreciative System diagram illustrates the process, but someone looking at it needs to bear in mind that it captures vickers' concept only if one imagines that the arrows are constantly in motion until the process is sufficiently complete so that action can actually be taken.
That this is a dynamic, not a static, image is critically important, because information gatherers and decision makers often make the mistake of assuming that one set of judgments or another is fixed. This mistake is most often made regarding value judgments, for it is all too easy to suppose that the actions or policies of another person or organization or government derive from ascertainable and more or less permanent interests. Such a supposition is often alluring because it is seldom wholly wrong. Interests are ascertainable and are more or less permanent. But the mix that answers the question "So What?" can change almost from moment to moment.
value judgments change along with changes in reality judgments and actions judgments. While interests may be constant, priorities among those interests are not.
initial reality judgments can be sharpened by taking care to distinguish clearly what is know from what is merely presumed to be true. presumptions often have great influence in appreciations. participants frequently push particular presumptions because they suit either their value preferences or their action preferences. information gatherers need to note presumptions of their own-and of others- that might have distorting effects, they need to gauge the mixture of known and presumed in the reality judgments of other agencies or organizations, and--at least as important--they must try to sort out what is known and presumed by the decision makers they hope to inform.
the precept here is not to counsel information gatherers to become advocates of particular courses of action. it is to counsel to become as sensitive as possible to the differing presumptions in the minds of users of information analysis.
"If analysts -information gatherers- see that there's a cliff on one side of the road, it's their duty to warn the people who are driving the bus."
- Robert Bowie, former overseer of CIA's analysts and before that an assistant secretary of state for policy planning
action judgments also fall into two broad categories. they answer one of two questions- what to do? And what to do? in other words, what strategy or broad course of action is appropriate, given the mix of reality and value judgments, and what tactics, or specific actions, could implement a strategy? there are at least five levels of disaggregation for these choices, applicable to both categories of action judgments, though in differing degrees.
first comes definition of the operational objective. though an individual or organization or government may have general objectives such as security or prosperity, an action judgment should include a definition of a measurably attainable objective.
second, an action judgment depends on some type of theory as to how the operational objective can be attained.
thirdly, an action judgment involves some type of plan that applies the theory to the particular case. the fourth and fifth considerations are resources and implementation. Are the necessary resources available? Can the plan actually be carried out?
Have questions? Want an answer more detailed than, well it depends? If so, submit your question to us and we will include your question and a detailed response in a future episode.
Enough with what others get wrong or don't do. Here is what we do and why we do it. This is a broad overview.
Functional Movement Screen
Tacit -v- Explicit Knowledge
Dysfunction -v- Compensation
Military Strategy, John M. Collins
The Sources of Military Doctrine, Barry Posen
In Episode 7, we take a detour to identify what is missing from self-defense and personal protection; identify how we address these gaps; why I believe there are no other complete self-defense/personal protection programs (aside from our own); and more. We are joined by one of our students, Crystal, who will share her unique insights, experiences, and thoughts on a range of subjects.
In this episode, we further explore what defines an expert, how to acquire expertise, and the relationship of Boyd's orientation, Robert Greene's Life's Task ("The first move toward mastery is always inward--learning who you really are and reconnecting with that innate force."), and answering the question, Who you are?
Discussion on the altercation between individual and law enforcement officer. Remember, viewing the video, what is visible in the video and excluding all else, what do you observe?
Deliberate practice involves two kinds of learning: improving the skills you already have and extending the reach and range of your skills.
"How can you tell when you're dealing with a genuine expert? Real expertise must pass three tests. First, it must lead to performance that is consistently superior to that of the expert's peers. Second, real expertise produces concrete results. Brain surgeons, for example, not only must be skillful with their scalpels but also must have successful outcomes with their patients. A chess player must be able to win matches in tournaments. Finally, true expertise can be replicated and measured in the lab. As the British scientist Lod Kelvin stated, "If you can not measure it, you can not improve it."
Things to Look Out for When Judging Expertise
Individual accounts of expertise are often unreliable.
Anecdotes, selective recall, and one-off events all can present insufficient, often misleading, examples of expertise. There is a huge body of literature on false memories, self-serving biases, and recollections altered as a result of current beliefs or the passage of time. Reporting is not the same as research.
Many people are wrongly believed to possess expertise.
Bear in mind that true expertise is demonstrated by measurable, consistently superior performance. Some supposed experts are superior only when it comes to explaining why they made errors. After the 1976 Judgment of Paris, for example, when California wines bested French wines in a blind tasting, the French wine "experts" argued that the results were an aberration and that the California res in particular would never age as well as the famous French reds. (In 2006, the tasting of the reds was reenacted, and California came out on top again.) Had it not been for the objective results from the blind tastings, the French wine experts may never have been convinced of the quality of the American wines.
Intuition can lead you down the garden path.
The idea that you can improve your performance by relaxing and "just trusting your gut" is popular. While it may be true that intuition is valuable in routine or familiar situations, informed intuition is the result of deliberate practice. You cannot consistently improve your ability to make decisions (or your intuition) without considerable practice, reflection, and analysis.
You don't need a different putter.
Many managers hope that they will suddenly improve performance by adopting new and better methods--just as golf players may think they can lower their scores with a new and better club. But changing to a different putter may increase the variability of a golfer's shot and thus hinder his or her ability to play well. In reality, the key to improving expertise is consistency and carefully controlled efforts.
Expertise is not captured by knowledge management systems.
Knowledge management systems rarely, if ever, deal with what psychologists call knowledge. They are repositories of images, documents, and routines: external data that people can view and interpret as they try to solve a problem or make a decision. There are no shortcuts to gaining true expertise.
How to Practice Deliberately
To people who have never reached a national or international level of competition, it may appear that excellence is simply the result of practicing daily for years or even decades. However, living in a cave does not make you a geologist. Not all practice makes perfect. You need a particular kind of practice--deliberate practice--to develop expertise. When most people practice, they focus on the things they already know how to do. Deliberate practice is different. It entails considerable, specific, and sustained efforts to do something you can't do well--or even at all. Research across all domains shows that it is only by working at what you can't do that you turn into the expert you want to become.
To illustrate this point, let’s imagine you are learning to play golf for the first time. In the early phases, you try to understand the basic strokes and focus on avoiding gross mistakes (like driving the ball into another player). You practice on the putting green, hit balls at a driving range, and play rounds with others who are most likely novices like you. In a surprisingly short time (perhaps 50 hours), you will develop better control and your game will improve. From then on, you will work on your skills by driving and putting more balls and engaging in more games, until your strokes become automatic: You’ll think less about each shot and play more from intuition. Your golf game now is a social outing, in which you occasionally concentrate on your shot. From this point on, additional time on the course will not substantially improve your performance, which may remain at the same level for decades.
Why does this happen? You don’t improve because when you are playing a game, you get only a single chance to make a shot from any given location. You don’t get to figure out how you can correct mistakes. If you were allowed to take five to ten shots from the exact same location on the course, you would get more feedback on your technique and start to adjust your playing style to improve your control. In fact, professionals often take multiple shots from the same location when they train and when they check out a course before a tournament.
This kind of deliberate practice can be adapted to developing business and leadership expertise. The classic example is the case method taught by many business schools, which presents students with real-life situations that require action. Because the eventual outcomes of those situations are known, the students can immediately judge the merits of their proposed solutions. In this way, they can practice making decisions ten to 20 times a week. War games serve a similar training function at military academies. Officers can analyze the trainees’ responses in simulated combat and provide an instant evaluation. Such mock military operations sharpen leadership skills with deliberate practice that lets trainees explore uncharted areas.
Let’s take a closer look at how deliberate practice might work for leadership. You often hear that a key element of leadership and management is charisma, which is true. Being a leader frequently requires standing in front of your employees, your peers, or your board of directors and attempting to convince them of one thing or another, especially in times of crisis. A surprising number of executives believe that charisma is innate and cannot be learned. Yet if they were acting in a play with the help of a director and a coach, most of them would be able to come across as considerably more charismatic, especially over time. In fact, working with a leading drama school, we have developed a set of acting exercises for managers and leaders that are designed to increase their powers of charm and persuasion. Executives who do these exercises have shown remarkable improvement. So charisma can be learned through deliberate practice. Bear in mind that even Winston Churchill, one of the most charismatic figures of the twentieth century, practiced his oratory style in front of a mirror.
Genuine experts not only practice deliberately but also think deliberately. The golfer Ben Hogan once explained, “While I am practicing I am also trying to develop my powers of concentration. I never just walk up and hit the ball.” Hogan would decide in advance where he wanted the ball to go and how to get it there. We actually track this kind of thought process in our research. We present expert performers with a scenario and ask them to think aloud as they work their way through it. Chess players, for example, will describe how they spend five to ten minutes exploring all the possibilities for their next move, thinking through the consequences of each and planning out the sequence of moves that might follow it. We’ve observed that when a course of action doesn’t work out as expected, the expert players will go back to their prior analysis to assess where they went wrong and how to avoid future errors. They continually work to eliminate their weaknesses.
Deliberate practice involves two kinds of learning: improving the skills you already have and extending the reach and range of your skills. The enormous concentration required to undertake these twin tasks limits the amount of time you can spend doing them. The famous violinist Nathan Milstein wrote: “Practice as much as you feel you can accomplish with concentration. Once when I became concerned because others around me practiced all day long, I asked [my mentor] Professor Auer how many hours I should practice, and he said, ‘It really doesn’t matter how long. If you practice with your fingers, no amount is enough. If you practice with your head, two hours is plenty.’”
Please take a moment to rate (5-stars, please) review (written reviews expose the podcast to a wider audience), subscribe (subscriptions are part of Apple's ranking algorithm which helps with expanding our audience), share (share the podcast with friends, family, and colleagues who seek self-improvement or professional development), engage (send your thoughts, questions, and feedback to Ben@DistrictCombatives.com). Thank you for your continued support!
In this episode, we provide greater color on the practical application of mental models. In personal protection terms, how it can affect your ability to prevail in a violent encounter. Remember, mental models, are comprised of your Individual Experiences, Cultural Experiences, and Institutional/organizational experiences.
We highlight a recent fight between two 14-year old girls in Brazil where one girl (or her mom, depending on where you source your information) brought a knife and used it to repeatedly stab and slash. This is a perfect example of mental models and getting inside your oppositions OODA loop. One girl was anticipating some type of fistfight; she came prepared for a fist fight; she entered into a fist fight. The other girl came prepared to prevail; she entered into a fist fight (showing the opposition what she anticipated); then she accessed and deployed a knife; she also came with her mother; her mother entered the encounter. The mental models of these two girls were in opposition. However, the girl armed with a knife and mother came better prepared.
One of the best discussions on mental models I've heard can be found here:
The Fog of War by Chris Hicks
Mental Models are incredibly important. We could easily spend several podcast episodes covering them. What is critical to understand is the importance of mental models not only as representations of how you perceive the world but also how others perceive the world. This is your initial access point to operating inside your opposition or teammates OODA loop.
Immediate Action Combatives "Biting to Defend against Triangle Choke"
As you can observe in the triangle choke video, mobility, specifically of the foot and ankle, hip and lumbar spine are requirements for both individuals. Given biting is not allowed in sport BJJ, we can safely presume the demonstration is directed to self-defense, which we can also safely presume if you find yourself in this position (either applying the choke or receiving the choke) you will be wearing shoes. Any shoe creates an external cage or brace to the foot and ankle significantly reducing ankle mobility and dexterity, in addition to impairing hip mobility. Next, the video literally shows the individual only attempting to use his jaw to "bite", no use of the hands, arms, or rest of his body. Clearly, this is bullshit, unless the individual is a quadriplegic. Since the individual has his left hand resting on his own left thigh, we know he is not a quadriplegic. Next, note how the position has to be perfectly applied, as verbally stated by Cecil AND how the opposition is only allowed to use his jaw when directed and then only after the leg is firmly clamped at or below the level of the jaw. Unfortunately, for the choker, your leg cannot teleport to this location and while it is moving into position, your opposition has ample time and space, as Cecil concedes, albeit reluctantly and hesitatingly, to bite. Of course, there are countless other issues with this particular example. Note, how there is no context in detailing how you arrive in a sport BJJ position. Let me ask you a question, how many street fight videos have you watched, either in person or on video, where someone pulls guard? Personally, I have not seen or watched a single fight where this takes place outside of sport or competition. And as we already know, in sport or competition biting is against the rules. What do rules create? Mental Models. What are we seeking to attack? The mental models of our opposition. Why does every sport fighter say their sport techniques will defeat any eye gouge, small joint manipulation, bite, etc. when the sport in which they compete makes each of these off-limits.
Let's check out a group of professional UFC fighters taking on some Marines in combatives. Before we watch the video or even click the link, write down what you think will happen. Be as detailed as possible.
The key concept to take away from this breakdown is that you can attack the mental model of your opposition, whose belief structure (mental models) may not apply to the situation at hand. What is key to comprehend and internalize is that you cannot just practice techniques and tasks, you must understand processes that can be applied to the environment that is going on. The environment is constantly changing and the only way to succeed, to prevail is if your rate of adaption is equal to or greater than the rate of change taking place around you. Your techniques become the baseline in which you leverage off of. What will make you great is an ability to perform the basics better than anyone else. And when you can do the basics incredibly well, when you need to adapt, when you need to change the adaption or the use of those basics, because you know them so well, you can keep principles and priorities in place. You won't lose the forest for the trees. You keep the principles in place but adapt the techniques in which they are applied.
Winning in a Complex World, by Col. Jim Czarnik
Mental models are the stories we construct to understand how things work. They mirror the events or system they are modeling, but they capture only a limited aspect of those events or that system. We form mental models from the way we understand causes.
The mental models of experts include more knowledge and also enable the experts to see more connections. These--knowledge and connections--are two defining features of complexity. The mental models of experts are more complex than of other people.
Expertise gives rise to errors. With more experience, we learn more patterns.
YOU CANNOT SEE WHAT YOU DO NOT KNOW
skilled performance depends on the way we look and listen. it depends on what we can notice and what kinds of discriminations we can make.
Our mindsets frame the cues in front of us and the events that are unfolding so we can make sense of everything.
Two conditions are necessary for reliable intuitions to develop.
- Situation must be reasonably predictable, and
- People must have opportunities to learn.
Low predictability situations make it unlikely that people can develop expertise because it is so hard to identify reliable cues.
Boyd's concept of Orientation is similar to Gary Klein's concept of Mental Models. Mental Models are comprised of you
- Individual experiences
- Cultural experiences
- Institutional/Organizational experiences
Boyd's Orientation includes
- Cultural traditions
- Analysis and Synthesis
- Previous Experience
- New Information
- Genetic Heritage
Both Orientation and Mental Models inform our Values. Values, as you may recall from Episode One: What is Judgment? are one of three elements to judgment.
- Action Judgments (What is to be done?)
- Reality Judgments (What is going on?)
- Value Judgments (What difference does it make?)
As you can now see, our value judgments are informed by who we are. If you cannot answer this question, your value judgments will be inconsistent; your observations will be inconsistent; and finally, your mental models will remain simplistic and lack the sophistication developed with the acquisition of expertise.
To improve Performance we need to do two)things.:The down arrow is what we have to reduce, errors. The up arrow is what we have to increase, insights. Performance improvement depends on these two things.
Insights shift us toward a new story, a new set of beliefs that are more accurate, more comprehensive and more useful. Which thereby sophisticates our mental models, sophistication and complexity are two defining characteristics of the mental models of experts.
Our insights transform us in several ways. They change how we understand, act, see, feel, and desire. They change how we understand. They transform our thinking; our new story gives us a different viewpoint. They change how we act. In some cases insights transform our abilities as well as our understanding. Insights change our notions of what we can do. These shifts go beyond a richer story about how the world works. Insights transform how we see; we look for different things in keeping with our new story. Insights transform how we feel--what excites us or makes us nervous. Finally, insights change our desires; the new stories shift our goals, leading us to give up some ambitions and pursue others.
"Insight is when it happens, everything that happens afterward is different." Hilary Mantel made the same observation in Wolf Hall: "Insight cannot be taken back. YOu cannot return to the moment you were in before."
Orientation is who you are. Always be orienting!
The OODA loop is commonly cited as a way to beat an adversary or opponent. From the military to business, completing your OODA loop faster than your adversary is cited as a path to victory. Further, Observation is cited as the key to beginning the OODA loop process and Orientation is defined as orienting towards the critical element identified during observation. This is bullshit. And I'm going to tell you why. Don't just take my word, I will use John Boyd's, creator of the OODA loops own words to destroy this gross misrepresentation.
First, Observation is NOT the first part of the loop. In fact, Boyd himself states:
"Note how orientation shapes observation, shapes decision, shapes action, and in turn is shaped by the feedback and other phenomena coming into our sensing or observing window.
Also note how the entire “loop” (not just orientation) is an ongoing many-sided implicit cross-referencing process of projection, empathy, correlation, and rejection." [Source: The Essence of Winning and Losing, John Boyd]Read More
In the first episode, we establish a working definition of Judgment and explain the three types of judgment.
We use the process of analytical discovery of how to match up an understanding of facts, assessment of their relative importance, and creative thought about what to do.
What is Judgment?
Judgment is an expression of a mental activity that may be exercised with greater or less skill.
What are the three types of judgment?
- Action Judgment "What is to be done?"
- Value Judgment: "What difference does it make?"
- Reality Judgment: "What is going on?"
Perfect knowledge of reality is not of much use without criteria for separating the important from the unimportant.
Knowing clearly what is important is not much use in practical affairs without capacity for matching knowledge to action.
Initial reality judgments can be sharpened by taking care to distinguish clearly what is known from what is presumed to be true.
Dealing with Dictators: Dilemmas of U.S. Diplomacy and Intelligence Analysis, 1945-1990, by Enerst May and Philip Zelikow
The Art of Judgment: A Study of Policy Making, by Sir Geoffrey Vickers
Col. John Boyd's Observe-Orient-Decide-Act "OODA" Loop
Boyd's OODA Loop Presentation Slide Deck
Thinking in Time: The Art & Science of Judgment origins.
From 1986 to 2002, Harvard University's Kennedy School of Government had an executive program for senior managers in the U.S. intelligence community, known as the Intelligence and Policy Program. It ran once or twice a year for one to three week. Participants typically had twelve to twenty years of experience. Most came from the Directorate of Intelligence (DI) in the Central Intelligence Agency (CIA). Some came from CIA's Directorate of Operations (DO), the home of the U.S. clandestine service; end-of-course certificates for the latter had blanked-out names because the names under which they enrolled were not their own. Some other participants came from the National Security Agency, the center for U.S. signal interception and code-breaking; the National Reconnaissance Organization, which oversees collection of intelligence by satellites; the Defense Intelligence Agency, the military establishment's not-so-mini-CIA; and the intelligence branches of the armed services, the Department of State, and the Department of Energy. A handful came from federal law enforcement agencies.
This executive program was the brainchild of Robert Gates, who in 1986 had just become Deputy Director of Central Intelligence. He would later be Deputy National Security Advisor for President George H.W. Bush, as well as Bush's Director of Central Intelligence (and later Secretary of Defense under President Barack Obama). In 2002, he became president of Texas A&M University. As he outlines in his unique, revealing, and quite wonderful memoir, Gates was a career analyst in the CIA who chanced early on to have tours of duty in the White House, first under Henry Kissinger, then under Zbigniew Brzezinski. It struck him that his work in the White House on policy issues did not connect often with the work of his former colleagues in the intelligence community. When back in that community, he puzzled over the fact that work there connected so little with what was being done by his whilom colleagues in the National Security Council staff and other parts of the government where policy decisions were framed, taken, and executed.
Gates devised the Kennedy Schools new executive program after consulting with, among others in the School, Richard Neustadt and Ernest May, Dean Graham Allison and Academic Dean Albert Carnesale, and Peter Zimmerman and Nancy Huntington, who managed the other executive programs. The Intelligence and Policy Program aimed to teach managers in the intelligence community how to think about the needs in the policy community and about ways in which they and their associates might better serve those needs. This would be done in part by exposing them to elements of decision, bargaining, and organization theory, but primarily through Socratic discourse centered on case studies.
Neustadt and May had, for several years, taught a case-based course, on the basis of which they had then just published Thinking in Time: The Use of History for Decisionmakers. The first cases used in the new program were cases that they had developed, with assistance from the National Endowment for the Humanities, all focused on presidential choice-making but tangentially involving use of intelligence. Of these cases, those that seemed best-suited for the new program concerned the Pearl Harbor attack, the onset of the Korean War, the Bay of Pigs affair, the 1962 missile crisis, and the Americanization of the Vietnam War.
A contract between the Kennedy School and CIA's Center for the Study of Intelligence provided not only for the new executive program but also for development of one to three new case studies annually, designed specifically to foster learning about the intelligence-policy nexus. The other cases were to be wholly unclassified and subject to no constraints other than those normal in any academic research project. The CIA undertook, however, to facilitate the effort, particularly by declassifying documents and identifying potential interviewees.
A steering committee helped guide the selection of topics for case studies. Chaired jointly by the Dean of the Kennedy School and the Director of Central Intelligence, the committee consisted of senior faculty from Harvard University, current and former high-level officials from Washington's intelligence and policy communities, and majority and minority members of the two Congressional intelligence oversight committees.
Between 1986 and 2002, the Program developed more than forty cases, some of which had multiple parts and two of which became books; one on the 1962 missile crisis, the other on Germany's defeat of France in 1940.
The CIA-Harvard contract was itself necessarily unclassified, for Harvard has since World War II been unbending in its refusal to do classified research. The CIA had as little experience of writing unclassified contracts as Harvard had of concluding contracts with an intelligence agency. On the Harvard side, with memories of Vietnam-era riots still all too fresh, deans and professors worried lest news of the contract lead to protest demonstrations. Although Harvard sponsors had agrees that the objective was important enough to justify running such a risk, they tried to minimize it by issuing a low-key press release lat on a Friday afternoon, when it was least likely to catch the attention of newspaper editors. The ploy failed; the story was featured in both Harvard student newspapers and in the Boston press. However, to the wonderment of the Harvard contingent, the headlines lauded Harvard for "opening up" the CIA.
The CIA had already placed a certain number of senior officials in universities as "officers-in-residence" to teach, do research, or just provide information about the intelligence community. The program at the Kennedy School benefited from having a succession of such officials, whom the School designated as research associates, who stayed for at least a semester, and who helped develop teaching cases for the program. Ocupants of this post included William Kline and James Worthen from the Directorate of Intelligence and two others who, after their retirement from the CIA, spent some additional years at Harvard: Charles Cogan, who had headed the Near East Division of the Directorate of Operations and been station chief in Paris, and Douglas MacEachin, an analyst of strategic forces, who had been head of the Directorate of Intelligence.
Neustadt and May managed the program from the Harvard side--May from beginning to end, and Neustadt keeping a hand in long after his formal retirement in 1989. They were joined from time to time by, among others, Gregory Treverton, who went on to become a Vice Chairman of the National Intelligence Council and then a Senior Policy Analyst for the RAND Corporation; Richard Haass, later Director of Foreign Policy Studies at the Brookings Institution, head of policy planning for Secretary of State Colin Powell, and president of the Council on Foreign Relations; Robert Blackwill, later Ambassador to India and then a deputy to National Security Advisor Condoleezza Rice; Kurt Campbell, later senior vice president of the Center for Strategic and International Studies; and Jessica Stern, a member of the Kennedy School faculty and an expert on terrorism.
From his arrival at the Kennedy School in 1991, Philip Zelikow became, along with May and Neustadt, a central figure in the program. After Neustadt retired, May and Zelikow became the principal architects, editors, and sometimes authors of its case studies. The executive program in its early years depended, like most programs in Harvard's professional schools, on cameo appearances by a variety of faculty and visitors. Later, the program was taught almost exclusively by May and Zelikow, who were sometimes in the classroom six to eight hours a day.
May and Zelikow--coached, of course, by fellow faculty members and by friends outside--gradually developed a set of precepts which participants in the executive program reported in feedback to have proved of real use in their work. The reader is warned that they are less than fully meaningful absent the opportunity to discuss them along with the case studies, which serve to explain them and amplify them.
We hope that this will be widely used in the study and teaching of intelligence analysis.