No one embodies the rising fortunes of counterinsurgency better than David Kilcullen. A political anthropologist with the Australian army, Kilcullen spent the 1990s at the Land Warfare Studies Centre in Duntroon, Australia finishing a doctoral dissertation on insurgency in Indonesia. At the time, counterinsurgency was largely relegated to military history after its heyday in the 1950s and ’60s, when it served as the imperial response to wars of national liberation and communist insurgencies.

Then came 9/11. By 2005 Kilcullen was in the Pentagon, collaborating on the Quadrennial Defense Review. He later became a special advisor to Condoleezza Rice.

Around the same time, General David Petraeus returned from Iraq, where the Americans were floundering in the face of a growing insurgency. Part of the problem, Petraeus later wrote, was that the U.S. Army had not produced a counterinsurgency manual for twenty years, leaving a doctrinal and instructional vacuum. So a group came together to rapidly compile what would become The U.S. Army/Marine Corps Counterinsurgency Field Manual. One of the leading contributors, John Nagl, is president of the Center for a New American Security—a think tank with considerable leverage in the Obama administration—where Kilcullen was a senior fellow. Kilcullen’s theories were prominent in the new Field Manual, and he was appointed Senior Counterinsurgency Advisor to General Petraeus during the 2007 Surge. The Field Manual has since acquired an almost iconic status, and Kilcullen has continued his work on counterinsurgency with his own book, The Accidental Guerrilla: Fighting Small Wars in the Midst of a Big One.

Although the Field Manual and Kilcullen’s book are different creatures—one is an instruction manual for commanders; the other combines instruction with memoir—together they provide valuable insight into the theoretical assumptions and operational practices that have transformed America’s war on terrorism.

The Field Manual has been called innovative, even radical. But those terms apply less to the strategic ideas it presents than to the new direction it advocates for the U.S. military in Iraq and Afghanistan. In conventional warfare, the objective—to destroy the enemy by killing its soldiers—is enemy-centric. In counterinsurgency operations, the objective is to provide immediate security to the population and earn its enduring loyalty.

In contrast to Donald Rumsfeld’s “shock and awe,” the new approach requires patient and long-term engagement. The Field Manual emphasizes the slow process of separating insurgents from the general population through methods of “population control”—the first stage in “clear-hold-build,” to use the jargon of the theory. As the section titled “Paradoxes of Counterinsurgency Operations” explains, “sometimes, the more you protect your force, the less secure you may be” and—in the most striking non-Rumsfeldian moment—“sometimes doing nothing is the best reaction.” The first paradox suggests that U.S. forces may have more success if they leave their heavily fortified compounds, set themselves up in the neighborhoods, and do the hard work of building relationships. The other paradox—sometimes doing nothing is best—encapsulates a complex reality of irregular war. Since the object is to win over the population, any strong use of firepower may do more harm than good in the long term, alienating civilians and pushing them into the arms of insurgents.

As the title of Kilcullen’s book indicates, the pivotal figure of analysis is the accidental guerrilla: a sometimes-tribal agent, often with specific local grievances, who is cynically manipulated or simply swept up into the larger global fight between al Qaeda and the West (or, in an earlier generation, between the Soviet Union and the United States). Although this person bitterly resists U.S. forces in Iraq and Afghanistan and participates in insurgent and terrorist activities, his interests are not the same as al Qaeda’s, nor is his fight ideological.

To address this phenomenon, Kilcullen recommends a policy of “disaggregation,” separating accidental guerrillas and local fighters from transnational terrorists. This approach is not entirely novel. As early as 2006, Great Britain ceased using the phrase “the war on terror” precisely because it allows disparate groups to “feel part of something bigger.” Until recently U.S. policy moved in the opposite direction, lumping together different actors and threats as elements in a global war on terrorism.

For Kilcullen, the accidental guerrilla is not only an actor, but a process. Here Kilcullen is both more original and more insightful. He shows how, through patronage and intimidation, al Qaeda insinuates itself in remote and largely self-governed localities. From these safe havens, al Qaeda initiates acts of provocative violence. When outsiders intervene to disrupt al Qaeda, they often inflame local resentments and create new accidental guerrillas. To break the cycle, Kilcullen proposes “counterinsurgency plus”: an approach that combines the focus on population security with counterterrorism and nation-building.

At first blush, then, the counterinsurgency efforts appear a tactically superior, yet more humane and culturally sensitive alternative to earlier U.S. strategies in Iraq and Afghanistan. But a deeper understanding requires a sense of history. And even a cursory review of that history reveals the colonial roots of contemporary counterinsurgency theory and practice. Thus we have to ask: how feasible is a counterinsurgency strategy without the support of colonial institutions and practices?

Moreover, colonial counterinsurgency campaigns were by nature anti-nationalist. So along with issues of feasibility, their modern-day equivalents face a serious problem of legitimacy. To be sure, current proponents of counterinsurgency argue that U.S. forces in Iraq and Afghanistan are not anti-nationalist warriors, but are allied with national leaders. But this response raises another question: can counterinsurgency efforts by foreign forces genuinely promote a nationalist project?

Strategists who see counterinsurgency as a straightforward tactical issue fail to appreciate the questions of legitimacy it raises.

A handful of texts make up the canon of counterinsurgency theory. The first, Colonel C.E. Callwell’s Small Wars (1896), offers a monumental study of the imperial “expeditions” in new territories or rebellious older ones, which occurred in every yearof Queen Victoria’s long reign. A commissioned officer of the Royal Artillery, Callwell fought in the Afghan War of 1878-1880 and in the first Boer War. From 1886 onward, he served in the intelligence branch of the War Office, making him one of the earlier soldier-bureaucrat-spies, a professional class that has come to dominate irregular war. Despite its dated quality and racist Victorian idiom (“expeditions against savages and semi-civilized races),” Small Wars retains the distinction of identifying precisely the key element of irregular war, which remains central to any modern understanding of counterinsurgency: “organized armies . . . struggling against opponents who will not meet them in the open field.”

If efforts to understand the nature of irregular warfare are born at the high noon of Victorian imperialism, they come to maturity in the sunset of empire. Responses to nationalist struggles from Algeria to Indochina (against the French) and from Kenya to Malaya (against the British) produced what are now regarded as the classics of counterinsurgency studies: works by Roger Trinquier, David Galula, Sir Robert Thompson, and Frank Kitson.

Trinquier, an officer in the French colonial infantry, served in China, Indochina, Algeria, and the Congo. In 1964 he published Modern Warfare: A French View of Counterinsurgency. That same year Galula, fresh from France’s disastrous loss in the Algerian war, published Counterinsurgency Warfare: Theory and Practice.

While the French were fighting in Algeria, the British were facing their own insurgency in Malaya. Thompson, a key figure in the British administration there, went on to writeDefeating Communist Insurgency in 1966. Five years later Kitson, a British Army officer who fought in Malaya as well as Northern Ireland, published Low Intensity Operations, which reiterated many of Thompson’s ideas and emphasized the use of propaganda, or, as it is now called in the military, “information operations.” While theField Manual’s annotated bibliography recommends all of these works, it reserves a special acknowledgement for Thompson and Galula.

It is worth noting that Galula’s book details lessons of a lost campaign. In fact, almost every foreign effort to defeat a nationalist insurgency in the twentieth century was a failure—Algeria and Vietnam are the norm.

The most-cited contrary example is the British “success” against a communist insurgency in Malaya that lasted from 1948 until 1960. Yet that case, too, is cold comfort. The Field Manual and Kilcullen both rightly note that the British efforts to separate the neutral population from the insurgents, to institute joint civil and military commands, and to train police as well as combat troops to suppress the insurgency are important lessons. But one undeniable element in Britain’s success was that the counterinsurgency resulted in independence for the Malayan Federation in 1957. Many of the counterinsurgency tactics worked primarily because the British promised to leave. Colonial counterinsurgency operations from the middle of the twentieth century onward made little progress until the colonizers offered concessions toward independence and self-rule. In other words, the successes that military strategists attribute to counterinsurgency tactics have far more to do with the end of foreign presence.

The intellectual legacy of counterinsurgency theory is further blurred by the changing nature of the opponents. Counterinsurgency is essentially a reactive doctrine, a response to a particular threat. An encompassing definition of that threat, of insurgency itself, is therefore hard to come by. The most common definition stresses the role of an armed rebellion against established authority, but, as Kilcullen argues, contemporary situations tend to be hybrids of local conflicts and transnational terrorism. Moreover, changes in technology have radically altered the landscape in which insurgencies grow and transnational terrorists operate: consider al Qaeda’s use of the Internet to recruit, raise funds, share information, and plan.

Counterinsurgency theory from the 1950s and 1960s also fails to account for cases—such as Afghanistan—in which “the insurgent movement pre-dates the government,” as Kilcullen puts it in his article “Counterinsurgency Redux.” But if insurgency is an armed movement to overthrow an established government, how can it predate its opponent? The answer is that it can’t, and this is why many of today’s counterinsurgency campaigns resemble colonial small wars: they aim to impose a new authority, not to re-establish a political order. These efforts may lie anywhere along a spectrum of legitimacy, from nation-states trying to impose a central authority on their under-governed territories—such as Pakistan in its Federally Administered Tribal Areas—to those seeking to bolster entirely new governments—such as the United States in Afghanistan. Establishing this authority always involves resistance from peoples and regions that were previously largely autonomous.

The most troubling legacy of the classical theory, however, lies elsewhere. Drawing principally upon Galula’s and Thompson’s reflections on the French and British experiences in Algeria and Malaya, respectively, Kilcullen and the Field Manual offer a comprehensive approach combining civil and military functions with a sustained focus on securing the population. Their recommendations are styled heavily on imperial policing. Many of the measures that theorists and military brass have put their faith in were successfully enacted in colonial counterinsurgencies because they already existedin the arsenals of colonial administration.

The blurring of combat and policing, normal and exceptional circumstances, defined imperial attempts at achieving law and order.

Thomas Mockaitis—a champion of British counterinsurgency and its current applicability—notes in British Counterinsurgency, 1919-1960, the tactics of colonial counterinsurgencies were “made possible by the extraordinary control that a colonial power could exercise over subject peoples and cannot easily be duplicated.” It is no wonder, then, that counterinsurgency specialists occasionally grow wistful for the colonial setting of early counterinsurgency efforts. In the preface to his influential 2002 book, Learning to Eat Soup with a Knife: Counterinsurgency Lessons from Malaya and Vietnam, Nagl considers the difficulty of applying those lessons in Iraq. The methods that succeeded in Malaya did so, he says, because “when the insurgency began [the British] had been in the country for well over a century.”

Colonial governance itself emphasized a comprehensive civil-military approach while also insulating the population from insurgents. These methods collectively allowed administrators to move easily into a counterinsurgency posture.

Much of the comprehensive civil-military approach employed by the British in their counterinsurgencies would not have been possible without the decidedly colonial and ubiquitous figure of the district officer, a staple of postcolonial films and literature, such as Wole Soyinka’s Death and the King’s Horseman. Whether referred to in different colonial bureaucracies as the “collector” or the “district magistrate,” he was at once chief executive, judge, tax officer, and administrator. Aided by another powerful local officer, the police commissioner, the district officer functioned in dispute resolution as both prosecutor and judge. When trouble arose and the army stepped in, the commanding officer would rely heavily on the district officer, and if the situation escalated into a full-blown insurgency, this arrangement was formalized into a committee system, as it was during the counterinsurgency in Malaya.

The mechanism of separating insurgents from the rest of the population also depended on the colonial setting. The blurring of combat and policing, normal and exceptional circumstances—endemic to colonial rule—defined imperial attempts at achieving law and order.

The indispensability of this slippage is expressed with particular force in a 1934 book: Major General Sir Charles Gwynn’s Imperial Policing. Gwynn’s title forces us to wonder what the difference is between imperial policing and the policing of, say, London. The answer, we quickly learn, resides in the imprecise yet vital “police duties of the army.”

Gwynn divides these duties into three components. First, small wars, in which policing comes closest to outright combat, and the army’s objective is to lay the groundwork for establishing civil authority. Second, conditions in which the army, rather than the police force, becomes the main agent for the restoration of order. And third, scenarios in which the army is employed as an aid to civil power. Successful imperial conquest pushes the first context aside, but because “the civil control which has been established still rests on insecure foundations,” the second category—the restoration of order—grows in importance. Importantly, Gwynn recognizes that in a colonial regime the establishment and maintenance of order cannot be separated and that the role of the military will continue to be fluid, mixing combat and policing functions. Thus, having delineated his three categories, Gwynn is careful to add a qualifier: “in the course of events an incident may pass from one category to another.”

Gwynn’s particular concern is armed rebellion, which involves indefinite objectives as well as “the admixture of rebels with a neutral or loyal element of the population.” If, at the close of the nineteenth century, Calwell detected a common feature of small wars, which would become an essential condition of insurgency—the refusal of those fighting the state “to meet in the open”—a generation later Gwynn completed the definition by adding a crucial element common to insurgencies: the coexistence in the population of resistance, neutrality, and support. Devising methods to distill this mixture, to separate insurgents from neutrals, would become the abiding preoccupation of imperial policing.

In the same year Gwynn’s book was published, the War Office put out Notes on Imperial Policing 1934, a pamphlet for soldiers. The pamphlet is one of the first official publications that details the procedures for “clearing up a disturbed area,” which include cordons and searches. In subsequent iterations of the Notes, counterinsurgency and ethnic segregation become less and less distinct, so much so that by 1947 a Colonial Office publication on Internal Security Duties describes distinct operations in colonies with “a homogenous population” and those with “groups or factions likely to have recourse to violence.”

The idea of semi-permanent ethnic partition raised no concerns precisely because these regimes were colonial, premised upon the supposed inability of the subject populations, at least in the short term, to establish and manage modern nation-states. For current U.S. counterinsurgency operations, with their avowed aspirations for immediate nation-building, the implications of this history are more disturbing.

Counterinsurgency theorists of the 1960s forgot (or ignored) much of the early history of clearing operations and offered them anew as a straightforward tactical issue. Galula, the patron saint of so much contemporary counterinsurgency theory, follows up his “first law” of counterinsurgency—protect the population—with a clear instruction: “isolate the population as much as possible, by physical means from the guerrillas.” Accordingly, he recommends sweeps of areas; a census followed by registration, identity cards, and passbooks; curfews; and even forcible resettlement.

Similarly, the Field Manual recommends “taking a census as soon as possible” in order to collect the data necessary for security operations; saturation patrolling; and population-control measures that include checkpoints, strict restrictions on movement between neighborhoods, and a pass system complete with elaborate biometric data to enforce compliance.

What the Surge actually instituted was an extensive and possibly permanent system of separation, of checkpoints and biometrics, all supported by walls.

The Accidental Guerrilla details the way these recommendations were put into practice during the Surge in 2007. By that time, Iraq had slipped into a near civil war, particularly in Baghdad. Sunni insurgents backed by al Qaeda terrorists attacked Shia neighborhoods, and Shia militias attacked Sunnis, leading to a series of revenge attacks and counterattacks. The counterinsurgency approach was designed to break this cycle and did so by installing a system of what Kilcullen euphemistically calls “gated communities”—walled-off neighborhoods with controlled access points that allowed U.S. and Iraqi forces to monitor “who was supposed to be in any part of the city and who was not.” The U.S. and Iraqi forces also formed so-called joint security stations, which combined police and combat troops into a hybrid that did succeed in reducing the number of casualties.

Putting aside Kilcullen’s cheery portrait, however, what the Surge actually instituted was an extensive and possibly permanent system of separation, of checkpoints and biometrics, all supported by walls and barriers. Nir Rosen describes a visit to the Dora neighborhood in Baghdad during the Surge:

Looming over the homes are twelve-foot high security walls built by the Americans to separate warring factions and confine people to their own neighborhood. Emptied and destroyed by civil war, walled off by President Bush’s much-heralded ‘surge,’ Dora feels more like a desolate post-apocalyptic maze of concrete tunnels than a living inhabited neighborhood.

Of course, concrete barriers and checkpoints were not entirely new to Baghdad. From 2003 onward, the Americans had utilized them to fortify the green zone. Moreover in 2005 an early experiment in counterinsurgency’s central maxim of clear-hold-build was conducted in Tal Afar, an experiment chronicled in one of the Field Manual’s many instructional vignettes. The process began with the construction of an eight-foot high berm around the city, which in turn funneled all traffic through checkpoints staffed by U.S. and Iraqi forces, who relied on local informants to identify insurgents. “House to house searches” and “the movement of civilians out of contentious areas” completed the clearing phase.

In the context, these measures—immediate and revocable—were designed to arrest a horrific bloodbath. Nevertheless, they may hinder the national political cohesion required for a long-term solution. Certainly the sheer scale of the barriers suggests something more permanent: a political device of partition. If counterinsurgency is marked by the three-fold progression of clear-hold-build, many colonial operations got stuck, as it were, on the clear and hold stages. And there is a danger that the current U.S. clearing efforts will extend into the future, as stopgap measures transform into a permanent state of siege. Whether this danger is real or not, whether these walls have or have not been matched with steps toward real political cohesion, will only become apparent after the withdrawal of U.S. troops.

All of which returns us to the question of legitimacy. Past regimes that engaged in imperial policing were colonial, regimes not of consent, but of coercion and co-option. For both moral and military reasons, the United States should not endorse their model. Contemporary counterinsurgency must go hand-in-hand with an acceptance of the right to national self-determination. This right, in conjunction with a general skepticism about U.S. foreign policy, imposes severe constraints on the sphere of legitimate U.S. action.

Both Kilcullen and the authors of the Field Manual understand these constraints, and therefore emphasize that the lead in matters of combat and reconstruction should be taken by what they refer to as the “Host Nation,” a somewhat odd choice of words, implying that invading forces are guests. Kilcullen advocates “close and genuine partnerships that put the host nation in the lead” as one of his eight “best practices” of counterinsurgency, and the Field Manual includes deferral to the host nation among its paradoxes: “the host nation doing something tolerably is normally better than us doing it well.”

But these recommendations will only work if the occupier is genuinely committed to disengagement. Disengagement, however, is a matter of grand strategy, not tactics. Just what are the United States’s intentions? Will there be real disengagement from Iraq, or will there be 50,000 residual troops and large bases? Will the United States continue to intervene worldwide, on a large scale, or will it shift toward containment? Will the United States maintain a neo-imperial stance, or shift to a smaller global political “footprint”?

On these questions the Field Manual is necessarily silent (it is, after all, a book of military doctrine), and Kilcullen is unnecessarily equivocal. He provides four models to comprehend the global threat environment: a backlash against globalization, a globalized insurgency, a civil war within Islam, and asymmetric warfare. But he does not include neo-imperialist U.S. foreign policy—sponsorships of repressive governments, from Egypt to Saudi Arabia; the intricate global lace of military bases; the effort to establish hegemony through direct and indirect means; and so on. The closest Kilcullen comes to considering the American role in creating this threat environment is reading suspicion of U.S. global policy as “an inherent structural aspect of a unipolar global system.” Since U.S. policy does not drive conflict, the causes of conflict must be “predominantly” outside American control. The conflict thus becomes a “protracted” one, which the Untied States did not cause and which it cannot terminate.

Although Kilcullen disavows large-scale interventions such as that in Iraq and warns about the difficulties of mounting a successful counterinsurgency program, he nonetheless seems committed to the long war. And while he offers counterinsurgency as the preferred method of dealing with small wars, he has little to say about an eventual change in the approach to the long war. Critics such as Andrew Bacevich, whose “realist” appraisals of Iraq and Afghanistan leave him skeptical of the new fashion of counterinsurgency, object to the U.S. military’s approach on just these grounds: “If counterinsurgency is useful chiefly for digging ourselves out of holes we shouldn’t be in, then why not simply avoid the holes?” he asks in The National Interest. “Why play al Qaeda’s game? Why persist in waging the Long War when that war makes no sense?”