With Hari Seldon-like ambition, biologist Peter Turchin, in his book War and Peace and War, offers a theory for the rise and fall of empires in agrarian societies. He points out that large scale political power often emerges far out on the "meta-ethnic frontier." Having a common enemy unifies people.
Here's Turchin's version of his basic theory:
Ethnicity is the group use of any aspect of culture in order to create internal cohesion and differentiation from other groups. There is an imaginary boundary separating the members of the ethnic group from the rest of humanity. For example, Greeks drew a boundary between themselves and barbarians, non-Greek speakers. The ethnic boundary can use a variety of symbolic markers—language and dialect, religion and ritualistic behaviors, race, clothing, behavioral mannerisms, hair styles, ornaments, and tattoos. The important thing is not which markers are used, but the distinction between in-group and outgroup members, between Us and Them.
People usually have multiple ethnic identities nested within each other. An inhabitant of Dallas can be simultaneously a Texan, an American, and a participant in the Western Civilization. The broadest groupings of people that unite many nations are usually called civilizations, but I prefer to call such entities metaethnic communities (from the Greek meta—beyond and ethnos— ethnic group, nation). My definition includes not only usual civilizations—the Western, Islamic, or Sinic—but also such broad cultural groupings as the Celts or Turco-Mongolian steppe nomads. Typically, cultural difference is greatest between people belonging to different metaethnic communities; sometimes this gap is so extreme that people deny the very humanity of those who are on the other side of the metaethnic fault line.
Historical dynamics can be understood as a result of competition and conflict between groups, some of which dominate others. Domination, however, is made possible only because groups are integrated at the micro level by cooperation among their members. Within-group cooperation is the basis of intergroup conflict, including its extreme versions such as war and even genocide. Different groups are characterized by different degrees of cooperation among its members, and therefore different degree of cohesiveness or solidarity. Following the fourteenth century Arab thinker Ibn Khaldun, I call this property of groups asabiya. Asabiya is the capacity of a social group for concerted collective action. Asabiya is a dynamic quantity; it can increase or decrease with time.
Each empire has at its core an imperial nation (some empires had more than one imperial nation for a time, but this structure appears to be unstable). The ability of an empire to expand territory and to defend itself against external and internal enemies is determined largely by the characteristics of its imperial nation, especially its asabiya. Because only groups possessing high levels of asabiya can construct large empires, the question is how do they gain it, and why do they eventually lose it?
Groups with high asabiya arise on metaethnic frontiers. A metaethnic frontier is an area where an imperial boundary coincides with a fault line between two metaethnic communities. Metaethnic frontiers are places where between-group competition is very intense. Expansionist empires exert enormous military pressure on the peoples beyond their boundaries. But the frontier populations are also attracted to the imperial wealth, which they attempt to obtain by trading or raiding. Both the external threat and the prospect of gain are powerful integrative forces that nurture asabiya. In the pressure cooker of a metaethnic frontier poorly integrated groups crumble and disappear, while groups based on strong cooperation thrive and expand. In order to match the power of the old empire, a frontier group with high asabiya—an incipient imperial nation—needs to expand by incorporating other groups.
On a metaethnic frontier integration of ethnically similar groups on the same side of the fault line is made easier by the presence of a very different Other—the metaethnic community on the other side. The huge cultural gap across the frontier dwarfs the relatively minor differences between ethnic groups on the same side.
Is it true? Hard to say. The most famous empire of them all, the Roman, doesn't seem to fit all that well, but the Macedonian Empire of Alexander and the far western Ch'in (or Qin) state that ended the Warring States period and unified China under the first emperor do fit.
In his technical book, Historical Dynamics, Turchin attempts to quantify his theory. I haven't read either one, so I can't judge how well he has succeeded. His theoretical model seems appropriately relativistic, but that can make data hard to nail down quantitatively, since the theorist can be tempted to redefine history to fit his theory. The fairest test of a theory like this is if it works for a database assembled by somebody else for some other purpose.
Turchin is not the first to notice this tendency of empires to arise on the margins. This is sometimes known as the "marcher lord" theory. A "march" is a margin, a border land. The term "marcher lord" is the English version of what the Germans call a "margrave," and marcher lords are the tough guys who who rule out on the frontier, like the Tudors on the Welsh border in the 15th Century, keeping the outsiders out and often bullying their way to take over central power themselves.
Greg Cochran points out that the Marcher Lord dynamic worked on a huge scale to eventuate in the superpower confrontation of 1945-1991, with the U.S. and Russia as the marcher lords of Europe. That's because another advantage of being out at the fringe of an advanced culture's geographical expanse is that your enemies, being from technologically more backward areas, are weaker, so you can easily expand territorially at their expense. If you are, say, France, you are hemmed in by similarly tough countries like England, Germany, and Spain. But if you are the United States or Russia, you can conquer vast amounts of land owned by Indians or Siberians.
As an example of a country notoriously lacking in asabiya, Italians had a hard time getting themselves organized into a large powerful state over the last 1000 years because the ethnic variations within Italy were so minor and so "clinal" as they varied from south to north that people tended to organize around family and undermine larger polities through backstabbing each other. Lacking a frontier with a common enemy, they lacked incentives to unite, much to the dismay of Italian patriots like Machiavelli. (Italy was ultimately unified under the House of Savoy from the French border.)
Germany had somewhat similar problems getting organized, except out in Prussia in the far northeast, where Germans fought Slavs and Balts, and in Austria in the southeast, where Germans fought Magyars and Slavs. There, the Germans learned to subordinate their differences and cooperate with their fellow Germans to whomp their non-German neighbors. Eventually, the Prussians took over all of Germany, with mostly disastrous consequences for Germans and the whole world.
The U.S. has the decided advantage of being located with oceans to the east and west, so since the defeat of the Indians, we haven't had much of a march. Yet, in recent decades we've developed a cohesively unified group of influential intellectuals and apparatchiks who devote much of their energies to trying to persuade the rest of America, with much success, that Israel is America's march with the whole nefarious Arab or Muslim or whatever world. As Francis Fukuymama pointed out after Charles Krauthammer tried to smear him as an anti-Semite, the neocon worldview is one that has largely tried to apply lessons appropriate to tiny, frontier Israel to huge, continental America.
Of course, while a handful of neocons actually have been in the Israeli military, most of them are purely pretend marcher lords, sofa samurai. But, they possess much of the cohesion characteristic of the breed. As Mearsheimer and Walt have discovered, they all work together to ruin their enemies.