TOM DEVINE

Why has the Union between Scotland and England survived for more than 300 years? Why is its future the subject of heated debate as never before? There is no more burning issue in Scottish public life than this country's constitutional future, as is confirmed by the high-stakes political controversy on an independence referendum raging at present. There could not be a more opportune time for the appearance of Scotland and the Union, 1707 to 2007, which is published by Edinburgh University Press on Wednesday.

The book is the first ever sustained study of the entire 300 years of the Anglo-Scottish Union, from its origins to the historic victory of the Scottish National Party in May last year. A distinguished group of historians and political scientists has been assembled to analyse the central aspects of the relationship.

When it was first established, the survival of that relationship was far from inevitable. Before the 1740s, the omens were not good. The Treaty of Union was carried in the Scottish Parliament of 1706 by a majority within a tiny patrician elite, resulting in a marriage of convenience passed in the teeth of both internal political hostility and external popular hostility. Continuing anti-Union disaffection was an important factor fuelling the Jacobite cause and influencing the major risings of the first half of the 18th century.

However, by the 1760s and certainly by 1800 the future of the Union seemed secure. Of critical importance were the spoils of empire greedily exploited by the non-inheriting male offspring of the Scottish landed elites.

In parallel, there emerged a consensus among the nation's intellectual leaders that progress and unionism were closely associated. Almost to a man, the literati of the Scottish Enlightenment were wedded to the idea that the Union was the prime source of liberation from what they saw as Scotland's dark past of religious obscurantism and feudal inertia.

However, there were two partners in the Union and the 1745 rebellion had fuelled English paranoia that the Scots were an inherently disloyal people. Even Culloden and its brutal aftermath did not entirely eliminate English suspicions that crypto-Jacobitism was a peculiarly Scottish disease.

Scotophobia reared its ugly head in the early 1760s, during the office of John, Earl of Bute, the first Scottish-born Prime Minister after the Union. His tenure was brief but his influence endured. So, too, did the relentless attacks on him and Scots in general. The number of Scots holding state office rose dramatically and it was easy to suspect that Bute favoured his own kind. With the exception of the French, no nationality other than the Scots was so derided in the caricatures of the London press. These cartoons were savagely racist in tone, portraying Scots as greedy mendicants growing rich on England's rich pastures.

A turning point in these strained relations came during the French Revolutionary and Napoleonic Wars. Between 1776 and 1783 the Scots were loyal to the British crown. Britain was victorious in the Napoleonic Wars and the foundation of Pax Brittanica across the world was established.

But at the time it was a close-run thing: from 1798 to 1805, Napoleon's armies were encamped a few miles across the Channel. Scots were already over-represented among the officer class in the field armies when 52,000 more joined the ranks of the volunteers at the hour of England's greatest peril, accounting for 36% of all the volunteer soldiery. The Scottish contribution in blood to victory had cemented the Union by 1815.

With security on the northern border established and underpinned by the loyalty of the Scots, Westminster could virtually afford to let Scotland go its own way within the Union. The English "elephant" reverted to a posture of benign neglect.

Parliament in London rarely intervened and the Lord Advocate in Edinburgh continued to control such key areas as law enforcement and policing. A new and powerful local state, run by the Scottish middle classes and reflecting their values, was created. It, rather than a usually indifferent Westminster authority, routinely governed Scotland. The middle classes had no reason to seek parliamentary independence or adopt a nationalism hostile to the British state.

On the eve of the Great War, the Anglo-Scottish Union must have seemed a rock of stability in an uncertain world. The Scots assumed their global economic eminence was rooted in the Union. However, between 1914 and the 1950s, this almost smug relationship was assailed to an extent unknown since the 18th century.

The First World War was a human catastrophe on an unprecedented scale for Scotland, with more than 100,000 killed. This disaster was followed by the collapse of the markets for Scottish heavy industry in the late 1920s and high levels of emigration, which, for the first time since census records began, caused a fall in population. Edwin Muir captured the crisis of national confidence in his Scottish Journey of 1935: "Scotland is gradually being emptied of its population, its spirit, its wealth, industry, art, intellect and innate character."

But the Union remained impregnable. The Conservative and Unionist party in Scotland was hugely popular between the wars, winning five of the seven general elections. During the long drawn-out economic crisis of these years, Scottish voters preferred the secure umbrella of the British state to any nationalist adventure. The foundation of the SNP in 1934 showed that not all Scots were in the Unionist camp, but it suffered successive failures at the polls.

The outbreak of the Second World War further strengthened British identity. For a time, plucky Britain and its Empire stood alone against an evil foe. Every nook and cranny of life was affected as the nation geared up for total war. The legacy of Britain united in a good cause endured in the folk memory of the post-1945 generation.

This was not the only factor buttressing Britishness. The foundation of the welfare state, promising cradle to grave security and the commitment to full employment in the post-war world, had enormous appeal for Scots who had suffered the full impact of market failure in the 1930s, as evidenced by serious unemployment levels and appalling housing conditions.

The impact of the Union in the 19th century was probably broadly neutral. Only from the 1950s, with welfarism and nationalisation of industry, did it once again have a marked effect on Scotland. A new bond had been formed. As living standards began to improve and the years of austerity faded, unionism in Scotland seemed unchallenged.

Indeed, in 1950 Labour dropped its long-standing manifesto commitment to Scottish home rule and the SNP continued to stagnate; 1955 saw the Conservatives achieve just over half of the popular vote, a unique achievement in Scottish electoral history.

But this political consensus did not mean that Scottishness had evaporated. On the contrary, the mass interest in the Scottish Covenant of 1949, advocating a parliament in Edinburgh within the Union and attracting nearly two million signatures, suggested Scotland's sense of itself remained robust.

Moreover, by the later 1950s, all was not well with the Scottish economy. The long period of Britain's post-war relative decline against international competitors had begun. The balance between "Scottishness" and "Britishness" shifted. The rise of the SNP, the new and pragmatic interest in devolution by Westminster and a fresh vitality in Scottish culture were all signs of the times.

A key decade was the 1980s, when the English "elephant'" for the first time since the mid-18th century, moved to the Scottish side of the "bed" with the imposition of hugely unpopular social and economic policies in the face of overwhelming opposition north of the border.

The Scots had not voted for Tory radicalism and many felt they were suffering under an electoral dictatorship. That experience put more steel into the Scottish electorate and their politicians. Any ambiguity about the relevance of a Scottish Parliament quickly receded.

More than half a century on from the high noon of unionism in the 1950s, the issue now is whether the connection between Scotland and England will survive for much longer. Scottish identity has become stronger and more confident while Britishness is on the wane because of the declining influence of Protestantism, Britain's fall to the status of a second-rate power, the increasing importance of Europe and the parallel decline in the authority of the British state.

Since the collapse of the Soviet threat there is a loss of a clear "other" to help sustain British national solidarity against a common foe. Whether these influences and others will fracture the Union remains an open question.

For many nationalists, devolution is a major step on the road to independence, a hope confirmed by last year's historic SNP victory in the Holyrood elections. For unionists, devolution may be one further manifestation of the resilience of the Union and its capacity to endure despite changing historical circumstances.