When Gordon Brown starts using the language of the British National Party, then you know something strange is happening in politics. The Prime Minister's talk of "British jobs for British workers" in his Labour conference speech, and his promise to crack down on criminal immigrants suggests that Labour has turned a corner on race. It is no longer taboo on the liberal left to talk about restricting the number of foreigners entering Britain.

The mood has been changing for the past year or so. The former Labour deputy leadership contender, John Cruddas, called in his campaign for action to stop migrant workers bidding down wage rates and conditions in English labour markets. Earlier this year, the Labour minister, Margaret Hodge, called for discrimination in favour of indigenous Britons in the allocation of council housing. That former left-wingers now talk this language suggests there is a profound shift of attitudes taking place. Immigration is no longer the issue that dare not speak its name.

In a week which promises to hold little cheer for David Cameron, as he is slaughtered in the opinion polls, one consolation might be that after 10 years in office, a Labour leader is left exploiting essentially Tory themes to kick off what looks like an early election campaign. Brown's speech borrowed liberally from the 2005 Tory manifesto, not just on race, but on dirty hospitals, crime and strong government.

It surely says something about the Labour years that Brown now has to invoke the spirit of Margaret Thatcher to appeal to Middle England. It is curious that the Conservatives, so lacking in self-belief, seem unable to recognise that this is their moment; that Gordon Brown has paid a remarkable tribute to their enduring relevance as a political force in the land.

Now, I am talking here about England because, of course, Scotland, as we all know, is different.

The Scottish Conservatives are up against a very different political culture. There has not been the same antagonism towards incomers, even though Scotland has had its share of EU migrants, as the proliferation of Polish delis in Scottish high streets testifies. Scotland never experienced the waves of Afro-Caribbean immigration that fuelled the rise of the British National Party. The Scottish National Party is colourblind, open to all and wants immigration to increase.

Nor have Scots ever been much enamoured of Margaret Thatcher. Many will have cringed inwardly when they saw Gordon Brown take her by the hand at the gates of No 10. The memory of the great industrial closures of the 1980s and, of course, the poll tax are too fresh here, a part of political folk memory. Tory is no longer a four-letter word in Scotland, but that doesn't mean Scottish voters are ready to make nice with Her.

This is one reason why the Scottish Labour MPs are so resistant to the attractions of an early election. They suspect Brown's "Torier-than-thou" speech will not have inspired Scottish voters to turn back to Labour, and they fear - correctly - that the SNP may be able to exploit a Labour election campaign which is based on "Britishness", immigration and tax cuts. (I expect Brown to match the cuts in inheritance tax and stamp duty announced by Cameron this week, but he will wait until an election campaign to do it.) So, the Tories - in England at least - still have a strong hand to play, provided they don't disintegrate into warring factions. Cameron's strategy was the right one: to change the image of the Tories as the "nasty party" by turning greenish and suggesting youth criminals need love as well as punishment, and to keep in reserve the Tory base issues of immigration, law and order, and family until nearer the election proper.

But Brown upset their schedule by ramping up speculation about an early election, and then pinching their clothes. This caused the Tories to panic, and desperately scrabble back on to their home turf, and away from what the Shadow Chancellor, George Osborne, called "uber- modernisation". They've been rattled and they show it.

But they weren't wrong. They hold a number of winning cards, had they a leader capable of playing them. Indeed, the Tories could do much to win the next election on immigration alone. Brown's flirtation with neo-Powellism is, if you think about it, as ridiculous as it is hypocritical. As Chancellor, he positively encouraged mass immigration in order to keep wages down and profits up.

The Tories may discover that, in England at least, there is now a much wider constituency for controlling immigration than at any time in the past. Immigration is a dangerous card to play, of course, but Brown's decision to join the "nasty party" certainly lowers the risk. The Tories could surf the change in attitudes on race that has occurred throughout English society in the past three years.

This transformation has occurred for a number of reasons. Post-7/7, there has been a reappraisal of the policy of encouraging cultural diversity in British towns and cities. Some believe this is a race-relations version of separate development that has led to islands of militant Islamism in English cities. The emphasis now is on integration, Britishness and learning English. Brown has ruled that in future only those immigrants who can speak the language properly will be allowed to seek work here.

Another reason has been the change in the racial character of immigration. For most of the past 50 years, immigration meant Asian or Afro-Caribbean incomers, who are defined by the colour of their skins. Since 2004, the focus has changed to white migration from the EU accession states, typically from Poland, which alone has reportedly sent around 600,000 plumbers and allied trades to the UK. The tide has been so great that in the forthcoming Polish elections, candidates are coming on campaign tours to Britain to canvass votes.

Questioning immigration need no longer appear to be racial intolerance, but be more about social administration. The government wilfully underestimated the numbers of immigrants who would come from the eastern European countries and, as a consequence, social services departments are overwhelmed. The housing shortage has not been caused by immigration, but it has certainly been aggravated by it. The high cost of rents in the south of England is in part a result of the practice of housing migrant workers four to a room in buy-to-let blocks owned by absentee landlords. They provide an identifiable enemy for young British couples who can't afford a house.

David Cameron has the basis here of a potentially election-winning agenda, if he is bold enough to build on it. Gordon Brown has cleansed immigration as an election issue, just as he has endorsed Conservative family morality, Conservative economics, Conservative nationalism and Thatcherism. The Tories should be saying: why accept second best, when you can have the real thing?