Forty years ago, when Enoch Powell delivered his "rivers of blood" speech, I was a lecturer in Birmingham. Of course, racism was widespread at the time - but Powell put his finger on a button that unleashed fear and fury.

At one meeting, I tried to debate with him. I can still see his bulging eyes and sweating face as he tore me to bits.

Handsworth was the district with the highest number of what were then called "residents born in New Commonwealth countries" - in other words, black people. I helped as a volunteer and met mothers who said that the lack of daycare stopped them getting jobs - and then they were blamed for living off the state. We acted together to found a daycare centre which, all these years later, still exists.

Have things got better? In some ways. The churches are now more likely to criticise racism than reinforce it. More black people are in managerial, professional and political posts, although not enough. Ours is still a society in which it is an advantage to be born white.

Racism always finds some outlet. As a boy, I recall the baiting of Jewish people in east London. Years later, Sir Keith Joseph - in some ways a humane man - asked me if the Irish really were "breeding like rabbits". Then arrivals from the West Indies became the focus of attack. Today the targets are economic migrants from eastern Europe and asylum seekers fleeing from persecution.

Many politicians are supporters of racial equality. Yet in May 2007, Labour Party minister Margaret Hodge proposed that local, British-born families should get social housing over new immigrants. The Conservative immigration spokesman Damian Green agreed and added that his party was calling for an explicit annual limit on the number of immigrants.

Both Hodge and Green failed to point out that the shortage of public accommodation stemmed from Margaret Thatcher's policy of selling off council housing.

Last week in Glasgow, I visited the excellent Positive Action in Housing, a voluntary agency that concentrates on supporting immigrants and asylum seekers. Last year, 123 of its users, 18% of the total, reported racial harassment, including repeated verbal abuse, physical attacks, damage to property and an arson attack. Victims often felt they could not go out at night, kept their children from using public play areas and sometimes asked to be re-housed. Positive Action has responded by working with immigrants, asylum seekers and agencies to create the confidence and knowledge to enable victims to report incidents to the police. One result is that the Glasgow Housing Association has improved the training of concierges and frontline staff.

As a boy in London, a young man in Birmingham and an old man in Glasgow, I realise that immigrants from different cultures and of different appearance can provoke hostility. Yet they can also bring about unity. In the church I attend, an African family became part of the body. One Sunday, they announced they had been granted asylum. We wept and cheered.

In Cardonald, a family from a war-divided state were suddenly removed by immigration officials. But their children had become so integrated into the school and community that it was residents who campaigned for their return.

Racism must be countered and unity promoted at the grass roots. Politicians should be there.