When the shortlist for the Turner Prize was announced in May, Cathy Wilkes drew a lot of flak. Most commentators, and not just those at the red tops and middle-market tabloids taking their annual pop at modern art, focused on a single element of the installation at Milton Keynes Gallery which earned Wilkes her nomination, turning up their noses at the fact that her work featured a shop mannequin sitting on a toilet, as if this one, apparently tawdry, image should stand for the artist's practice as a whole.

Part of this refusal to look beyond a sole, headline-friendly part of Wilkes' work can be put down to the London-centric approach of the press. Wilkes has represented Scotland at the Venice Biennale, shown work at the prestigious roving biennial, Manifesta, and regularly mounts exhibitions at major galleries in Europe. But compared to artists of similar international standing, she is rarely seen in London.

The hoopla surrounding the Turner might also be to blame. In recent years, regardless of the artists nominated, the same story plays out. The moment the nominees are made known, dead cert is picked by critics and bookies alike (Mark Leckey this year), an outsider is identified as a possible contender (think Tomma Abts, or Grayson Perry), and one or more of the remaining prospects is branded a bit of an eccentric, as Wilkes has been, or offered up as a sacrificial lamb for those who like to have a wry chuckle at the supposed pretensions of contemporary artists.

None of these reasons for the reaction to Wilkes' Turner Prize nomination have much to do with the artist or her work, but looking at her latest installation at the Modern Institute - Prices - it is easy to see how observers might be tempted to latch on to that mannequin on the loo. This is because Wilkes work is, for want of a better word, difficult. Her installations, or tableaux, are made up of arranged or altered found objects matched with sculptures, paintings and, sometimes, texts that, taken together, hint at themes and concerns that are never made explicit.

They shrug off any attempt by the viewer to decide what a work is about, offering up then confounding easy interpretation. Even the broad themes that can be identified in Wilkes' work - the autobiographical sources, femininity, feminism and domestic politics - are decidedly ambiguous.

Prices is no different. Tightly assembled at the far end of the institute's main gallery, the piece revolves around a supermarket checkout, complete with till. On top of the reclaimed unit, there are glass and plastic bowls, each containing the dried-out residue of what might once have been soup, cups of tea long drunk, and a scattering of spilt sugar crystals. On the floor beside the checkout are more dirty bowls, and a fish tank packed with more found objects and sculptural assemblies. There's a squeezy bottle of honey in there, a battery, grains of sand in a jam jar, and a grubby decorative jug.

Looming over all this is a mannequin, its left hand bearing traces of the food that fills the nearby bowls while, almost standing apart from the body of the installation but recognisably a part of it, are three more obviously sculptural works.

The first of these is a flat board covered in a yellow material that calls to mind Marigold gloves, its surface inscribed with a heart shape, which is marked out by tiny whelk shells, more of which have been scattered around the floor. A pair of squat towers finish the piece, each made of terracotta tiles and with a cross scraped into, or painted on, their sides.

And so the difficult business of untangling Wilkes work begins. These objects are bound together, thanks to Wilkes' unerring knack for arranging discrete elements into a sculptural whole. Sometimes these connections are self-evident but more often there's a slippery connection that only reveals itself after a long look.

There is, for example, a sort of ley line of molluscs that links the fish tank vitrine to the mannequin, and the bowls on the floor match those on the checkout, as if their placement is governed by some invented mathematical rule, like the Fibonacci sequence that governs the growth of the shells beside them. The tile stacks occupy the corners of an unfinished oblong, with one reflected in a mirror on the side of the shop unit, suggesting a second, impossible installation through the looking glass.

When it comes to decoding the meaning in Prices, Wilkes again provides obvious clues, only to undermine them. There is an air of domestic drudgery, with the allusions to supermarket queues and the mealtime frustrations of a young child, allied to the objectification of women implied by that mannequin. It might have been possible to reconcile this with the religious monuments in miniature and the scattered whelks, to identify a feminist critique of a patriarchal society - but there is nothing so strident, or coherent, here - just a set of oblique allusions.

Muddying the waters further is Wilkes' tendency to return to the same artefacts, reworking them with each new installation. The Prices mannequin has a few strands of hair pasted to its scalp, a reminder that, in the past, Wilkes's shop dummies have worn glossy wigs. The bottle of honey echoes her past use of jars of jam, while the printed card that advertises this show bears an image of the yellow board, but with the heart shape marked out in flowers, not shells. And it seems safe to say that the towers of tiles, or the bowls and spoons, will show up, altered and renewed, when Wilkes mounts her Turner Prize show, continuing the long, slow-shifting development of her private language, with its vocabulary of objects and grammar of arrangement.

That language is, in the end, what makes Wilkes so engrossing. There is a sense that there is a key to translating or decoding the arrangement - placed just so for purposes known only to Wilkes, and even then, perhaps, only in the moment of arrangement - but one that will be forever out of reach. The result is work that, almost uniquely, satisfies and frustrates in equal measure.