Your problem
sounds very similar to a common issue many people had with the early 350z's (03 & 04 built cars). Nissan recogised it as a fault and issued a TSB (Technical Service Bulletin) In England they repaired it free, under warranty - like other people have mentioned it was associated with the rear transmission linkage/driveshafts - near to the rear wheels. Some repairers replaced the parts, but others regreased the original parts and turned them 180degrees before replacing them. I had it done on my '03, but it hasn't been an issue on my '06.
The best way to test for it is to wind down both windows, then drive slowly backwards, slowly forwards, slowly backwards, slowly forwards, slowly backwards, slowly forwards, again and again and either listen for it yourself or get a friend to help. You're right to describe the sound as ratchety and with a bit of a clunk.
If you can hear it, it ain't right - don't be fobbed off by a dealer who says it's nothing, unless you want to live with it and have the parts fail early. It doesn't sound like a 'characteristic of the car' to me. Bummer - I thought Nissan had sorted this one out
I'll see if I can find any more information.
UPDATE - If you register with Nissanhelp.com, then click on Models, all models, 350z, 2004, then Bulletins - go to page 3 of the bulletins and you'll find a PDF of the official Nissan TSB (I won't paste it here because of possible copyright issues)
"-REVISED-
2003–2005 NISSAN 350Z; CLICKING NOISE FROM REAR AXLES
A "clicking" noise is coming from the left and/or right rear axle (driveshaft) when accelerating from a stop. "
You're allowed to download it.