There's another possibility - the turbocharger is water-cooled, and if it develops a leak into the exhaust then you have the same result.
The EGR cooler is really easy to bypass and doesn't really need extra hose, just a piece of pipe the same size as the EGR pipe the hose comes off, about 50-100mm long is more than enough. Remove both hoses from the EGR pipe, place over each end of the new piece (make sure the new piece has no burrs or flakes or anything) and clamp in place. EGR cooler bypassed. I recommend blanking the EGR if you're going to do this (at the EGR valve end is ok), the 2004 model won't have flow detection and won't have a clue what you've done. However ...
If there's a leak between EGR and coolant, the coolant will have been entering the intake manifold, and it means that disconnecting the coolant hoses will allow the exhaust to escape into the engine bay. Something tells me this may not be it though, because this should pressurise the coolant system - the EGR in that pipe is under considerable pressure, particularly under load.
Checking the turbocharger is not so easy, because there's a lot of exhaust pipe in the way. A good mechanic may have exhaust gas analysis equipment and may be able to tell if coolant is in the exhaust - it shouldn't cost a lot to have them hook it up and give it a few revs to see what's there. Coolant should show up because it's got a specific chemical formula and fluorescent additives.
You could manually inspect the turbo - remove the intake pipe from it (cold engine!) and give the impeller a wiggle and a twirl. The shaft should NOT wobble. I've got a near-new turbocharger sitting in a box here and it's shaft doesn't budge sideways, rotation is smooth with no roughness whatsoever. If you experience either, the journal bearings are shot. It might just be that the seals are gone - the ones isolating the coolant - it's obviously not leaking back into the sump (there's a pipe that uses gravity to return oil back to the sump, it's immediately below the turbocharger, pipe is about 20mm diameter). It could be a cracked turbine housing (the turbine is the part of the turbocharger that's driven by the exhaust and is in the "brown" side not the silver side of the turbo). The housing might try to push (the relatively high pressure) exhaust into the coolant system through the crack but liquid is much denser than air so it might struggle to do that, and coolant might easily flow into the exhaust if your engine is revving but you're not on the gas (engine braking). If you notice the smoke doesn't appear if you do NOT engine brake, chances are this is the issue.