If you're not running your air conditioner then the only temperature-affecting thing inside the ductwork is the heater matrix. If that's not being offset by the refrigerant, then the cabin can only heat up.
I had thought I'd read somewhere that the heater matrix was supposed to be fully isolated from the rest of the ducting when "cold" was selected, but mine tends to get a little warm too and I'm not sure if it's age, or just the proximity of near-boiling water that lets a little heat leak in, and over time that adds up to a warm cabin!