I think what Prometheus was getting at was that there are other factors at play than just heat absorption by radiation. If the surrounding air (or the air boundary layer around the clothes) has a lower temperature than the clothing, the heat will be convected away at a rate proportional to the temperature difference.
However, the heat also convects and conducts into the body underneath the clothes, and likely at a greater rate in a hot climate since heat transfer coefficents into the body are naturally higher with direct skin contact and moist air compared the drier air outside as well as the (often) higher then body temperature external conditions. Since the internal areas of the clothes are roughly the same internally and externally, the rate of heat transfer (flux and overall) is higher going INTO the body.
HOWEVER this does not take into account that the air pocket beneath the clothing may become heated to above ambient temperature, a limit that may already be achieved with the energy absorbed by light clothing. Hence, if wearing black clothing, excess absorbed energy would then transfer to the environment as opposed to into the inner air space.
HOWEVER this applies to clothes that have large airspaces between it and the body. If wearing tighter clothing, say a jersey, larger amounts of heat can transfer to the skin without the insulation of air to protect the wearer. This heat may overheat the body directly OR prevent inner heat generated by the body from escaping due to a lower temperature gradient.
I'm talking myself in circles; theres a lot to consider and I just scraped by in thermodynamics classes. The point is, there may not be a single answer for all conditions.