66 Degree Outside Air Becomes 80 Degree Inside Heat!

Fountainhead

I think OP was noticing the effect of no refrigerant flowing to remove heat from the coils, the coils if no refrigerant is flowing are at the same temp as ambient engine air, so when you turn on the vent it's still flowing across the AC evaporator coils which are heated by ambient air around engine, heat from road, many heat sources radiating into those pipes. Beside, the coolant doesn't circulate when the A/C compressor isn't pumping.
I think modern cars don't have a complete bypass for outside air only.