hey Nick- here's my comments... and yes, it is worth the effort.
1. keep stock HU and swap out speakers (polk DB6501's components are drop-ins).
2. keep stock speakers and swap HU. get a HU like my pioneer (i have the older gen 3100DVD) with great EQ control. im happy with this cause now not only i have improved sound, i have USB playback, and rear view camera.
3. swap HU and speakers - even better.
4. swap HU and speakers with amps, add sub - best.