How much performance is lost when you switch to 20" rims?

Csquared

Gear Ratio doenst' change unless you change the rolling diameter. With larger wheels you should actually be lowering your side walls.

Weight aside, you're be paying more for your tires, your tires won't last as long. You'll curb your rims more easily and if you hit a pot hole the chance of you bending your rim is much greater cause there is a lot less rubber and air to cushion it. And as mentioned above your ride is gonna be a bit harsher.