Aggressive driving will piss people off. I had a guy rage really hard the other day when I wouldn't let him cut me off to get in the left lane, after he came flying up in the right lane, passing like a dozen cars. He tried to fight me at the red light... I lol'd and rode off. Motorcycles fit between cars a lot better than a Dodge Durango
It's possible that you're getting the finger because people think you're driving aggressively or cutting them off.