I am assuming you don't have a racing transmission. Probably not a great idea to downshift into first that often. I never downshift to first unless I'm moving <5mph... Anything faster than that and you're likely to encounter difficultly getting the shift knob in gear, and doing it a lot could potentially damage something
Difficulty when completely stopped is different. Might want to have it looked at...