yah it makes a huge difference. you are using a usb port and not a usb-c. so you only get the power the usb port can put out which is not anywhere close to fast charge power. usually around 30w or so is fast charge levels and i doubt you'll get that out of a mobo usb-c port either. but i could be wrong there as some higher end boards may support it.
do yourself a favor and use the charger that came with the phone to get the nice fast charge
