I agree, you probably need to slow it down, take your time, practice, and build up speed gradually.
Frankly, riffs with fast downstrokes can be one of the most frustrating techniques because to a degree either your right hand moves fast enough or it doesn't, and it can take a long time to build up the required speed and accuracy.
Things like the riff at the start of this are damn, damn, damn hard even though they might sound relatively simple: