Let me just say that my KLR650 with 37 horsepower is just about as fast as my GSA1250 with 136 horsepower, but only off-road. A good suspension matters the most, followed by traction from the tires. The overall weight of the bike also matters, and more importantly how that weight is distributed, a low CG and within the wheelbase being better. My friends riding with me on the same trails have bikes ranging from Yamaha WR250Rs at the low end (29 bhp) to KTM 1290s at the top end (175 bhp) with a variety in between so it is easy to compare notes as we ride. Engines making torque down low are better, as well as proper gearing, specially first. Ratios requiring excessive clutch slipping are not good.
(I have done the same trails on both, at speeds of up to 90 mph, so this is a direct comparison over thousands upon thousands of miles. I have also done the BMW Performance Off-road Training Course which is a pretty tough two days of riding over very challenging terrain in dry and wet weather. I have also worn out several sets of new tires in 1500 miles or so, both on- and off-road while enjoying myself greatly. The above is just to say that I have some idea of what I saying here, not boasting about anything.)
Lastly, the ride is always in the rider and never the bike. Trust me on that! 
