I agree with you on half things you said, but brand matters a lot!
Antutu benchmark is just a RAW performance benchmark, it has nothing much to do with real life performance. A phone may have very powerful chipset scoring well in benchmarks, but still be a mess, like the great Xperia Z5!
Look at Snapdragon 810 found in Z5, HTC M9, Nexus 6P, LG V10, OnePlus 2 etc. They are all same chipsets, but the real life experience varies a lot. Z5 is an awful mess even with S810.
Chipset is just like a motherboard. There are many other components and factors attached to it that make a huge impact like, RAM, camera sensor, display, heat-sinks, quality & compatibility of components, R&D, standard of quality controls, software optimisations, manufacturing techniques etc etc. So same chipsets in different brands are NEVER the same, even if they score equally.
Better brands put more "quality"