That's very true. Great observation, as most people don't notice these things. They just know that TouchWiz is bad!. Even Note series have a ton of useful features, which half people are not aware of.
Once in a while, one or two corrupt apps or even some corrupt Samsung app makes the phone battery hungry or laggy and people who cannot figure out software issues tag these issues with phone. The latest ones are actually very stable, fast and efficient.
Even Stock Android in most Nexus phones, having none of the features like TouchWiz is not as battery friendly as it should be. But Nexus phones corrupt less.
Those CM and AOSP roms were way more better than Samsung roms back then, but now most of them for flagships are not too impressive.