Software is indeed most important as it controls the hardware, though most of us falsely believe that hardware (specifications) is everything. You can see how these software mods can pretty much improve everything. Like performance, battery timings, even camera quality etc. Latest vs outdated TouchWiz is the best example.
Image processing works as, camera takes RAW (unprocessed) photos and then it automatically processes (software optimize as per OEM settings) the photos to create a good photo. So image degradation occurs in most custom roms IMO, but it happens a little more in Samsung as their software is highly optimised especially in their latest phones.
Try to get RAW (DNG) photos on custom roms and then manually process them as per your liking. Its easy. In this way, you can get even better quality than stock rom. Do some Camera API2 mod if possible to unlock manual controls.
Don't fall for outdated stuff. Get Nexus 5X used, latest model and will be less used and abused. Its worth increasing budget slightly, great camera and exceptional developer support in one package (impossible to find in low price).
