We are all in the same boat here, my trick is not to explore all the places in one go.
Like, took the family to SWAT first time in 2015 on public transport, did only White palace, Malam Jabba and one day trip to Kalam. Did Swat again in 2017 on my own car, avoided Malam Jabba altogether, did White palace, Butkara 1 & 2, Swat museum, stayed kalam 2 days with Mahodand lake visit only. Places like Gabral, Shahi bagh are left for next trip. Did Neelam valley first time this year in September, went straight to Jagran, Keran and Upper Neelam only. Pir Chanasi, Shahdra, Kel are left for next visit.