Day 1:
- Lahore to Mingora
- Hotel checkin, get fresh and visit White Palace at Marghzar.
- Back to hotel, night stay.
Day 2:
- Checkout, visit Malam Jabba
- Descend back from Malam Jabba and go to Kalam directly.
- Night stay at Kalam
Day 3:
Visit Kalam forest, Matiltan or jahan tk road allows.
Visit Bayyun if possible, or spend quite some time in Kalam Forest. Its beautiful thing to do in Kalam.
Night stay at Kalam.
Day 4.
- Checkout, Back to Lahore from Kalam.
You've 1 day extra which you can utilize as per plan to see some other places but in my opinion with family you can't roam each and every place it will be hectic for you and your family as well.