Just wanted to update for the benefit of those who contributed and others who may read my post.
The other day while talking to my car electrician he told me that he was 99% sure, in his experience that its the starter motor that probably has to be serviced/repaired. So he told me to lets give it a try before going for the expensive fuel filter replacement as it will also get serviced.
When he removed that starter (which was a pain) as its right under the airfilter box, infact at the bottom. Not much space. The bushes were cut and had to be repaired and also the inner spindle was loose due to that. (Forgive my lack of explanation as I'm not familiar with the technicalities of car mechanics :-p) Anyways, he cleaned up the whole thing and after getting the bushes repaired thru a lathe machine, he fixed it back and now that sputtering problem at startup seems to have vanished so far, 3 days down the line since he did it. The car starts smoothly both in the mornings and evening after a long duration.
I remember on my 1st Gen vitz too he solved a similar problem regarding startup where he diagnosed the armiture was damaged and needed replacement including the bushes in the starter.... which resolved that problem too.