when we press the o/d button it is usually written on speedometer as "o/d OFF" it means that on pressing the button O/D gets off and by default it is "on".
Now what is O/D? O/D is usually the highest gear of the transmission as in case of Vitz it is 4th gear(may be 5th, not sure). So when we turn o/d OFF it means we r not allowing the car to reach the highest gear and when it is "ON" then it means that the transmission will try to reach o/d i-e highest gear and in this attempt it will shift up even with the slighest increase in rpm.
Most professionals say that it should be turned Off while we r driving in city i-e below 80km/h.Since turning the o/d "ON" in city will cause damage to ur transmission as you have noticed that the car often keep on changing gear when o/d is "ON".Simple is that it is unable to decide which gear to use.
It should be turned ON when driving on Highway, motorways etc.