Brakes work in any case in Vitz 1st Gen, they are just not boosted by Brake booster. And that also happens when your car and its pipes or vacuum gets weak. This happens only in stop and go rush; when AC is on, the pedal gets hard. This doesn't happen always as car relearns the new settings and increases the RPM to compensate. And there is always reserve vacuum in car so while its running there is no issue.
Ever applied brakes after the car has been turned off?, the first 2 or 3 pedals works as intended.
Source: Personal Experience