AOA,
bro bellow is the option and reason for your problem:
throttle body service
i know u will think what could be the link of throttle body with battery removal, but believe me bro its gona work. this issue normally happens with camry and its class. because due to continuous intake of dust with air in throttle body a layer of dust in maintained and then the ecu by default maintains the current position of the throttle and saves it while u switch off engine and restart it, due to disconnection of battery the throttle position returns to its default and the RPM fluctuations starts beacuse the dust gives very narrow space for the air to travel
and reset is done with ecu grip removal not with battery because it may left some memory on board.
i hope that will serve your need and solves your problem
regards