Driving the car with unbalanced tyres isnt recommended as you feel the vibrations only at a certain threshold but the tyres are sending vibrations to all car body parts continously and may result in rattling; creaking sounds from different body parts due to loosening of clamps etc..
Additionally it will also damage the tyres due to uneven tread wear off.