Then you better lock your sim too and even encrypt the device. Also put sim change alert.
If you store contacts in sim, anyone can take out your sim from your phone and steal your contacts in a minute. its that easy. But if you lock sim, it will not read/use the sim unless you unlock the sim.
However, even phones itself are vulnerable to breaches, but still its a very good move to make reasonable efforts to protect it.