First of all, the fuel pump running while you are on CNG will not do ur fuel pump any harm as the excess fuel will go back to the tank. The pump is designed to run all the time while the engine is running.
Secondly, fuel pump failure can be due to a short circuit too. Could be an electronic issue. I would NOT suggest having a cutoff switch fitted as the more u let the electrician play with ur wiring, the more chances of it going bad later.
Just keep the fuel tank atleast a quarter full and resolve any electrical issue.
Peace!