The outside air selection pulls air from outside and reaches you after passing through evaporator.
If you switch off compressor, select out side air, it depends on the ambient temperature how cool the entering air will be. But while doing this( only using fan), the selection of cooling knob should be at max coooling( even if compressor is switched off.
Note that the outside air enters through the grills adjacent to the windshield bottom area. Hence, due to engine heat desipation in this area, the outside air gains some temperature before it reaches you.
My suggestion to you is as follows:
1- Replace refrigerant accoumulator/ filter
2- Check for leakages
3- Maintain proper refrigerant.
All above from a good ac repair workshop.