The fact that it only happens with certain women decreases the probability of a pathphysiological basis for the premature ejaculation, so it could be more psychological. There's a number of psychological theories about why it happens but the mechanism hasn't really been discovered and there is really not much evidence for any of them. With the information you gave I immediately thought of performance anxiety as it only happens with women who you are keen on (i.e. you would want to impress them I imagine). There is a theory that premature ejaculation is related to performance anxiety so you would need to tackle this. I suggest you talk about it with your doctor and he may give you a course of cognitive behavioural therapy which has shown some efficacy for the condition. There are also medications such as the SSRIs that may be useful.
The main thing is to try and not be embarrased. Our bodies all work/don't work in different ways and many people suffer from PE in their lives as well.