The issue is that if you need his services and (especially) if he is the only choice for his specialty available to you (For insurance or geographical reasons, etc) if you don't agree with his political views can you feel entirely comfortable that you would get the best care available to you there? That right there violates the oath.
I don't know that it would bother me. Maybe it is just me though. I suppose if enough people did feel that way it would violate the oath.