your friend doesn't have a hold on you - you are saying that he has.
if you feel a certain way towards your friend, own up to it .. don't wait for him to make the first move. you can wait until the cows go home, you know .. eventually, like now, you're held back from pursuing other relationships simply due to the words, what if.