Change is always scary, but it is sometimes for the better. Since, as you say, have some aspects of your life covered, you should balance your overall feelings and then decide with conviction. I am sorry you have to consider quitting something you liked.