Contract say I can complete but must repay % of previous billed revenue?

I am an IT consultant who has recently left my employer after 5 and a half years due to an unhealthy work environment (working 6+ days a week, 12+ hours a day).
I have been working in the industry for 10 years and have become very specialized in the level of work and service I am able to provide. Due to my skill, personality and work ethic, I now have former clients who are not happy with the level of support they are receiving from my old employer and have contacted me to retain my services to help them under my new firm.
To my unfortunate mistake when I started, I signed an employment agreement with some very unusual terms. I have received a letter from my past employer saying that if I pursue any business relationship with these former clients that I have the following obligation:
14.1. Emplover's Customers. In the event that this Agreement is terminated or Employee's employment by Employer is terminated by either party for any reason, the Employee shall not be prohibited from competing with Employer. If, however, at any time within a period of two (2) years after such termination, the Employee, directly or indirectly, solicits any customer of Employer in his/her individual capacity, as a member of a firm or as an employee, the Employee agrees to pay Employer an amount equal to one hundred fifty percent (150%) of the highest annual revenue billed by Employer for services rendered to such customers) during the three (3) year period immediately preceding the date of the termination of this Agreement or the employment relationship.
Can they legally enforce this? I was the direct face of the company to these clients, put in the blood, sweat, and tears, and set the level of service so high that just my leaving has dropped the bar. The client no longer wishes to maintain the relationship and has valid cause to no longer be a customer. Am I essentially helpless, being someone who is capable and knowledgeable on these client's environments, from helping them for fear of major financial loss?