Ex-Employer wants to sue me for starting my own buisness

I worked for a company for about 3 years. I filled out normal employment application and a Confidentiality agreement. That agreement was basically to ensure all company secrets remained company secrets, like ink formulas, customers. Towards the 3rd year of my employment I quit. I couldn't stand the corruption I uncovered while working there as the General Manager and didn't agree with the owner's approach to certain issues. About 2 months after my leave I began my own company similar to the job I quit from. However I have incorporated a few additonal services, etc. Now my ex-employer wants to sue me because he thinks I started my own company in the same field. However, I did start my own company in the same field, just no one from my old job knows that. Where they only provided 3 kind of services, I provide the same 3 and 6 additional services they didn't. Some of the customers from that employment I now do work for indirectly. Meaning its subcontracted out to me by another company but the end customer is still the same. What I want to know is can my ex-employer sue me because I started my own company and is considered a competitor to him now? My ex-employer "hates" me because I just quit without noticed. I just got fed up, and packed all my personal items and left. I was the salesman, marketing, buyer, production manager, pretty much general manager and some. So when I left, they lost all that talent and now are bitter for it. Does my ex-employer have any legal grounds?