Was my case considered Fraternization?

I was a program manager for a government contractor and terminated from my job for having an affair (happened only one time) with someone who worked under my management. I self reported the affair to my superior in which he reported it to the vice president and the HR department, the woman (Tiffany) i had the affair with was then transferred to another program. Nothing was done until someone reported it to Ethics and during the investigation I was never interviewed nor was my manager, the only people interviewed was my employee in which I had the affair, my wife (who also works for the same company) and the program manager that Tiffany was transferred to. In our employees handbook the only thing the handbook says is that a manager cannot have a relationship with anyone under them directly. Do you think I would have a case for a lawsuit?