Can a company make me pay back relocation expenses if I have not signed a contract

I have been working at my present job for almost 2.5 years. The company has a good relocation package, and another that is even better for existing employees transferring to another company location. I initially received the new hire relocation package as part of my acceptance of employment. As soon as I accepted the offer, I put the house on the market. As you can imagine in this economy, it took me over 2 years to sell it. The company actually paid my temporary living expenses until my home sold. While trying to sell my home, the top manager and HR decided to give me the better relocation package, because it offers to make up losses if I sell the home below appraised value and provide a bridge loan so equity can be pulled out to purchase a home near work. Both packages state the company will pay up to 7% Realtor fee to sell my home. I traveled almost 3 hours one way every weekend for the two years so I could be with my family!
The company never stated verbally or in writing that I would be required to repay any relocation fees, even after selling my home and getting moved.
The top manager asked me to bring any offer to them, even if it is very low and they would try to make it work so I could get moved and get this behind us. Well, I got an offer $43,000 below appraised value. I reluctantly took it to the top manager. The manager asked me to try and counter offer, but to not lose them. I made a counter offer and they pulled the offer. The manager asked me to take their initial offer and would make it work. I stated several times that I would not take an offer that would require me to come out of my pocket. So, the seller and I signed contract and moved toward closing. When the closing office informed me I would need to pay over $20,000 at closing I gave it to HR so they could issue me a check to pay this difference. The manager also tried to re neg on paying the realtor's fee which was clearly in the relocation package and offered in their offer of employment.
I failed to mention that the better relocation package offers a bridge loan where you pull out 90% of your equity so you can go ahead and move. I refused to take them up on this because we were concerned they would backout on some of their relocation coverage. They finally gave me an ultimatum to go with the bridge loan or they will stop providing me living expense help. The manager said I needed to show I was committed to the company before they could continue helping with my relocation. So, after reaffirming with the manager and HR that they would make sure I would not have to come out of my pocket to pay for any relocation expenses, I took the bridge loan. Three months after taking the bridge loan and living where I work, we received an offer on the home we had been trying to sell for over 2 years. When I received the $20,000 check to pay at closing (which was a few days before the closing), I received and email from the top manager that stated basically that I would not have to pay back any relocation expenses except the $20,000 dollars if I decide to leave within the next four years!!! FOUR YEARS! This was a bomb shell and to not mention this until I received the check and after relocating, accepting the offer to sell the home, and a few days before closing. I did not reply to the manager's email and they have not asked me to sign any agreement/contract. As stated, I've been working for this company for 2.5 years now.
Lately, my manager and top manager have been treating me as though they can abuse me and I don't appreciate it. I guess they figure I will take what they dish out since they fill I can not afford to pay back the $20,000.
With this information, can you tell me if I would have to pay back this money if I leave within the next 4 years?
I feel like they were dishonest after telling me I would not owe anything if I accepted the bridge loan and move. They tried to be sneaky and tried to not pay the realtors' fee when it was clearly offered to me in writing when they offered me the job. And the email floored me. I can't trust them now.
I feel like I am trapped for the next four years and they know it, like an indentured servant. I have had other companies give me relocation packages, but they were very upfront with me that I would have to pay the relocation fees back if I left in the first 12 months. At 6 months I would owe 50%.
I would greatly appreciate your feedback.