Is it permissible for me to marry the boy I had an affair with?

I am from harur. I had a love affair with a boy without knowing he was married. The local jammat people called me to ask the details. I told them that I did not know he was married so I had a affair. They took a promise that I should not talk to him and the matter was settled. My dad wanted to get me married but because the jamat people problem the new alliance were broken. The boy who had an affair with me gave thalaq to his wife for zina, which was proved in jamat 5 years back and he is looking after his two kids since then. He has given back mehar and his ex wife jewels as per fatwa issued by district Qadhi.
Now all the people from past 5 years are saying I am married to the boy, but I have not married. Since all are saying so does Islam permit me to marry him, so kindly give me a fatwa so that I can take a good decision?