Known for his fortitude in the boxing ring – and later, his inner strength, as Parkinson’s Disease ravaged his body – Muhammad Ali had “one weakness” that ended his early marriages.

“It was too much temptation for him, with women who threw themselves at him,” his third wife, Veronica Porsche, 60, tells PEOPLE. “It didn’t mean anything. He didn’t have affairs – he had one-night stands. I knew beyond a doubt there were no feelings involved. It was so obvious, It was easy to forgive him.”

Nevertheless, Ali’s philandering eventually “became too much,” and the two divorced in 1986 after nine years of marriage.

A pre-med student at USC in 1974, Porche met Ali in 1974 while volunteering as a hostess at a charity event at the legendary Rumble in the Jungle in Zaire. The boxing champ was married to second wife Khalilah Ali at the time.

For much more about Muhammad Ali’s life and legacy, read the new issue of PEOPLE, on newsstands Friday

Khalilah remembers her split from Ali as “clean and smooth… no fights or struggles.”

“I hope a lot of people learn from us, as being kind to each other and as being free to see their children, anytime, if a divorce occurs,” she told PEOPLE.

Ali found fidelity as Parkinson’s Disease took a toll on his body, beginning in the 1980s. He married Yolanda “Lonnie” Williams in 1986, and the two remained inseparable until his final day. Ali had nine children, including two with women to whom he was not married. All nine were at his bedside when he passed away on Friday.

For more on Muhammad Ali, tune in to PEOPLE’s List, Saturday, June 11, 8/7c on ABC

“We had a beautiful life,” Khalilah Ali says. “It was like a roller coaster, it went up and down, and at the end you get that thrilling moment, which is always good. Muhammad Ali was my husband, my first love, my best friend and also my legacy.”

“I have felt guilty to this day for divorcing him because I knew it hurt him,” says Porche. “We hurt each other. And I still love him. He’s had a profound effect on everyone.”