Scaife also said philandering "is something that Bill Clinton and I have in common."
Scaife made the remarks in an interview with Vanity Fair, his first interview in eight years, according to the magazine. The comments appear in the February issue in a story about Scaife's troublesome divorce from Margaret Ritchie Battle Scaife.
In the story, Scaife described the lunch at Clinton's New York office as "very pleasant."
"I never met such a charismatic man in my whole life," Scaife said.
Scaife, a billionaire heir to the Mellon banking and oil fortune, said he tried to show Clinton that he isn't "a total Republican libertarian" by talking about their mutual friend, U.S. Rep. John Murtha, D-Pa.
Scaife also told the magazine: "I don't want people throwing rocks at me in the street. But I believe in open marriage."
Clinton gave Scaife an autographed copy of his book and Scaife said he later sent $100,000 to the Clinton Global Initiative.
Forbes magazine has estimated Scaife's wealth at $1.3 billion.
Several foundations controlled by Scaife gave millions of dollars to organizations run by critics of Clinton, including $1.7 million for a project at the conservative American Spectator magazine to dig up information about his role in the Whitewater real estate scandal.
To comment, go to Scaife.