Former “Access Hollywood” host Billy Bush appeared on CBS’ “The Late Show With Stephen Colbert” Monday night in his first interview that Bush has done since Donald Trump became president and since Mr. Trump has publicly denied that it was not his voice. Bush said he was “infuriated” when he learned President Trump didn’t take responsibility for the 2005 recording. Bush also mentioned he “would have called the FBI” had he known “there was a man detailing a sexual assault strategy.”

“Last week he [Mr. Trump] came out saying ‘that’s not my voice on the tape,'” Bush said. “Like I said … you can’t say that … that is your voice … I was there … you were there … that’s your voice on the tape.”

Bush then went on to say: “Then you got 20 women at the time — I don’t know what the exact number is because there is more — but 20 women who used their names. We’ve got powerful people being held accountable now and sometimes there are anonymous sources. All of these women came out with their names and told their detailed accounts. So when he said this … it infuriated me on the personal front and then I ended up Googling and I read an article with all of these women’s accounts — when that article came out it was 13 days after I was fired so I was in my own personal shock — fight or flight — and so I never really read it all the way.”

He continued: “20 women don’t get together and say, ‘Hey you know what would be really fun? Let’s take down a powerful guy together … haha.’ No they don’t … and I say you’re reopening wounds on them too. Enough’s enough … stop playing around with people’s lives.” The audience then cheered and clapped. “That upset me … so I wrote!”

Moments later, “Late Show” rebroadcast a portion of the “Access Hollywood” tape and got Bush’s reaction almost a year and half later to Mr. Trump’s boasting of grabbing “women by the p****.”

“The first time I ever heard it was about three days before it leaked … and so it was like a gut punch … and it’s a gut punch now … it will always be,” Bush said. “At the time, it was 2005 and his ratings went through the roof. He was the big star of NBC and I sort of equated what he was saying to some kind of crass standup act.”

When Colbert pointed out that there was no audience, Bush said that he and the crew on the bus were Mr. Trump’s audience.

“He was just regaling the seven guys on there and doing his bit,” he said. Bush continued, “If I had thought there was a man detailing a sexual assault strategy, I would have called the FBI, not just reported to my executive producer, so that was it for me.”

Bush added that he was with Mr. Trump “all the time” because “The Apprentice” was doing so well.

“Like I wrote in the op-ed, I think a lot of us were complicit … everyone kind of had to kiss the ring of The Donald because he was making so much money for NBC … so I was with him all the time,” Bush told Colbert.

Bush then explained how he felt shame with his behavior being associated with Mr. Trump.

“No question I feel like I sacrificed a little bit of who I am in the moment — and I don’t know if we will ever get to the place to advance the conversation — right now we’re talking to victims and perpetuators — and eventually we all say in the media we like to see the conversation elevate and when we do … how we talk to boys … when the conversation gets degrading … how do you exit the conversation … change the topic or intervene some way. I hope we do get there.”

Colbert cross-examined Bush on the contents of the “Access Hollywood” tape and asked what it meant when Mr. Trump said, “I moved on her like a b***h, but I couldn’t get there.”

Bush said he was confused as well. “I think he meant [himself] … which I was totally confused and backwards. He gets going and I don’t think he really knows,” he said.

Colbert cracked, “‘I took her furniture shopping.’ Does that seem like common foreplay to you?”

Bush said one very sobering moment was after he checked himself into a a soul-searching retreat center in California to “get better, be a better man” the day Mr. Trump was being sworn in as president. He said as he checked into his cabin, he saw a TV screen with Mr. Trump on it.

“He’s got the hand up; I’m going into my little cabin to do the work,” said Bush. “Irony.”

Bush then reflected on how he felt about NBC firing him — and his response to Matt Lauer’s firing after allegations of sexual misconduct.

“It was a difficult time … it was very hot emotionally in the country. A lot of people weren’t comfortable with the man who was close to winning the election … a lot of things happened quickly. My boss … months later said publicly that I think we may have moved a little quickly …” Bush said. “I would have liked to address the audience that following Monday.”

Bush continued: “Women must be believed,” he said. “We kind of have to find our way to have the dialogue. I’m worried that the dialogue isn’t going to be around because we move onto the next thing.”

“The bus ride was the tip of the iceberg,” Bush said.

Bush also said that he is “finally lucky” because he recently survived getting hit in the head with a golf club; a doctor told him if it had hit him slightly above where it did, he could have died.© 2017 CBS Interactive Inc. All Rights Reserved.