Rivers referred to Amanda Berry, 27, Gina DeJesus, 23, and Michelle Knight, 32, who for over a decade were tortured and held captive.
Ariel Castro, Berry, DeJesus, Knight's kidnapper, pleaded guilty to charges of kidnapping, torture, rape.
"They got to live rent free for more than a decade," Rivers.
"I'm a comedienne. I know what those girls went through. It was a little, stupid joke. There is nothing to apologize for," Rivers.
"Our clients have become aware of Joan Rivers' unfortunate comparison of living in her daughter's guest room to their captivity," Attorneys for DeJesus and Berry.
The remark is "a new low" that warrants an apology, Attorneys continued.