Charlie Sheen did his best to enjoy a day at the beach on Saturday despite the presence of an unwelcome guest: a court-ordered escort who happened to be an assistant to his ex-wife Denise Richards.

"Here is my and Brooke's weekend with [daughters Sam and Lola] being sullied by one of her spies," Sheen told PEOPLE during an exclusive interview at "Kids Connection" – an Art of Elysium benefit hosted by Brooke Shields and Chris Henchy in Malibu.

Sheen, 41, showed up to the beach front party with his girls, fiancée Brooke Mueller, 30, and Richards's assistant in tow "because a real nanny was unavailable."

"That is the thing I have to put a stop to. I can't spend quality time – times I'm never going to get again – quite frankly being observed by a perfect stranger," Sheen said of the escort chosen to oversee his visit with his girls due to current custodial arrangements with Richards, 36, which he is battling.

The actor has hopes the situation will get sorted out by the courts soon so that he can further pursue an "unpolluted relationship with my children" with the help of Mueller. "It's a very loving, safe, comforting, supportive, nurturing environment for these children," he added. "Brooke and I are incredibly stable."

"We go to malls, to movies, to parks – we just try to keep it interesting," he said. "We have stuff at the house that they are really excited about that isn't double what they have at [Denise's] so when they come to my place, it's special."

At the party it was painting, cookies and dancing that had Sam, 3, and Lola, 2, excited in front of their dad, who spent time chatting with "old friend" Shields and mingling with event organizers.

Mueller said Sam, 3, and Lola, 2, had a blast at the party. "They played games, loved the arts and crafts, kids band and all of the food," she said. "We stayed for quite awhile and made a day of it." Adding: We all ate too much!"

Legal issues aside, Sheen concluded that his family day was "fabulous." "It's one of those gifts," he said. "We are out with the kids and supporting what matters."