Reality star Jon Gosselin admits he's going through a tough time at home right now, but he wants fans to know he's been faithful to his wife.

Last week, Gosselin was photographed leaving a bar in Reading, Penn., with a woman named Deanna Hummel. Hummel later came forward to say that she and Gosselin are just friends – but her brother, Jason Hummel, claims Deanna and Gosselin, a married man with eight kids at home, have been having an affair for three months.

"These allegations are false and just plain hurtful," Gosselin says in a statement exclusively to PEOPLE. "As I adjust to the attention that comes from being in the public eye, I need to be more careful and aware of who I am associating with and where I am spending my time. But the bottom line is, I did not cheat on Kate. I am sorry for putting my family in this awkward position, given some poor decisions and bad judgments I've made recently."

Earlier Wednesday, Deanna Hummel told PEOPLE she was devastated by her brother's comments. "My brother is making this all up," she says. "I can't even stomach the lies he's saying about me."

Gosselin, whose family life is chronicled on TLC's hit Jon and Kate Plus Eight, says he's also grappling with his family. "I am working through this difficult time with my family," he says. "My family is my top priority."

When asked by Entertainment Weekly whether they will address this on air, Kate Gosselin says, "I think we will address the bigger picture, regarding the very public focus on our family and the stress it's caused in our relationship." She adds: "Our main focus is on our family and working through this."