16:36 22 February 2016
During the announcement, he made it clear that he quit the show and wasn’t fired. He said his decision came during his time off at Christmas after the 2015 final.
"I did love it and it's not a diss to the show. It's a great show and I loved it. But it's definitely a once in a lifetime style thing," he said.
He then joked: "You can't have a curry when you're on it on a Saturday night because you're there.
"What's really good about X Factor is watching it at your house with your friends [and] having a curry.
"When you're there, there's no curry. Simon promised me curry."
Meanwhile, there is no confirmation yet as to whether Rita Ora or Cheryl will return but it is thought that Simon will remain on the panel.