Olly, 25, was supposed to be Ben's best man at the afternoon ceremony in Chelmsford, Essex but had to cancel when the ITV show's producers could not find a compromise.
A family friend told the Daily Star: "Olly's absolutely gutted. He can't believe that the producers couldn't have come to some arrangement."
The friend added: "He was hoping that Simon Cowell would have been able to organise something like a taxi motorbike to take him to the ceremony and rush him back to the studios."
An X Factor spokeswoman confirmed: "Unfortunately Olly's not going to the wedding. It's too difficult to get from the show's studios in Wembley to where the wedding is taking place and then back again."
She added: "I'm sure they will be in touch on the day and do something personal to mark the occasion."