I've been wondering for a while now if maybe Daniel isn't the lovechild of Victoria Grayson and David Clarke...if that was perhaps even the impetus for Conrad framing David Clarke in the first place (not unlike Victoria effectively "banishing" Lydia after learning that Lydia and Conrad were having an affair).
The one scene in particular that really got me thinking Daniel could be their lovechild is the flashback in which Victoria is being coached about how to testify against David Clarke; she doesn't seem to be too bothered when Conrad says that they could lose the house and all their possessions, but she reacts very strongly when he suggests she could lose Daniel. Connect that with all the other dots in the plot line of Victoria favouring Daniel over Charlotte, of the fondness and closeness displayed between Victoria and Daniel, and, well, it does seem plausible...
For all the Daniel-and-Emily fans, this would of course be a very unpleasant (or cringe-worthy)twist, but I do think that it would explain a lot in terms of the back story, and why I found myself almost inexplicably starting to feel sorry for Victoria Grayson during the last couple of episodes, despite her, um, Queen Victorianess.