I think this is actually an important part of Mycroft’s character that people tend to overlook in favour of the silly (and very enjoyable) cake jokes.
When Sherlock Holmes calls someone the most dangerous man you’ve ever met, I don’t think he’s speaking lightly. And I think a huge part of Mycroft’s danger is the very fact that he seems so innocuous. You’re chuckling along with him, chatting about this and that, thinking what a loveably clueless old thing he is, right up until the point where he tells you he’s poisoned your tea and would you be so kind as to tell him the name of the government Mole in your employ, because really, dead bodies do so clutter up one’s study.