Fantasy depictions of huge castles usually show gargantuan white walls reaching the skies. Let's have a walk through a real, monstrous castle that never fell to a siege and see what was practical in castle construction
Lets start from here:
Many castles would fit among these walls. But this is just a yard before the outermost gate.
When you pass the gate here you are still a long walk from the real entrance:
The tower in the distance is the actual heart of the castle. It's a long walk even if nobody bothers you.
Note that the large windows are most likely XIX century additions when the castle was used as a storage house by the Prussians.
And then when you get inside you reach the castle-within-castle:
Which is protected by a serious gatehouse with all the traps money could buy:
And then there is still another castle-within-castle:
And within this innermost castle there is still the tower to be handled:
So the bottom line is that it was much more practical to create a labyrinth of walls and gates (that can be opened wide for a legal visitor) rather than a very high structure that would require an hour of mountaineering just to get to the bathroom…