I dont think this is poor design rather than "the element of surprise." Attacking with 16 people, the defenders become wide awake and active and actually try. When only 2 persistant people are attacking, the defenders tire out and think "meh, someone else'll get 'em" and everyone afks and its an easy take. I see this in other towers too. Some days ill get Castle flag in the double digits before SoN wakes up.

Imo, the only design that needs change is for castle & deadwood to have closer attacker respawns to balance the time it takes to get back to flag room a little more.