I would be very comfortable with the conclusion the narrower firebox/grate area on the Castle is the main constraint here.

The A4 has a broader box supported by that trailer rather than a narrower taller one that fits between the last set of driving wheels on the Castle. More grate area, heating area, and a better draught up through the grates can make a huge difference in available power, all other things being more or less equal.

I would believe the GWR got about the best out of the basic design concept of the non-trailered locomotive that anyone could, but their aversion to a trailer-supported box on express locos limited their top-end performance. One wonders whether this was due to a corporate conservativeness, especially as the Pacific-types became pretty mainstream in engineering thought for the other three companies.