A strong castle is critical to your ability to fight over the long term. You should always have your craftsmen working on upgrades, and when they are working you should be thinking about the next upgrade, how long it will take and ensuring you have kept or gathered enough resources to build it before it finishes. Wasting time asking for caravans or farming for rss because you spent everything on more units is a dumb move.
If you are ever planning on buying sapphires I suggest getting them as early as you can and using them to buy more craftsmen before anything else, it will help you the most over the long term.
