The all_party_privilege_map queries can be expensive, too.  "nested loop" means just what it says, and when you have nested loops nested a few deep you're far, far from the O(log2(N)) comfort zone.  We've been zapped with this on SSV2 ...

So unfortunately, with things as they are one must experiment.