Judging is Hard

This month I have a few thoughts on aerobatic judging to share.

Judging is Hard

I sometimes hear grumbling about the imperfection of judges. It’s true that sometimes judges make mistakes. But let’s have a bit of compassion. Judging is hard!

Easy Errors

Some errors in competition flights are easy to spot. For example, I think pretty much everyone can see mistakes in the angle of a line. If a competitor is supposed to fly a vertical line and flies it at 85 degrees from the horizontal instead, that is a visible error. Someone off the street could probably do it as well as any of us.

It’s also usually easy for judges to spot when people are flying in entirely the wrong direction. I think all of the pass/fail criteria are pretty easy for judges. In the same category are Hard Zero criteria about tail slides, and the direction of rollers. These criteria don’t pass the “man on the street” test because they involve an awareness of jargon and details particular to our sport, but they aren’t exactly hard for people to see once the details are explained.

Judges may not exactly agree what the score should be for a particular effort at a round loop, but they pretty much all agree whether it is a perfect circle or not. Perfect circles are easy to visualize.

The Not-So-Easy

But! The thing that most dominates the scoring (and therefore is most important in determining the final rankings) is the rotations. Errors in rotation are the most common errors made by competitors. These errors are not just common, they are also highly penalized by our system. A 15-degree error is almost unheard of on a vertical line, but a three-point deduction on a roll element is quite easy to earn. Only the worst pilot misses a vertical by 15 degrees, but even the very best pilots in the world sometimes miss a roll stop by that amount.

Unfortunately, while the “one point per every 5 degrees” rule is the most easily memorized criteria in the rule book, it is surely the hardest to apply. I know this because when I teach judge school, I put the “Judge Roll Trainer” from jimbourke.com on the projector and test each student’s ability to assess roll errors. What I’ve found is that judges dramatically understate errors in roll. If a roll error is anywhere under about 30 degrees, judges want to give a one- or two-point deduction. Once the roll error is over about 45 degrees, they want to give a Hard Zero. In between they want to give about 3 points. They almost never want to give a specific point deduction bigger than that.

It’s not that judges are not trying. They really are. But it takes a lot of experience to see roll errors in actual 5-degree increments and even with experience human beings just can’t do it perfectly within the pace of an upper category flight. Judges who do not feel confident tend to under-penalize, and very few people feel confident that they can assess these errors accurately.

By the way, this is why it is so important for competitors to never fix a rotation error. If you fix it, you give judges a second chance to see how big the error is.

What to Do?

What are the solutions? One thought is to change the scoring to reflect what judges are actually able to do. Work with their strengths instead of trying to get them to do impossible things. If we went this route, we would drop the “one point per five degrees” rule and go with something like “small error in rotation is 1 point, medium error is 2 points, big error is 3 points, more than 45 degrees is a zero”. That might seem too simple. Maybe it is. But maybe it will start a conversation that helps us out in the future.

In the meantime, what I’ve learned is that when competitors fly with high rotation rates and athletic stops, judges give high scores. Judges respond to this style of flying more than they respond to precision. Competitors, use that to your advantage if you aren’t already.

Another reason that judging is hard is because there is so much to know. Our rule book has a couple dozen instances of language like “if the criteria is not met, deduct at least one point”. But the effect of this language is not worth the cost. The minimum deduction is half a point. Therefore, the only effect this language has is to keep judges from giving a half point deduction. They must give at least one point instead. That doesn’t seem so hard. But how many of our judges remember all these criteria? Do they remember these criteria as well on a contest day as they do when taking an open book test? Doesn’t it seem likely that most judges forget all about this and give a half point when they feel like it? Probably. And does it really hurt anything when they do? Probably not.

I think it would help judges a lot if we could find a way to get rid of some of the memorization and give them more time building the practical skills that they need. I’ve been raising this subject with the Rules Committee this year and I’d love to hear your thoughts.

Contact Me

Remember that you can always reach me at president@iac.org and I love getting your emails so please keep sending them!