A problem with this suggestion is that the average changes! So, if you are above average, you get a score. The next person does the same puzzle in the same time, but their score would now be less, because your solve upped the average.
Another problem is that you would need to average these in themselves. If you do 10 at a lot above average it is the same as doing 20 at a little bit above average, or 100 at just a tiny bit above average, so you still have no gauge.
So, you still have no clearcut unless you put all of that in to play.
Also, a score of +10 on a quick 4x4 is actually better than +10 on a difficult 5x5 - so you still have not taken out the skewing, unless you scale score in proportion to the average time.