Maybe this is a better analogy. Your dinged prop is like a wrinkled up car trying to keep up with the draft @ Daytona. Wrinkles whether it's your prop or the stock car, prevent the water/air from flowing smoothly over it. Think about how much harder those wrinkles are to push through the water/air. Knife edge vs. spoon edge.
The prop obviously is what pushes the boat through the water and the hull does provide some of it's own lift, but, the prop lifts the hull as well, especially as you're approaching WOT and don't have much boat left in the water anyways.
Your numbers pencil out pretty well. Your lower unit is 1.86 so: 1.86 x 5000 =2688 prop RPM x22"/rev (prop pitch) = 59136"/min. 59136/12= 4928'/min. 4928/5280' per mi=.93. 60MPH x .93=56MPHx10% slip =50.4MPH
Using the same formula, that same prop running 5800RPM should net you ~61MPH.
- - - Updated - - -
Have you checked your throttle linkage to make sure your getting WOT ? Also will a 4 blade prop have more drag from friction than a 3 blade ? My Sea Ray is a mutt , Its I/O merc with a v-8 chev and with 3 or 4 in the boat and full fuel/water tanks it has a hard time getting up on plane. I have a 4 blade prop I could put on it but logic seems like it would have more drag ?? I'm thinking I should start shopping for a SS prop.
Update- From my experience a dinged up prop will gain RPM and lose top end because it doesn't "hook up" as good.
Yes, a 4 blade prop has more drag as well as robs HP via extra weight. However, a 4 blade MIGHT also have more lift than a 3 blade as well as having greater "traction" due to the extra blade. It might plane out faster, it might allow the boat to carry plane at a lower speed and it might create more bow lift. More bow lift = more boat outta the water = maybe faster/smoother. An extra blade on the prop might be a worthwhile trade if you get enough extra boat outta the water. Bolting it on is the only way to compare.