There's a plethora of valid ways to compare apples and oranges

Apples and pears can *cross pollinate and can be grown on the same root stock.

I've always taken it to imply that at a certain number of differences, the comparison isn't useful anymore, not that there are no avenues of comparison at all.

"This orange isn't nearly red or crunchy enough to be a good apple."

I don't know about you, but I sort all my fruit according to tanginess.

They are very comparable. Both round fruit, often cut into wedges, both can be used to obtain juice.

I think what they saying "don't compare apples and oranges" is trying to convey is that we shouldn't judge an orange by the standards we judge apples. It would be silly to eat an orange and complain that it wasn't enough like an apple.