You are correct. The overwhelming consensus of the Poskim is that one must wash for wet foods. However since there are a number of Rishonim who write that this washing only applied when people were careful to keep themselves tahor, we do not say a beracha and consider this a safek berachos l'hakel, even though one should certainly follow the majority opinion to wash. The Vilna Gaon even held that a beracha should be recited.
While it is not fair to say that no one ever washes, because this is the source of Urchatz at the Pesach Seder, it certainly is not nearly as common as it should be. Some explanations given are that if one eats a wet food with a utensil and does not touch it with his fingers, and it is normal to eat this food in that manner, it is not necessary to wash. As a side point, this is one problem with the minhag of some to use boiled potato for Karpas. Additionally, if the food is smaller than a kezayis there are opinions that one need not wash. Mishna Berura is lenient on this in the first edition, but in a later print he added in the margin that he changed his mind and writes that one should wash.