I'm not sure that's entirely true.
In Temple of Doom, Chattar Lal recognizes Indiana Jones in the middle of nowhere in India, and during the banquet scene, points out Indy's dubious exploits in Honduras, and that the Sultan of Madagascar threatened to cut off his balls for graverobbing if he...