In ancient England, archery contests were held to see who could make the longest shot with a bow and arrow. This expression is so old it has become imbedded in the language as an idiom associated not with a particular sport, but all sports,games, and activities associated with risk. For example, if "long shot" was used literally in a sport, it would be associated with shooting an .arrow, a bullet or something thrown in track and field. Thus it's a long shot for an archer to hit the bullseye at 90 meters.
This idiom refers primarily to the word "swing" as an action and not to a specific sports term.