The evidence that this part of the island was inhabited from neolithic times is the Grabceva cave (Grabceva spilja) within reach of Jelsa, the discoveries of which gave the essential information for the study of the life of neolithic man and his culture on the island Hvar in general, and the wider area around it. The immediate vicinity of the Greek Pharos and the exceptional location of Jelsa indicate the existence of the settlement from Greek times.

On the mountain spur that projects to the south above Jelsa, is Tor, a solid, massive fortification built of huge regular shaped blocks of stone joined without mortar. It was a Greek observation point, which stands on an older Illyrian fort. The location is ideal for observation as it dominates the valley of Stari Grad, the Hvar channel and Jelsa, and there are high cliffs behind it. Slightly east of Tor there are the ruins of the fort of Grad (Galesnik), which as an observation point has the same importance as Tor. It is probable that this fort is from the late ancient period, even though it was of great importance in the Middle Ages.

The material remains of the Roman period prove the existence of a settlement in the area of Jelsa. Several villas are preserved in the area of Kutac, round St. Rock and in the cove called Carkvica of St. Luke.

The medieval settlement of Jelsa formed at the end of the 14th century as the port of the village of Pitve, was mentioned in the Statute of 1331 as a "Fons vocata Jelsa" and "Portus se Pitve". The same statute mentioned "Civites vetus Jelsae" (the old town of Jelsa), which referred to the already dilapidated remains of the port of the Middle Ages on the peninsula of Gradina . On this site in 1605 the Augustinian monastery, of which only the church remained, was deserted, the cemetery was laid out in the 19th century and has been in use up to the present day.

Jelsa developed primarily, especially in the 19th century, due to shipbuilding and navigation, from small seaport town whose inhabitants dealt mainly with fishing and agriculture. This conditioned its urban expansion and its acquisition of the main role on the central part of the island. The harbour and break-waters were begun around 1830, after which in about 1850 the reclamation of marshy land started, round Vela and Mala Banda and Soline, and on the resulting dyke poplars were the town park was created.

On the sea-board next to the town, a town council building and a town hall were built. The town hall has neo-Gothic furniture from the end of the 19th century, made by Perisuti, an artist from Jelsa.

In the area of Mala Banda there is the Dubokovic-Nadalini house from the end of the 19th century, partly built in the 16th century. The house has some well-preserved antique furniture, an extensive library, the family archive, and a series of paintings and items of art; the house alone is a small museum.