Vendors only pay about 1/10 the value of gems. They resell them to gem merchants who pay about 1/4 the value and who then sell them to expensive jewellery shops abroad who pay about 1/2 the value. The jewelers use them in jewellery or simply polish and resell them to the aristocracy for about twice their value.
Want to get rich? Hoard your gems for a long time, then give up adventuring, move to a big city, and open a jewellery store!