The main difference between trills and mordents is in in the number of alternations between the adjacent notes. With mordents, we alternate only once between the principal note (the note symbolized by the mordent) and either the upper note (in the case of a mordent) or lower note (in the case of an inverted mordent). With trills, the number of alternations is longer, though the exact number will depend on the rhythm as well as the musician’s individual choice.
Also, trills by definition alternate with the higher note, while with mordents it can be either the higher or lower note (see diagram below).
A mordent is sometimes referred to as a short trill, which contributes to the occasional confusion between the two terms.