True, but I'm talking about native English speakers. Those people likely have their own terminology in their own language to describe themselves.
Also an ex-patriate is typically in the professional class. So those "English" teachers who teach in Japan, etc., may think of themselves as ex-pats or try to frequent "ex-pat" hangouts but they aren't necessarily because of two things: one, they have not been working at their home office and then transferred and typically they do not hold prelesional degrees -though they may hold "certificates" or whatever. They are in effect temporary workers on a limited stay visa, often needing annual renewal by hopping to a third country to have it renewed themselves. For ex-pats all this or arranged by their employers.
Looking at online dictionaries there is no hint of temporariness [0], [1]. Wikipedia refers to it as "a person who resides outside their native country" and often referring to "a professional, skilled worker or artist from a wealthy country" [2], which matches exactly the way I see it used. Similar to other commenters here, I also mostly encounter it being used by skilled, first world professionals to separate themselves "from the plebs" of poorer immigrants.
PS I do not disagree that some use cases could include temporariness (wikipedia mentions academic discourse and something about some british civil workers a few decades ago) but this is by far neither the unique nor the most common way it is used nowadays, nor how historically it has often been used long before.