Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
amilios
on April 5, 2023
|
parent
|
context
|
favorite
| on:
OpenAI Tokenizer
Yes, the model has something like an EOF token which it emits for the output to end. It is part of the probability distribution that the model predicts.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: