Create tokenizer_config.json
#1
by lysandre HF Staff - opened
The model max length is currently set at a value within the transformers toolkit directly, as can be seen here: https://github.com/huggingface/transformers/blob/main/src/transformers/models/squeezebert/tokenization_squeezebert.py
We're removing these definitions as legacy; we're therefore updating the remote checkpoints to ensure that this information is not lost. Thank you!