tokenizer.json and tokenizer_config.json files?
#9
by ajaeger - opened
Can tokenizer_config.json and tokenizer.json files be provided for these models as is the case with many other models on Hub? When trying to use these models with instructlab to ingest PDF documents, I get an error that these files are missing. Maybe there is a workaround, but I am new to this stuff.