Tao He 294f8f6665 [BugFix] Pass tokenizer_config to local_tokenizer_group (#3754) ... Signed-off-by: Tao He <sighingnow@gmail.com>		2024-04-03 20:31:46 -07:00
..
configs	[Model] Add support for DBRX (#3660)	2024-03-27 13:01:46 -07:00
tokenizer_group	[BugFix] Pass tokenizer_config to local_tokenizer_group (#3754)	2024-04-03 20:31:46 -07:00
tokenizers	Re-enable the 80 char line width limit (#3305)	2024-03-10 19:49:14 -07:00
__init__.py	[Tokenizer] Add an option to specify tokenizer (#284)	2023-06-28 09:46:58 -07:00
config.py	[Model] Add support for DBRX (#3660)	2024-03-27 13:01:46 -07:00
detokenizer.py	[Misc] Some minor simplifications to detokenization logic (#3670)	2024-04-01 13:22:06 -07:00
tokenizer.py	[Misc] Some minor simplifications to detokenization logic (#3670)	2024-04-01 13:22:06 -07:00