Deep dive into tokenizers | S03 E19 | Build On Generative AI
Let's deep dive into tokenizers with Mistral, the french unicorn in generative AI !
Tiffany Souterre
Amazon Employee
Published Aug 1, 2024
Last Modified Aug 8, 2024
In this episode we have the chance to welcome Mistral for a deep dive into tokenizers. The amazing Harizo Rajaona goes into the details of how tokenizers are architected. Also Mike Chambers makes a recap of the best annoucements from the NYC summit including Memory retention, code interpretation, prompt management and prompt flows.
Check out the recording here:
Loading...
Feedback:
Did you like this episode ? What are the other topics you would like us to talk about ? Let us know HERE.
Did you like this episode ? What are the other topics you would like us to talk about ? Let us know HERE.
Shared links:
- AWS news blog: https://aws.amazon.com/blogs/aws/
- Top announcements from NYC summit 2024: https://aws.amazon.com/blogs/aws/top-announcements-of-the-aws-summit-in-new-york-2024-2/
- Memory retention and code interpretation: https://aws.amazon.com/blogs/aws/agents-for-amazon-bedrock-now-support-memory-retention-and-code-interpretation-preview/
- The doc page on tokenizers: https://docs.mistral.ai/guides/tokenization
- The GitHub page for mistral-common: https://github.com/mistralai/mistral-common
- The HF model card for 7B-v0.3: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3
Reach out to the hosts:
- Mike Chambers: linkedin.com/in/mikegchambers/
- Tiffany Souterre: linkedin.com/in/tiffanysouterre
- Harizo Rajaona: linkedin.com/in/harizorajaona/
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.