AWS | Community | Deep dive into tokenizers | S03 E19

In this episode we have the chance to welcome Mistral for a deep dive into tokenizers. The amazing Harizo Rajaona goes into the details of how tokenizers are architected. Also Mike Chambers makes a recap of the best annoucements from the NYC summit including Memory retention, code interpretation, prompt management and prompt flows.

Check out the recording here:

Feedback:
Did you like this episode ? What are the other topics you would like us to talk about ? Let us know HERE.

Shared links:

AWS news blog: https://aws.amazon.com/blogs/aws/
Top announcements from NYC summit 2024: https://aws.amazon.com/blogs/aws/top-announcements-of-the-aws-summit-in-new-york-2024-2/
Memory retention and code interpretation: https://aws.amazon.com/blogs/aws/agents-for-amazon-bedrock-now-support-memory-retention-and-code-interpretation-preview/
Prompt flows: https://aws.amazon.com/blogs/machine-learning/streamline-generative-ai-development-in-amazon-bedrock-with-prompt-management-and-prompt-flows-preview/
The doc page on tokenizers: https://docs.mistral.ai/guides/tokenization
The GitHub page for mistral-common: https://github.com/mistralai/mistral-common
The HF model card for 7B-v0.3: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3

Reach out to the hosts:

Mike Chambers: linkedin.com/in/mikegchambers/
Tiffany Souterre: linkedin.com/in/tiffanysouterre
Harizo Rajaona: linkedin.com/in/harizorajaona/

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Site Terms, Privacy, and more.

Deep dive into tokenizers | S03 E19 | Build On Generative AI

Let's deep dive into tokenizers with Mistral, the french unicorn in generative AI !

Comments