Classify Jira Tickets with GenAI On Amazon Bedrock
Replace traditional NLP approaches with prompt engineering and Large Language Models (LLMS) for Jira ticket text classification. A code sample walkthrough
Tanner McRae
Amazon Employee
Published Nov 4, 2024
Photo by Annie Spratt on Unsplash
Remember the days when classifying text meant embarking on a machine learning journey? If you’ve been in the ML space long enough, you’ve probably witnessed at least one team disappear down the rabbit hole of building the “perfect” text classification system. The story usually goes something like this:
- Month 1: “We’ll just quickly train a NLP model!”
- Month 2: “We need more training data…”
- **Month 3: “**This is good enough”
For years, text classification has fallen into the realm of classic ML. Early in my career, I remember training a support vector machine (SVM) for email classification. Lots of preprocessing, iteration, data collection, and labeling.
But here’s the twist: it’s 2024, and generative AI models can “generally” classify text out of the box! You can build a robust ticket classification system without, collecting thousands of labeled training examples, managing ML training pipelines, or maintaining custom models.
In this post, we’ll go over how to setup a Jira ticket classification system using large language models on Amazon Bedrock and other AWS services.
A common ask from companies is to understand how teams spend their time. Jira has tagging features, but it can sometimes fall short through human error or lack of granularity. By doing this exercise, organizations can get better insights into their team activities, enabling data-driven decisions about resource allocation, project investment, and deprecation.
Traditional ML models and smaller transformers like BERT need hundreds (or thousands) of labeled examples, while LLMs can classify text out of the box. In our Jira ticket classification tests, a prompt-engineering approach matched or beat traditional ML models, processing 10k+ annual tickets for ~$10/year using Claude Haiku (excluding other AWS Service costs). Also, prompts are easier to update than retraining models.
This github repo contains a sample application that connects to Jira Cloud, classifies tickets, and outputs them in a format that can be consumed by your favorite dashboarding tool (Tableu, Quicksight, or any other tool that supports CSVs).
Important Notice: This project deploys resources in your AWS environment using Terraform. You will incur costs for the AWS resources used. Please be aware of the pricing for services like Lambda, Bedrock, Glue, and S3 in your AWS region.
Pre Requisites
You’ll need to have terraform installed and the AWS CLI installed in the environment you want to deploy this code from
The architecture is pretty straight forward. You can find details below.
Step 1: An AWS Lambda function is triggered on a cron job to fetch jira tickets based on a time window. Those tickets are then formatted and pushed to an S3 bucket under the /unprocessed prefix.
Step 2: A Glue job is triggered off /unprocessed object puts. This runs a PySpark deduplication task to ensure no duplicate tickets make their way to the dashboard. The deduplicated tickets are then put to the /staged prefix. This is useful for cases where you manually upload tickets as well as rely on the automatic fetch. If you can ensure no duplicates, you can remove this step.
Step 3: A classification task is kicked off on the new tickets by calling Amazon Bedrock to classify the tickets based on a prompt to a large language model (LLM). After classification, the finished results are pushed to the /processed prefix. From here, you can pick up the processed CSV using any dashboarding tool you’d like that can consume a CSV.
To get started, clone the github repo above and move to the /terraform directory
Run terraform init, plan, & apply. Make sure you have terraform installed on your computer and the AWS CLI configured.
Once the infrastructure is deployed into your account, you can navigate to AWS Secrets Manager and update the secret with your Jira Cloud credentials. You’ll need an API key, base url, and email to enable the automatic pull
And that’s it!
You can (1) wait for the Cron to kick off an automatic fetch, (2) export the tickets to CSV and upload them to the /unprocessed S3 bucket prefix, or (3) manually trigger the Lambda function using a test.
Jira fetch uses a Lambda function with a Cloudwatch cron event to trigger it. The Lambda pulls in the AWS Secret and uses a get request in a while loop to retrieve paginated results until the JQL query completes:
It then creates a string representation of a CSV and uploads it into S3:
An S3 event on the /unprocessed prefix kicks off a second lambda that starts an AWS Glue job. This is useful when there’s multiple entry points that Jira tickets can enter the system through. For example, if you want to do a backfill.
The Glue job itself is written in PySpark and can be found in the code repo here. The important take away is that it does a leftanti join using the issue Ids on the items in the new CSV against all the Ids in the /staged CSVs.
The results are then pushed to the /staged prefix.
This is where it it gets interesting. As it turns out, using prompt engineering can perform on par, if not better, than a text classification model using a couple techniques.
- You can define the classifications and their descriptions in a prompt,
- Ask the model to think step-by-step (Chain of Thought).
- And then output the classification without having to train a single model. See the prompt below:
Note: It’s important to validate your prompt using a human curated subset of classified / labelled tickets. You should run this prompt through the validation dataset to make sure it aligns with how you expect the tickets to be classified
We’ve added a helper class that threads the calls to Bedrock to speed things up:
Lastly, the classified tickets are converted to a CSV and uploaded to S3
The project is dashboard agnostic. Any popular tool/service will work as long as it can consume a CSV. Amazon Quicksight, Tableu or anything in between will do.
In this blog we discussed using Bedrock to automatically classify Jira tickets. These enriched tickets can then be used to create dashboards using various AWS Services or 3P tools. The takeaway, is that classifying text has become much simpler since the adoption of LLMs and what would have taken weeks can now be done in days.
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.