Building an AI-Powered Minecraft Bot with Amazon Bedrock

It's Hackathon month! This November AWS is running a hackathon to build your own game and we jumped in! Let's discover how to add AI to Minecraft!

What is Minecraft?

Minecraft is a phenomenally popular sandbox video game that has captivated players of all ages since its initial release in 2011. Created by Swedish game developer Markus "Notch" Persson and later developed by Mojang Studios, Minecraft offers a unique blend of creativity, exploration, and survival in a blocky, procedurally generated 3D world. At its core, Minecraft is a game about placing and breaking blocks. The game world is composed entirely of cubic blocks representing various materials such as dirt, stone, wood, water, and lava. Players can mine these blocks, collect resources, and use them to craft tools, build structures, and create virtually anything they can imagine.

Ok but... what's the link with Amazon Bedrock?

Besides being a block material in the game (yes bedrock is a block in Minecraft :p) it's also a fully managed service that offers a range of foundation models (FMs) from leading AI companies through a single API. And this is where the magic happens! You can plug a bot in Minecraft with Amazon Bedrock through an API. To achieve this, we are going to leverage the Mineflayer library. This library allows to create bot in the game and send them instructions to make them execute actions. By giving control of the bot to an AI, we will not only be able to discuss with it the way we discuss with an LLM but we will also be able to make it perform actions through agents.

Let's play !

The first step would be to clone the github project. In the project, you can find a CloudFormation template agent_cfn/amazon-bedrock-minecraft-agent-roc.yaml from which you can deploy the agent. Then from your AWS console, you can deploy the agent by uploading the template.

In the Amazon Bedrock/Agents section of you console you will then find the MinecraftAgent deployed. Note the agent ID and alias ID.

Using the agent is local dev and test

From there you can launch Minecraft v1.20.1, start a single player (local) game and in the game, use "Open to LAN" and set the port to 25565.

From the agent_ts/ folder, open a terminal, install the node packages as required npm install.
Update the config, either by setting environment variables, OR editing agent_ts/.env OR by editing agent_ts/config.ts. Set the agentId and the agentAliasId to values output from the agent CloudFormation stack. Feel free to change the name of the bot by changing the MC_USERNAME entry.
Run the TypeScript code in agent_ts/ with npm run start:dev
Here you go! You should be able to play with the Rocky in game by using the in game chat, pressing t. Try "hello", "come to me" and "dig a 2 by 2 hole". Watch the debug output in the terminal you are running from.
For now, the code is running locally. We also want to deploy the logic of the agent.

Installation and Usage - Cloud Hosted (ECS)

This repo also contains a CDK stack that will deploy the agent client code to a task (container) within ECS. You should only consider using this if you are familiar with the technology and running a Minecraft server. Review the code carefully and at a minimum note:

Security Risks: The Minecraft server deployed is publicly accessible, and since the server does not verify usernames, anyone can join with any username, including ones that are already taken by other players. This can lead to impersonation and griefing.
Server Control: Currently the RCON port is not available on the server, so there is no admin access. This will be addressed in the future.

Steps to deploy:

From the cdk/ folder, open a terminal, install the node packages as required npm install.
Deploy the CDK project from cdk/ using context variables agentId and agentAliasId, e.g.:

Usage:

Launch Minecraft v1.20.1.
Select Multiplayer and select Add Server.
Enter the address of the Network LoadBalancer that was deployed by CDK, using port :25565.
Interact with Rocky in game by using the in game chat, pressing t. Try "hello", "come to me" and "dig a 2 by 2 hole". Watch the debug output in the terminal you are running from.

How does this work?

If you have tried a little bit to play with Rocky, you'll notice it's capabilities are limited. That because if you take a closer look at the agent, you will see different sections including Model details and Action groups. In the Model detail you will see the Instruction for the agent. This section is a preprompt sent to to the model along with the player input. This allows us to do a little bit of prompt engineering and dictate how the model should behave.

The action groups section contains all the logic, meaning all the different actions that the bot can perform aside from just talking. For example when clicking on the MinecraftActions action group you will see the different actions that where precoded such as action_dig. For each of the function, you need to specify a description that is what the LLM is going to read to make a decision as to why it should use it or not depending on its interaction with the player.

You can check what each function does in the code agent_ts/src/actions/

If you want to add other capabilities to your bot, feel free to add more functions in here.

As you can see the possibilities are endless, because you can code any functions you would like your bot to perform. Thanks to the description of each of the functions and the built-in capabilities of models to reason, the bot will be able to choose by itself which functions it needs to perform to answer the player the best of its abilities. Only sky is the limit!

If you're ready to build your own game, AWS is running a hackathon through the month of November. From a simple tic tac toe game built with HTML/CSS/JS to a complex Unity game -- all skill levels and tech stacks are welcome. Join now on DevPost: https://awsdevchallenge.devpost.com!

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Site Terms, Privacy, and more.