Testing DynamoDB Interactions using LocalStack and Testcontainers

Overview

Testing the data layer and datasource interactions is very crucial when developing a backend application. In this article, we’ll explore on how we can write Integration Tests for our Java Spring-boot application which interacts with AWS DynamoDB for storing and retrieving records.

We'll be creating an example data model and a repository class which performs basic CRUD operations on the provisioned DynamoDB Table. Then we'll be looking at how we can write Integration tests for our repository using LocalStack and Testcontainers where we'll be creating an In-memory DynamoDB Table inside a disposable docker container and making our application connect to it while running the tests.

With this setup, we'll be ensuring a controlled and isolated environment for testing our DynamoDB interactions. The code referenced in the article can be found in this Repository.

Contextualizing Keywords

Spring Boot : is a popular framework for building Java-based, production-ready services with minimal configuration.
AWS DynamoDB : is a fully managed NoSQL database service provided by AWS and we'll be using it as the datasource that our backend application connects to.
LocalStack : is a cloud service emulator that enables development and testing of AWS services without connecting to a remote cloud provider. We'll be provisioning an in-memory DynamoDB table inside this emulator.
Testcontainers : is a library that provides lightweight, throwaway instances of Docker containers for integration testing.

AWS Configuration

In order to facilitate connection between the application and DynamoDB table, we'll need to define beans of AmazonDynamoDB and DynamoDBMapper in a configuration class.

When connecting to an actual provisioned DynamoDB table in AWS, we don't need to specify an endpoint URL since it automatically uses the default endpoint for each service in the configured AWS Region. But when connecting to the Table created inside the LocalStack container, we'll be needing to define an endpoint URL as well.

To solve the above issue, we'll be using Spring Profiles to define seperate beans of class AmazonDynamoDB, that'll get injected into the application context depending on whether the application is running in test or non-test profiles.

Additionaly we'll make use of @ConfigurationProperties to map the values defined in the active .yaml or .properties file to a POJO that'll be referenced by our configuration class.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
@Getter
@Setter
@Validated
@ConfigurationProperties(prefix = "com.behl.aws")
public class AwsConfigurationProperties {

	@NotBlank(message = "AWS access key must be configured")
	private String accessKey;

	@NotBlank(message = "AWS secret access key must be configured")
	private String secretAccessKey;

	@NotBlank(message = "AWS region must be configured")
	private String region;

	private String endpoint;

}

As mentioned above, The endpoint field is optional and is needed to be defined only for testing and local development when using LocalStack.

Below is a snippet of our application.yaml file where we've defined the required properties which will be automatically mapped to the above defined class, along with the current active profile that we'll be using in our bean configuration class.

1
2
3
4
5
6
7
8
9
10
11
spring:
  profiles:
    active: ${ACTIVE_PROFILE}

com:
  behl:
    aws:
      access-key: ${AWS_ACCESS_KEY_ID}
      secret-access-key: ${AWS_SECRET_ACCESS_KEY}
      region: ${AWS_REGION}
      endpoint: ${AWS_ENDPOINT_URL}

Now let's create our AwsConfiguration class that defines the beans required for connecting to DynamoDB depending upon the active profile.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
@Configuration
@RequiredArgsConstructor
@EnableConfigurationProperties(value = AwsConfigurationProperties.class)
public class AwsConfiguration {

	private final AwsConfigurationProperties awsConfigurationProperties;

	@Bean
	@Profile("!test && !local")
	public AmazonDynamoDB amazonDynamoDB() {
		var credentials = constructCredentials();
		var regionName = awsConfigurationProperties.getRegion();
		return AmazonDynamoDBClientBuilder.standard()
				.withRegion(Regions.fromName(regionName))
				.withCredentials(credentials)
				.build();
	}
	
	@Bean
	@Profile("test | local")
	public AmazonDynamoDB testAmazonDynamoDb() {
		var credentials = constructCredentials();
		var regionName = awsConfigurationProperties.getRegion();
		var endpointUri = awsConfigurationProperties.getEndpoint();
		var endpointConfig = new EndpointConfiguration(endpointUri, regionName);
		return AmazonDynamoDBClientBuilder.standard()
				.withEndpointConfiguration(endpointConfig)
				.withCredentials(credentials)
				.build();
	}

	@Bean
	public DynamoDBMapper dynamoDBMapper(AmazonDynamoDB amazonDynamoDB) {
		return new DynamoDBMapper(amazonDynamoDB);
	}

	private AWSStaticCredentialsProvider constructCredentials() {
		var accessKey = awsConfigurationProperties.getAccessKey();
		var secretAccessKey = awsConfigurationProperties.getSecretAccessKey();
		var basicAwsCredentials = new BasicAWSCredentials(accessKey, secretAccessKey);
		return new AWSStaticCredentialsProvider(basicAwsCredentials);
	}
	
}

The bean specifying the endpoint URL for connecting to the LocalStack container will be created only if the active profile is test or local. With this, we've successfully created the correct AWS DynamoDB configuration for our application.

Data Model and Repository Layer

We'll be creating an entity class MedicalRecord to represent the data stored in DynamoDB with the below defined attributes. We're leveraging lombok for generating getter and setter methods for the attribute to reduce boilerplate code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
@Getter
@Setter
@DynamoDBTable(tableName = "MedicalRecords")
public class MedicalRecord {

	@DynamoDBHashKey
	@DynamoDBAttribute(attributeName = "Id")
	private String id;

	@DynamoDBAttribute(attributeName = "PatientName")
	private String patientName;

	@DynamoDBAttribute(attributeName = "Diagnosis")
	private String diagnosis;

	@DynamoDBAttribute(attributeName = "TreatmentPlan")
	private String treatmentPlan;

	@DynamoDBAttribute(attributeName = "AttendingPhysicianName")
	private String attendingPhysicianName;

}

To perform basic save and retrieve operations on the above defined entity, we'll be creating it's corresponding Respository class. The earlier defined bean of DynamoDBMapper will be autowired into this class via Constructor Injection with Lombok.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
@Repository
@RequiredArgsConstructor
public class MedicalRecordRepository {

	private final DynamoDBMapper dynamoDBMapper;

	public void save(@NonNull MedicalRecord medicalRecord) {
		dynamoDBMapper.save(medicalRecord);
	}

	public Optional<MedicalRecord> findById(@NonNull String medicalRecordId) {
		var medicalRecord = dynamoDBMapper.load(MedicalRecord.class, medicalRecordId);
		return Optional.ofNullable(medicalRecord);
	}

}

Testing

Now that the context is established, we'll be moving on to the main agenda of this article: To test the data layer's interaction with DynamoDB.

Dependencies

Let’s start by declaring the required test dependencies in pom.xml:

1
2
3
4
5
6
7
8
9
10
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>localstack</artifactId>
    <scope>test</scope>
</dependency>

Prerequisite: Running Docker

The prerequisite for running a LocalStack container via Testcontainers is, as you've guessed it, an up and running Docker instance. We need to ensure this, when running the Test suite either locally or when using a CI/CD pipeline.

When using Github actions, The default Ubuntu runners come pre-equipped with Docker, eliminating the necessity for manual Docker setup procedures.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up JDK
        uses: actions/setup-java@v3
        with:
          java-version: 21
          distribution: adopt
          cache: maven

      - name: Run Integration Tests
        run: mvn integration-test verify

Creating DynamoDB Table using Init Hooks

Localstack gives us the ability to create resources when the container is started via Initialization Hooks. We'll be creating our init-dynamodb.sh script in the src/test/resources folder which we want to be executed when the LocalStack container is started.

The script creates a DynamoDB table MedicalRecords for which we've already created a model class in our application.

1
2
3
4
5
6
7
8
9
10
11
12
#!/bin/bash
table_name="MedicalRecords"
hash_key="Id"

awslocal dynamodb create-table \
    --table-name "$table_name" \
    --key-schema AttributeName="$hash_key",KeyType=HASH \
    --attribute-definitions AttributeName="$hash_key",AttributeType=S \
    --billing-mode PAY_PER_REQUEST

echo "DynamoDB table '$table_name' created successfully with hash key '$hash_key'"
echo "Executed init-dynamodb.sh"

Now in our Integration Test class, we need to do the following before we start to write any test cases:

Activate the test profile using @ActiveProfiles
Copy our init-dynamodb.sh script into the container to ensure DynamoDB table creation.
Configure a Wait strategy to wait for the log Executed init-dynamodb.sh to be printed, as defined in our init script.
Dynamically configure the AWS configuration properties needed by the application in order to create the required connection beans using @DynamicPropertySource.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
@SpringBootTest
@ActiveProfiles("test")
class MedicalRecordRepositoryIT {

	@Autowired
	private MedicalRecordRepository medicalRecordRepository;

	private static LocalStackContainer localStackContainer;

	static {
		localStackContainer = new LocalStackContainer(DockerImageName.parse("localstack/localstack:3.2"))
				.withCopyFileToContainer(MountableFile.forClasspathResource("init-dynamodb.sh", 0744), "/etc/localstack/init/ready.d/init-dynamodb.sh")
				.withServices(Service.DYNAMODB)
				.waitingFor(Wait.forLogMessage(".*Executed init-dynamodb.sh.*", 1));
		localStackContainer.start();
	}

	@DynamicPropertySource
	static void properties(DynamicPropertyRegistry registry) {
		registry.add("com.behl.aws.access-key", localStackContainer::getAccessKey);
		registry.add("com.behl.aws.secret-access-key", localStackContainer::getSecretKey);
		registry.add("com.behl.aws.region", localStackContainer::getRegion);
		registry.add("com.behl.aws.endpoint", localStackContainer::getEndpoint);
	}

}

Now once we're done configuring the Localstack container, we can test the MedicalRecordRepository class with appropriate test cases:

1
2
3
4
5
6
7
8
9
10
11
@Test
void shouldReturnEmptyOptionalForNonExistingMedicalRecord() {
    // generate random medical record Id
    var medicalRecordId = RandomString.make();

    // call method under test
    Optional<MedicalRecord> medicalRecord = medicalRecordRepository.findById(medicalRecordId);

    // assert response
    assertThat(medicalRecord).isEmpty();
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
@Test
void shouldSaveMedicalRecordSuccessfully() {
    // create medical record
    var medicalRecord = new MedicalRecord();
    var medicalRecordId = RandomString.make();
    var patientName = RandomString.make();
    var diagnosis = RandomString.make();
    medicalRecord.setId(medicalRecordId);
    medicalRecord.setPatientName(patientName);
    medicalRecord.setDiagnosis(diagnosis);

    // call method under test
    medicalRecordRepository.save(medicalRecord);

    // assert record existence in datasource
    Optional<MedicalRecord> retrievedMedicalRecord = medicalRecordRepository.findById(medicalRecordId);
    assertThat(retrievedMedicalRecord).isPresent().get().satisfies(record -> {
        assertThat(record.getId()).isEqualTo(medicalRecordId);
        assertThat(record.getPatientName()).isEqualTo(patientName);
        assertThat(record.getDiagnosis()).isEqualTo(diagnosis);
    });
}

And with the above, we've successfully tested our repository class that interacts with DynamoDB table for storing and retrieving records.

The created LocalStack containers will be destroyed automatically once the Test Suite is completed, hence we do not need to worry about cleanup.

Pro-tip: Reduce Boilerplate Testcontainers Code

In our Integration Testing setup, we encounter repetitive boilerplate code for initializing and configuring the LocalStack container, as well as dynamically adding properties to it. These steps will be repeated in every Test class that tests a functionality related with the data layer.

This duplication clutters our Test classes and we'll be looking at a possible solution for this issue using the lifecycle callback extension available in JUnit 5 called @BeforeEachCallback.

Inside our test folder itself, we'll be creating a class DynamoDBInitializer which implements the BeforeEachCallback interface and performs the boilerplate functionality.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
@Slf4j
public class DynamoDBInitializer implements BeforeAllCallback {

	private static final DockerImageName LOCALSTACK_IMAGE = DockerImageName.parse("localstack/localstack:3");
	
	private static final LocalStackContainer localStackContainer = new LocalStackContainer(LOCALSTACK_IMAGE)
			.withCopyFileToContainer(MountableFile.forClasspathResource("init-dynamodb.sh", 0744), "/etc/localstack/init/ready.d/init-dynamodb.sh")
			.withServices(Service.DYNAMODB)
			.waitingFor(Wait.forLogMessage(".*Executed init-dynamodb.sh.*", 1));

	@Override
	public void beforeAll(final ExtensionContext context) {
		log.info("Creating localstack container : {}", LOCALSTACK_IMAGE);
		
		localStackContainer.start();
		addConfigurationProperties();
		
		log.info("Successfully started localstack container : {}", LOCALSTACK_IMAGE);
	}

	private void addConfigurationProperties() {
		System.setProperty("com.behl.aws.access-key", localStackContainer.getAccessKey());
		System.setProperty("com.behl.aws.secret-access-key", localStackContainer.getSecretKey());
		System.setProperty("com.behl.aws.region", localStackContainer.getRegion());
		System.setProperty("com.behl.aws.endpoint", localStackContainer.getEndpoint().toString());
	}

}

Then, we'll be creating a custom Marker Annotation @InitializeDynamoDB that registers the above created Extension class using @ExtendWith.

1
2
3
4
5
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@ExtendWith(DynamoDBInitializer.class)
public @interface InitializeDynamoDB {
}

Now, we can annotate all of our Integration Test classes which require DynamoDB Table creation to test functionalities with our custom annotation @InitializeDynamoDB.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
@SpringBootTest
@InitializeDynamoDB
@ActiveProfiles("test")
class MedicalRecordRepositoryIT {

	@Autowired
	private MedicalRecordRepository medicalRecordRepository;

	@Test
	void shouldReturnEmptyOptionalForNonExistingMedicalRecord() {
    	// test case
	}

    @Test
    void shouldSaveMedicalRecordSuccessfully(){
        // test case
    }

}

NOTE: To have a single shared container that is used by all Integration Test classes, we've implemented the BeforeAllCallback interface. This approach will reduce test execution time significantly and eliminate the need to start and stop containers for each test class.
For cases where this behaviour is not needed, BeforeEachCallback interface can be implemented instead.

Conclusion

Through this article, we've explored on how we can create an isolated testing environment with LocalStack and Testcontainers to write Integration Tests for our data layer, specifically for one that interacts with AWS DynamoDB Database.

Using what we've learnt, we can confidently write integration tests without relying on external resources or impacting production data.

Site Terms, Privacy, and more.