How to use object mapping in AWS DMS for AWS Kinesis?

Hi there,

We're planning to configure CDC for some of our tables in PostgreSQL: RDS -> DMS (ongoing replication) -> Kinesis.

Everything works fine, we see messages about data changes in the Kinesis stream, but we want to change the data format.

Now we receive data in the following format:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
    "data": {
        "id": ...,
        "result": ...,
        "black_elo": ...,
        "white_elo": ...,
        "opening_name": ...,
        ...
    },
    "metadata": {
        "timestamp": "2024-06-24T19:08:53.329671Z",
        "record-type": "data",
        ...
    }
}

For our purposes, we need to change the format so that we only get the fields from the "data" key as a result:

1
2
3
4
5
6
7
8
{
    "id": ...,
    "result": ...,
    "black_elo": ...,
    "white_elo": ...,
    "opening_name": ...,
    ...
}

We configured the rule according to this guide: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Kinesis.html#CHAP_Target.Kinesis.ObjectMapping

Here's the JSON we use for the Table mappings in the DMS task:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
{
    "rules": [
        {
            "rule-type": "selection",
            "rule-id": "1",
            "rule-name": "1",
            "rule-action": "include",
            "object-locator": {
              "schema-name": "public",
              "table-name": "gaming_onlineboard"
            }
        },
        {
            "rule-type": "object-mapping",
            "rule-id": "2",
            "rule-name": "DefaultMapToKinesis",
            "rule-action": "map-record-to-record",
            "object-locator": {
                "schema-name": "public",
                "table-name": "gaming_onlineboard"
            },
            "mapping-parameters": {
                "exclude-columns": [
                  "moves"
                ]
            }
        }
    ]
}

It looks like this is exactly what we need, but we see data in the stream in the same format.

We completely recreated the stream, endpoint, and migration task. The result remained the same.

Can someone please explain where our error is?

Does the "object-mapping" rule type even work? This type is not available in the console when setting up Table mappings but the "exclude-columns" part works and we don't see the "moves" field in the Kinesis stream.

If this doesn't work, what other options can we use here: a lambda function, AWS Glue, or something else?

Thanks,

Dmitry

Select your cookie preferences

Site Terms, Privacy, and more.

How to use object mapping in AWS DMS for AWS Kinesis?

Using the "object-mapping" rule has no effect

Comments