Unleashing CRUD Prowess: OpenSearch and Golang, a Dynamic Duo

If you ever need to build an application for intensive searching of data, use a database that is optimized for such. OpenSearch is a great option for this. Its a real-time distributed search and analytics engine that provides full-text search capabilities and scales horizontally, while still providing sub-second latency for results. OpenSearch is an open-source project available under Apache License 2.0. You can work with OpenSearch locally, using distributions like Tarball, RPM, Debian, Windows, Docker, Helm, and Ansible playbooks. Alternatively, you can use OpenSearch with Amazon OpenSearch, a fully managed service for OpenSearch that allows you to focus on building solutions while the infrastructure and the operations are taken care for you.

In this tutorial, you will learn how to build an CRUD application for OpenSearch using the Go programming language. This blog post was created from the example of a Terraform provider found in this code repository. If you want a complete example of an application that uses Go to connect with OpenSearch, this repository will provide you with that.

The first thing you need to do to use OpenSearch with Go is importing the Go client for OpenSearch and the AWS SDK for Go if you intend to work with Amazon OpenSearch. You can do this either manually or using the tools provided by your IDE.

As a developer, the most important thing to understand about OpenSearch is all its operations are available to the world via REST APIs. This is true for Go and any other programming language. For this reason, your code must have a client connection with the cluster.

In Go, use the opensearch.NewClient() function to create a new client.

This function expects a single parameter containing the configuration necessary to establish the HTTP connection with OpenSearch. With the opensearch.Config struct you can define this configuration. It supports many fields that you can use to access different OpenSearch deployments. The most important field is the Addresses array, which expects you to inform one or multiple endpoints that can establish the connection. All the other fields are optional, and you may use them when it makes sense.

For instance, the code shown above is enforcing SSL verification, which is why a value was provided for the field Transport. Similarly, it provides a username and a password using the fields Username and Password, to authenticate with the cluster exposed at the provided endpoint. This approach may work well if your cluster is running on-premises, or if your Amazon OpenSearch cluster is configured to use the master database for authentication.

However, if you have configured Amazon OpenSearch to use IAM authentication, you are going to need a different approach to create the client.

This approach is slightly more elaborated than the previous one, but it remains semantically the same. This is the key difference:

When using IAM for authentication, instead of providing credentials for the OpenSearch cluster, you provide the credentials of an IAM user allowed to connect with OpenSearch using a signer. A signer encapsulates everything you need to connect with the service, such as your AWS access key, secret access key, and token. Once everything is set into the signer, you can authenticate with Amazon OpenSearch.

Before using the client to send requests to OpenSearch, it is a good idea to always check if the connection is valid first. Doing this will save you from connectivity headaches, hours of debugging, and improve your code robustness.

Note that we have used http.StatusOK from Go to compare the status of the ping request. As mentioned earlier, OpenSearch makes all its operations available to the world via REST APIs. Therefore, if you need to check if the connection has been successfully established, you will check if the status is 200. Similarly, if you need to check if the authentication failed, you will check if the status is equal to 401 or 403, for example.

Now that you have a valid client to work with, you can use this client to index documents. In OpenSearch, the operation of writing data is called indexing, and the data itself is called documents. Let's say you want to index the following document:

You would need a struct to represent this document in your code.

Along with the fields of the document, it was included an additional field called ID in the struct. This will hold the document identifier, which is a string that uniquely identifies each document on OpenSearch. This is how you are going to index documents.

Similar to checking if the connection is valid, indexing a document requires a request and response interaction with OpenSearch. Once a request is executed, a response will be replied and it is up to you to process it. Here, we will process the response, as OpenSearch always returns the document identifier of the newly indexed document. To fully process the response, we need a struct that represents the response.

Now you can simply:

If you have the document identifier of a specific document, you don't need to search for the document. Instead, you can simply execute a look up. Given the documentID value retrieved earlier, here is how you can look up the document and write the value of the FullName field to a variable called fullName.

To update an existing document, you will need the document identifier of the document. Then you will need to provide a new document containing only the fields whose values you want to update. Here is how we are going to update the KnownAs field of the Deadpool character to Weapon XI.

Different from the create and read operations, you don't need to process the response returned by OpenSearch. If there weren't any errors returned, you can be sure that the document is going to be updated. Moreover, it is important to understand that OpenSearch applies eventual consistency for updates.

Internally, the old document is marked as deleted and a new document is added. The old version of the document doesn't disappear immediately, but you won't be able to access it. OpenSearch cleans up deleted documents in the background as you continue to index more data.

The process to delete an existing document is fairly simple, as the only thing needed is the document identifier. Here is how you can delete the document containing the Deadpool character.

Once OpenSearch receives this request, it will mark the existing document as deleted and a background process will delete it, eventually.

A database such as OpenSearch that is optimized for intensive searching of data is not very useful if you don't send any searches for it, right? At this point, you have learned how to perform CRUD operations but not how to do searches. The good news is that the process is similar to CRUD operations. It requires a request and response interaction with OpenSearch.

For instance, let's say you want to search for all the characters given their identity. To write this search, you need to leverage the Query DSL feature, and create a search body using one of the supported full-text queries from OpenSearch.

To illustrate this, let's say you want to search for a character whose identity is Matt Murdock. If anything is found, you will retrieve the first document returned and read the FullName field into a variable called fullName.

The first step is to create a query of type Match. It allows you to perform full-text search using a specific document field. In this case, the field Identity. In Query DSL, this is how you would implement this query:

In Go, this is how you are going to implement this query:

With the query created, you can use it to perform a search request:

Since you need to process the response as you are interested in the results of the search, you will need a struct in your code that represents the result returned by OpenSearch:

Finally, this is how you are going to read the results:

If you are acquainted with the Marvel universe, you know that any searches for someone whose identity is Matt Murdock will return the character Daredevil. Also, did you notice that it's the t-shirt I am wearing in the banner of this blog post? If caught that, add a comment below.

Now that you learned how to implement searches using the Query DSL, let's take one step further and implement the same query before but now using SQL. OpenSearch provides native support for SQL, which can come in handy if you are new to OpenSearch but have a previous background with SQL databases.

To implement the previous query using SQL, you need to execute the following statement:

Here, you are searching for the character Matt Murdock in the build-on-aws index, which is the index where all the documents have been written. OpenSearch allows you to execute this SQL query using a plugin. Therefore, you need to write a code that will send an SQL query request to be executed in this plugin.

There is no high-level API in the Go client for OpenSearch that allows you to interact with the plugins. However, you can send virtually any HTTP request to the client created before and, if the request is well constructed, you will get the same results as if you were using a high-level API. Here is how you can do this.

First, you need to create a body that will contain your SQL statement. Then, this body must be serialized into a io.Reader just like you have done in the previous version of the search implementation.

Next, you need to create a valid HTTP request and execute using the client, which acts as the transform layer for your HTTP request.

If no errors are returned, you can process the response just like you did before, even using the same struct that provides a representation of what OpenSearch sends you back.

Follow me on LinkedIn if you want to geek out about technologies like OpenSearch, Golang, and many others.

See you, next time!