Live Streaming from Unity - Real-Time Playback (Part 7)

In this series, we've been focusing on broadcasting from a game built in Unity to an Amazon Interactive Video Service (Amazon IVS) real-time stage. But it's not just broadcasting that is possible with Amazon IVS - playback is also possible. In this post, we'll focus on adding real-time playback to the HUD of a Unity game. This capability provides a really unique way to add managed game chat (audio only streams) or team chat (with audio and video) directly into the game experience.

We'll use the same WebRTC package that we used for broadcasting for playback, and we'll also require a stage token for playback, so if you've not yet read part 2 in this series, now would be a great time to do that. The main difference for playback is that we'll need to render the incoming frames to the UI, and we'll need to modify the URL that we use to connect based on the contents of the stage token. We'll also need to know the participantId of the stream that we'd like to subscribe to, so we'll need to construct a way to obtain that. Let's start by getting that participantId.

We'll use the Amazon IVS chat integration that we learned about in part 4 in a different way than we've previously seen. This time we'll use the chat WebSocket connection as a message bus so that we can be notified when another participant's video is available to display.

We're going to walk through the various elements in the WebRTCPlayback script below, but you can refer to the final script as a reference.

Each participant that connects to an Amazon IVS stage is assigned a participantId, and we can use the Amazon IVS integration with EventBridge to get notified when another participant has joined the stage. For this, I've created an AWS Lambda function that is triggered by an EventBridge rule filtered to look for events with the detail-type of IVS Stage Update with the an event_name of Participant Published or Participant Unpublished. This rule will trigger the UnityParticipantUpdated function and here is the SAM yaml used to create the rule.

The UnityParticipantUpdated function is also defined in yaml. This function needs two variables, the UNITY_CHAT_ARN that we'll need to send a message to the game via the WebSocket connection, and the UNITY_STAGE_ARN to make sure that we're only notifying the message bus when a participant has joined/left the specific Amazon IVS stage that we're interested in.

The event that the AWS Lambda function will receive will have the following format:

The function will check the stage ARN, and if it matches it will utilize the SendEvent (docs) method of the IvsChatClient to send a custom event with the name STAGE_PARTICIPANT_UPDATED to the chat room.

Another (easier, but less "dynamic") way to list stage participants would be to create an AWS Lambda function to list the participants (see ListStageParticipantsCommand). This method would need to be refreshed occasionally as participants enter and leave the stage.

For this demo, we'll add a Raw Image in the FPS demo game's HUD that we'll ultimately use to render the live stream. We'll add a child Audio Source as well for the live stream audio playback.

We'll add a script called WebRTCPlayback to the Raw Image to handle listening for participants and rendering the video.

We'll set up a chat connection (see part 4) to listen for events in our WebRTCPlayback script. When we receive the publish event that our AWS Lambda function publishes, we'll establish the peerConnection and connect the live stream for playback. If the event is an 'unpublish' event, we'll clear the render texture and dispose of the peerConnection.

In previous demos, we didn't need to parse the JWT stage token at all - we just passed it along when we established the connection. But for playback, we'll need to get the whip_url from the token and use that to get our SDP. Let's create a class to model the stage token.

Now we can decode and parse the token by adding the following to our GetStageToken() function.

We'll declare a variable in our WebRTCPlayback script for the RawImage that we'll use to render the video.

The EstablishPeerConnection() function renders the live stream to the RawImage by setting the texture of the receiveImage every time a new frame is received

Finally, in DoWhip() we use the participantId and the whipUrl from the StageToken to construct the URL used to obtain the SDP.

To test out our playback, we can create a page to broadcast to the Amazon IVS page using the Amazon IVS Web Broadcast SDK, or use this CodePen demo with a manually generated token. Launch the game, then connect to the stage and the remote participant will be rendered in the HUD.

In this post, we learned how to add real-time live stream playback directly inside of our Unity built game.