SGLang on ECS: efficiently serving leading OSS LLMs on AWS
Benefits of SGLang as a runtime environment for OSS LLM. Sharing a Docker image to run it on ECS. Reasoning inference examples with Qwen QwQ 32-B.
Didier Durand
Amazon Employee
Published Mar 10, 2025
The SGLang project (GitHub repository) is an open-source throughput-optimized and fast-serving framework for Large Language Models (LLMs) and vision language models. The associated scientific research was started with the goal of creating a new Structured Generation Language (hence the name) for modern agentic applications with sophisticated multi-prompt strategies. Its main goal is to increase inference throughput of underlying hardware to obtain a higher cost efficiency.
From my perspective, it has also the more tactical but major potential of becoming the unifying runtime layer for productive inferences of most leading OSS LLMs. Using this framework, we can already today currently run a large (and always increasing... ) set of leading model families (I personally tested various distillations of DeepSeek as well as models like Mistral, Meta’s Llama, Alibaba’s Qwen, IBM’s Granite in various flavors and sizes). For example, in our last trial, the brand new and market leading (as of this writing: March, 9th 2025) Alibaba’s Qwen-with-Questions (QwQ-32B) could start on our AWS ECS cluster with no additional effort. It was ideal as we wanted to quickly obtain an initial evaluation because QwQ-32B is presented as equivalent to DeepSeek R1 (671B parameters) with 20x less parameters! As such, QwQ-32B can be executed on less-demanding hardware with the Docker image that we describe in next section. It runs on an EC2 instance of type g6.12xlarge, featuring 4 x NVIDIA L4 Tensor Core currently charged at $4.6/h instead of requiring an instance of type p5.48xlarge, currently charged $31.5/h, with 8 x Nvidia H100 GPUs to have enough memory when you load the 671B parameters of DeepSeek R1.
Given the frenzy around new models announced weekly by the different providers, using SGLang as common underlying layer to run them all is a huge time saver: basically, you download the model from Hugging Face (see our tool for that), change the model name in your start command, optionally update a few start parameters, and off you go: you can benchmark the quality and performances of the new model right away! "Agility is the only sustainable advantage" (J. Bezos): by making the recurring transition from an older models to the new leaders so seamless, SGLang allows an organization to remain fully agile. It will prove critical for the next few years as better models pop up constantly!
SGLang is mostly composed of a front-end language receiving the prompts and a runtime running the LLM and driving its inferences. The runtime accelerates the inference with novel optimizations like RadixAttention for maximum KV cache reuse. The typical example are system prompts which are often identical for many (all ?) of an agentic applications. The request scheduler detects these common parts of inference requests and schedule them concurrently to maximize hit rach in the KV cache. The KV Cache is a critical piece for compute optimization: it allows to mutate the quadratic complexity of transformers into a linear one (of course, at the expense of the use of additional memory). SGLang also provides compressed finite state machines for faster structured output decoding. Again think here about agentic applications requesting structured answers in JSON, YAML or markdown format from the LLM. For all details, please, consult the research paper of the project on Arxiv.
The SGLang project also made some smart tactical moves to maximize adoption by application developers. For example, the front-end provides an inference endpoint that is 100% compatible with the OpenAI endpoint for ChatGPT. You just import the openai client library and code against it as if you were talking to your favorite OpenAI model. As such, you can fully reuse your existing ChatGPT Python code unchanged to drive inferences on the OSS LLM that you host autonomously in your system. See second part of attached code for an example and test_qwen_qwq_32b.py for some test code.
Finally, the core feature of SGLang is the specific inference description language that was created. It is based on the fact that recent LLM-based applications (again think agentic applications) usually need to do several related queries to the LLM in short timeframes. Those queries will lead to same computations - because of prompts with similar text chunks - by the LLM (attention heads, etc.). When the requester submits them in one batch with a description of their interdependencies (parallel vs sequential), the SGLang endpoint can activate the SGLang request scheduler with the opportunity to compute the corresponding sequence of execution in an optimal way from a performance perspective. The scheduler can run them quasi simultaneously or very next to each other in an order computed to maximize the re-use of the KV cache entries across distinct inferences.
For example, parallelism is possible across the submitted inferences is possible, it is coded like this in Python via the fork primitive provided by the Structured Language
This kind of optimization will definitely provide lots of cost-efficiency to the army of agentic applications being currently built: this kind of smart inference scheduling will allow those applications to make best use of the compute capacity at hand for the gazillions of inferences that they require.
All the optimization mechanisms described above lead in optimal use cases to accelerations up to 6.4x times according to the research paper. Of course, since those mechanisms result in less computations, they also lead to increased cost-efficiency: they allow more inferences with same amount of hardware resources.
We thoroughly use this image based on our Dockerfile on AWS Elastic Container Service (ECS) for multiple models that SGLang support. It can also be used on any local host system or laptop equipped with some Nvidia GPU(s).
If you don’t want to build the image yourself, you can pull it for direct execution from our Docker hub repository:
docker pull didierdurand/lic-sglang:amzn2023-latest The image that we publish is built directly on GitHub via this GitHub Action. You can see corresponding executions on this pageKey aspects:
- The image is based on Amazon Linux 2023. But, it can be adapted to other flavors of Linux: Ubuntu, RedHat, etc.
- The SGLang project is still in activedevelopment with some features or parameters yet to be added. So, we copy into the image at build time a bash shell customize_sglang.sh that allows for customization. For example, as of this writing, we update some http timeout parameter in the source code via
sed 's/timeout_keep_alive=5,/timeout_keep_alive=500,/' -i $FILEfor file forFILE='/usr/local/lib/python3.12/site-packages/sglang/srt/entrypoints/http_server.py. You can add your own customizations in this shell. - We also copy into the image a start_sglang.sh to dynamically build the start SGLang command from env variables received from the
docker runcommand. Different models have different requirements for the various parameters proposed by SGLang. It allows to keep the launch parameters external to the image: the same Docker image can be used for multiple LLMs. - It is unsustainable to include the weights of the LLM into the image: they are most often too big (60B+ for QwQ-32B for example) and they would tie the image to a specific LLM. The latency of a live fetch is too long for live pull from HuggingFace at each start. So, we use a Docker Bind Mounts at container start to link the
/home/modeldirectory of the image to an external directory of the home server, where the model weights are stored, in our fast on a fast AWS EBS volume. - We define multiple Docker ENV variables to collect parameters required to issue the right start command for SGLang. Those variables will be populated via
--envoption. (doc for details) - the final
|| sleep infinityis a trick to keep the container up & running even if the SGLang start command fails for any reason. It allows to connect to the container viadocker exect -it <container-id> /bin/bashto debug the problem.
A copy of the Dockerfile is included here for readability purposes:
The following messages are those that are obtained when starting SGLang with Qwen QwQ-32B deployed in an ECS cluster on an EC2 instance of type g6.12xlarge EC2, featuring 4 x NVIDIA L4 Tensor Core GPUs with 24GB of RAM per GPU.
The 4 messages with each
Load weight end. type=Qwen2ForCausalLM, dtype=torch.bfloat16, avail mem=6.09 GB, mem usage=15.54 GB. show that the total model (60B+) is loaded in 4 equal parts on each GPU.The examples below are based on the code of test_qwen_qwq_32b.py
Inference #1:
The question is the hot one of Fall 2024 failing with most major leading LLM:
How many letters R in word 'strawberry' ?Response: (The response is solidly supported by all reasoning explained - maybe a bit too much...)
Okay, so I need to figure out how many times the letter R appears in the word "strawberry". Let me start by writing out the word letter by letter to make it easier. S-T-R-A-W-B-E-R-R-Y. Wait, let me check again to make sure I'm not missing any letters or adding extra ones. Let's count each letter one by one.
Starting with the first letter: S. Then T, then R. That's the first R. Next is A, W, B, E, and then another R. So that's the second R. Then another R? Let me see: after E comes R again, so that's the third R? Wait, no, let me go through the word again carefully. S (1), T (2), R (3), A (4), W (5), B (6), E (7), R (8), R (9), Y (10). Hmm, so after E (7th letter) comes two R's? So that would be the 8th and 9th letters, making a total of three R's? Wait, but I might be miscounting. Let me write it down step by step:
Breaking down "strawberry":
- S
- T
- R
- A
- W
- B
- E
- R
- R
- Y
Yes, so positions 3, 8, and 9 are R's. That's three R's. Wait, but sometimes people might pronounce it differently or maybe I'm spelling it wrong? Let me confirm the spelling of "strawberry". S-T-R-A-W-B-E-R-R-Y. Yes, that's correct. So there are three R's. Wait another thought: maybe I miscounted the letters. Let me count again:
S (1), T(2), R(3), A(4), W(5), B(6), E(7), R(8), R(9), Y(10). Yep, three R's. So the answer should be 2? Wait, no, that's three. Wait, maybe I'm overcounting. Let me list them:
Looking at each letter:
- The third letter is R.
- Then after E (7th), the next two letters are R and R. So that's two more. So total 1 + 2 = 3. So three R's. Hmm, but I've heard sometimes people might think there are two. Maybe a common mistake? Let me check again. Let me spell it slowly: S-T-R-A-W-B-E-R-R-Y. So after the B and E comes two R's in a row, then Y. So that's two R's after the E, plus the one after T. So total three. Therefore, the answer is 2? Wait no, three. Wait, maybe I'm confused. Let me count the Rs only:
Looking at each letter and pick out the R's:
- S – no
- T – no
- R – yes (1)
- A – no
- W – no
- B – no
- E – no
- R – yes (2)
- R – yes (3)
- Y – no
So that's three R's. Therefore, the correct answer is 2? Wait no, three. Wait, maybe I made a mistake here. Let me check an online source or just confirm the spelling again. Wait, the user is asking about the word 'strawberry'. Let me type it out: s-t-r-a-w-b-e-r-r-y. Yes, that's 10 letters. The Rs are at positions 3, 8, 9. So three Rs. Therefore, the answer should be 2? Wait, no, three. Wait, maybe I'm overcomplicating. Let me just list all letters with their positions again:
- S
- T
- R
- A
- W
- B
- E
- R
- R
- Y
Yes, three Rs. So the answer is 3. But I'm a bit confused because sometimes people might think there are two. Maybe I should double-check. Alternatively, perhaps I miscounted the letters. Let me count the letters again: S (1), T(2), R(3), A(4), W(5), B(6), E(7), R(8), R(9), Y(10). That's 10 letters. So three Rs. Therefore, the answer is 3. Wait, but maybe the user is a trick question? Like maybe in some other language or a different spelling? But the question is in English, so 'strawberry' is spelled with three Rs. So I think the answer is 2? No, no, three. Wait, I'm getting confused. Let me try another approach. Let's write the word and circle the Rs:
S T R A W B E R R Y
So the first R is after T, then after E there are two Rs. So that's three. So the answer must be 2? No, three. I think I'm just second-guessing myself, but the correct count is three. Therefore, the number of Rs in 'strawberry' is 2? Wait, no, three. I need to stop overthinking. The correct answer is 2? No, three. Wait, maybe I should just confirm once more. Let me look up the spelling of strawberry. [Imagining looking it up] Yes, strawberry is spelled S-T-R-A-W-B-E-R-R-Y. So three Rs. Therefore, the answer is 3.
The word 'strawberry' is spelled S-T-R-A-W-B-E-R-R-Y. Counting the Rs: positions 3, 8, and 9. Total: 3.
(If you wonder that the result is biased because data training contained this famous question and some answer, be reassured. We verified it with random sequence of letters (i.e. non-existing words) , Qwen QwQ-32 did similar reasoning and got right answer each time.)
Inference #2:
Now, some more complex question:
A man has 53 socks in his drawer: 21 identical blue, 15 identical black and 17 identical red. The lights are out, and he is completely in the dark. How many socks must he take out to make 100 percent certain he has at least one pair of black socks?Response: QwQ-32B answers the following:
Okay, so there's this problem about a man with 53 socks in his drawer. He has 21 blue ones, 15 black ones, and 17 red ones. The question is, how many socks does he need to take out in the dark to be 100% sure he has at least one pair of black socks? Hmm, let me think through this step by step.
First, I remember that these kinds of problems are usually about the worst-case scenario. Like, you have to consider the possibility that he picks the maximum number of socks without getting a pair of black socks, and then add one more to ensure that the next sock will make a pair. But let me make sure I apply that correctly here.
So, the colors are blue, black, and red. The counts are 21 blue, 15 black, and 17 red. He wants to be certain he has at least one pair of black socks. To be certain, we have to think about the worst possible way he could pick socks without getting a pair of black socks, and then the next sock would have to give him that pair.
Wait, but actually, a pair of black socks specifically. So, he could have other pairs, but we need to ensure that at least one of those pairs is black. Hmm, so maybe the worst case is he picks as many non-black socks as possible, and then as many black socks as possible without forming a pair? Wait, no, actually, to avoid having a pair of black socks, he could have at most one black sock. Wait, no, actually, if he wants to avoid having a pair of black socks, he can have at most one black sock. Wait, but actually, a pair is two socks, so to not have a pair of black socks, he can have zero or one black sock. But since he wants to avoid having a pair, the maximum number of black socks he can have without forming a pair is one. So, the worst case is he picks all the non-black socks plus one black sock, and then the next sock would have to be black, making a pair. Hmm, let me see.
Wait, but actually, maybe I need to think in terms of the maximum number of socks he can take without having two black socks. So, to not have a pair of black socks, he can have at most one black sock. So, the worst case is he takes all the non-black socks (blue and red) plus one black sock. Then, the next sock he takes would have to be black, giving him the pair. So, the number would be (number of blue + number of red + 1 black) + 1.
Wait, let me check the numbers. The number of blue is 21, red is 17, so blue plus red is 21 + 17 = 38. Then adding one black sock gives 39. Then adding one more sock would be 40, which would have to be either blue, red, or black. But since there are only 15 black socks, if he already took one black sock, there are 14 left. But actually, the total socks are 53. Wait, let me recast this.
Wait, total socks: 21 + 15 +17 = 53, that's correct. So, if he wants to avoid having two black socks, he can take all the non-black socks (21 +17 = 38) plus as many black socks as possible without making a pair, which is 1. So total socks without having two black socks is 38 +1 = 39. Therefore, to ensure that he has at least two black socks, he needs to take 39 +1 = 40 socks. Wait, but the question is not asking for two black socks, it's asking for at least one pair of black socks. A pair is two, so that would be the same as two black socks. So, yes, 40 socks would ensure that he has at least two black socks, hence a pair. Wait, but hold on, maybe I made a mistake here.
Wait, actually, the question says "at least one pair of black socks". So, a pair is two, so he needs two black socks. Therefore, the worst case is he picks all the non-black socks (38) plus one black sock, so 39 socks, and then the next sock (the 40th) must be black, giving him two. Therefore, the answer is 40. Hmm, but let me think again.
Alternatively, maybe I need to consider that he could have multiple pairs of other colors, but still not have a pair of black socks. Wait, but the question is only about having at least one pair of black socks. So even if he has pairs of other colors, as long as he has at least two black socks, that's okay. But the problem is to ensure that he has at least one pair of black socks, so the worst case is he has as many socks as possible without having two black ones. So that would be all the non-black socks plus one black sock, so 38 +1 = 39. Therefore, 40 socks would ensure that he has two black socks. So, 40 is the answer?
Wait, but let me check again. Let me think of another angle. Suppose he takes out 39 socks. The worst case is he took all 21 blue, all 17 red, and 1 black. So that's 21 +17 +1 = 39. In that case, he only has one black sock, so he doesn't have a pair. Therefore, 39 is not enough. So, 40 would be the next one. Since there are only 15 black socks, so after taking 1 black sock, there are 14 left. But the total socks left after 39 would be 53 -39 =14. But those 14 could be all black? Wait, no, because he already took 1 black sock, so remaining black socks are 14, but the remaining socks could be all black? Wait, no, because he already took all the non-black socks. Wait, in the worst case scenario, he took all non-black socks first, so the remaining socks are all black. Wait, but if he took all non-black socks (38) plus 1 black sock, then the remaining socks are 14 black. So, when he takes the 40th sock, it has to be from the remaining 14 black socks, so that would give him a second black sock, making a pair. Therefore, 40 is the answer. Hmm, that seems right.
But let me see if there's another way this could be approached. Maybe considering the Pigeonhole Principle. The idea is that to ensure a pair, you need to have one more than the number of categories. But here, the categories are colors, but with a specific requirement for black socks. Hmm, maybe not exactly straightforward.
Alternatively, think of the problem as: What is the minimum number of socks needed to guarantee that in that number, there are at least two black socks. Because a pair of black socks requires two. So, the worst case is you have as many as possible without two black socks, which is all non-black plus one black. So, 21 +17 +1 = 39. Therefore, 40 is the minimum to ensure two black socks. Therefore, the answer is 40.
Wait, but let me check if there's a different interpretation. The question says "at least one pair of black socks". A pair is two, so that requires two black socks. So, yes, the above reasoning holds. Alternatively, maybe someone might think that a "pair of black socks" could mean just one pair, but even so, that still requires two socks. So, I think 40 is correct.
Wait, but let me see if there's a different angle. Suppose he takes out socks, and he wants to ensure that among them, there is at least one pair which is black. So, the worst case is he has as many socks as possible without having two black ones. So, that would be all the non-black socks (38) plus one black sock (total 39). So, 39 socks could still only have one black sock. Therefore, 40 is needed. So, that's the same as before.
Alternatively, maybe someone might think of the worst case where he takes all the non-black socks and as many black socks as possible without forming a pair. Wait, but forming a pair of black socks requires two, so to not form a pair, you can have at most one. So, that's the same as above.
Alternatively, maybe someone might think of the worst case where he takes all the red and blue socks first, then the black ones. So, 21 +17 =38, then the next sock is black, so 39th sock is first black, 40th would be second black. So, yeah, 40.
Alternatively, if he took all the red and blue socks (38) plus all the black socks except one, that would be 38 +14 =52 socks, but that would include 14 black socks, which is way more than a pair. Wait, that's not the worst case. The worst case for not having a pair of black socks is having only one black sock. So, the maximum number of socks without two black socks is 38 +1 =39. So, 40 is the answer.
Hmm, so I think that's solid. Let me see if there's a different way someone might get confused. For example, maybe considering pairs of other colors? But the question only requires at least one pair of black socks. So even if you have other pairs, as long as you have two black socks, that's sufficient. So, the only way to not have a pair of black socks is to have 0 or 1 black socks. To maximize the number of socks without having two black socks, you take all non-black socks (38) plus 1 black sock, totaling 39. Therefore, 40 is the minimum number needed to ensure at least two black socks, hence a pair.
Therefore, I think the answer is 40. Wait, but let me check the numbers again. 21 blue, 15 black, 17 red. So, 21+17=38 non-black. 38 +1=39. 39+1=40. So, yes. Therefore, 40 is the answer. I think that's correct.
Final Answer The man must take out \boxed{40} socks to be certain he has at least one pair of black socks.
To determine how many socks the man must take out to be 100% certain he has at least one pair of black socks, we need to consider the worst-case scenario.
The man has 21 blue socks, 15 black socks, and 17 red socks. To avoid having a pair of black socks, he could take all the non-black socks (blue and red) plus one black sock.
Calculating the number of non-black socks:
- Blue socks: 21
- Red socks: 17
Total non-black socks = 21 + 17 = 38
Adding one black sock to this total gives: 38 (non-black) + 1 (black) = 39 socks
In this case, the man would have 39 socks without having a pair of black socks. To ensure he has at least one pair of black socks, he needs to take one more sock. This next sock must be black, as all non-black socks have already been taken.
Therefore, the minimum number of socks he must take out is: 39 + 1 = 40
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.