How EdgeTier Built ‘Ask Spotlight’

An engineering deep-dive into the architecture behind Ask Spotlight

Recently we launched Ask Spotlight. The idea of Ask Spotlight is to allow users to ask anything they want to know about their data. For example, in our system, they might ask “How many chats have I had every day since Monday?” or “What contact reasons have changed most significantly in the last week?”. Here’s how we built it.

MCP (or Alternative)

The first thing you might need to build is an MCP to your API(s) or various data sources. We already had one we built for internal use only initially. Since we use Python for almost everything backend-related, we went with FastMCP, but there are many alternative frameworks to choose from.

Can you skip the MCP? Probably yes, but you will end up creating something like an MCP for your agent to use. Ultimately it needs some way of retrieving the data it needs. As you will discover as you read further, MCP is a good standard if you choose an out-of-the-box solution such as AWS AgentCore.

Proof of Concept

As big users of AWS, we naturally looked for an AWS-based solution to the problem. We also wanted to keep all client data inside the AWS ecosystem.

Our first objective was to create a POC relatively quickly. The first prototype was quite simple and didn’t really introduce much new infrastructure. As current users of AWS Bedrock, we built an endpoint into our API that sent requests to Bedrock Converse directly. Our MCP was already deployed on an EC2 at this stage.

Here is a diagram of the POC, which probably looks more complicated than it was in reality (you may need to open in a new tab to see the full thing):

This solution worked fairly well for evaluating the concept and to support the build and test of our frontend. And ultimately it proved that we wanted to continue with the project. However, we never expected that this solution would ever get close to production, as it has many limitations:

• It doesn’t scale well. Interactions with an agent are relatively long-running tasks as far as an API is concerned. One user question blocks one of our API worker processes for a long time. Handling many concurrent user requests at once would become a problem.
• We had to write a fair bit of code to handle the back and forth between agent and MCP. This isn’t something we really wanted to maintain ourselves ideally.
• The lifecycle of a conversation or history is not handled well without writing a lot more code. Users will expect proper session management and maybe long-term memory these days.

Can you address these limitations? Almost certainly, but it would require a decent amount of engineering effort and ongoing maintenance. There are enough solutions out there these days that you probably don’t need to build your own unless you have extremely specific requirements or are really optimising for running costs.

AWS Bedrock AgentCore

We decided AWS Bedrock AgentCore was the most obvious alternative for a production-ready solution. It basically addresses all the limitations mentioned above:

• It scales automatically and won’t introduce any blocking into our API.
• More native support for MCPs i.e. less custom code.
• Sessions are managed properly and there is an optional memory feature.

Apart from this there are other advantages:

• AgentCore comes with optional observability features and CloudWatch dashboards for tracking session traces, tool invocations, token consumption, execution latency etc.
• While we don’t use them yet, you can write Cedar policies in AgentCore for specific security functionality.

The disadvantages of using AgentCore:

• It probably isn’t the cheapest option available. We suspect an evolution of our own POC solution, improved to be production worthy, could be significantly cheaper. However, we would have spent a lot in developer build and maintenance time.
• There is a reasonably steep learning curve. Of course all AWS services have a learning curve, but AgentCore is fairly complex. In a way, it’s like a little ecosystem itself as there are multiple services within it; runtimes, the AgentCore Gateway, memory, policies etc.
• Authentication is a bit complicated to set up. We contact our MCP via the AgentCore Gateway, and that requires using AgentCore Identity as well. Not really a long-term downside as once set up it will work.
• Significant vendor-lock in. As far as AWS services go, this would be one of the harder ones to transition to another cloud provider.

We also decided to use the Strands SDK and, rather than storing prompts in markdown files, we chose to use Langfuse. Langfuse was something we are already using for other use cases so it was an easy transition in this case.

Here is the final (somewhat simplified) diagram of our production solution (again, you may need to open in a new tab to view it fully):

Omitted from the diagram for size:

• You probably shouldn’t call the Agent runtime directly, but rather put something in front (either your API which will just re-introduce the blocking issue, a Lambda, API Gateway etc.).
• We have a small DynamoDB table for storing recent events in case a frontend refresh breaks its stream and needs to resume the live results display.

This was the solution we eventually ran in our alpha test with real users before going fully to production. So far, we have been really happy with the result.

Costs

Cost per user question for us is on average about 7 cents USD in the eu-west-1 AWS region. You can add at least another 30% onto that cost for supporting infrastructure like Cognito, CloudWatch etc. depending on what you have enabled. Ultimately, this cost is only about 16% of the total cost per question of this feature for us, considering the token usage makes up the vast majority of the cost. But that can vary wildly depending on what model you choose.

Conclusion

Overall we and our users are happy with how we built this feature. In some ways, we could have gone straight to the AgentCore solution without the POC version, but building a POC like that these days with AI is no longer a big task, so in our opinion it was worth getting to a prototype as fast as possible to test the frontend quickly.