As a Solutions Architect at Solace, I work with several Capital Markets customers. These customers range from large investment banks to leading hedge funds. Historically, these firms have been always at the forefront of cutting-edge technology with special focus on performance and resiliency. For several decades, they have invested in large scale infrastructure for heavy workloads, especially real-time jobs.
For example, most large buy-side and sell-side firms have made significant investments over the years building grid compute infrastructure for computing heavy jobs to perform analytics. One such usecase is Risk Analytics where the firm performs compute jobs regularly to gauge their exposure. Over the years, these calculations have become more real-time. While they used to be batch jobs earlier, they eventually migrated to being daily jobs that would be executed at the end of the day. In the last few years, several firms have started exploring and migrating to more real-time jobs to be able to get a more accurate representation of their risk.
One way to achieve that is with the help of the elastic compute provide by the major cloud providers such as AWS, GCP, and Azure. Instead of having a large on-prem grid compute that only gets used certain times of the day, Capital Markets firms are starting to leverage elastic compute available via managed kubernetes services such as EKS (AWS), GKE (GCP), and AKS (Azure). With these managed services, firms don’t have to manage their own server farm and waste resources while they sit idle most of the time. Instead, they are charged for whenever resources are used via pay-as-you-go model. Additionally, firms are able to easily scale horizontally without worrying about procuring and deploying additional servers themselves. With such flexibility, Capital Markets firms have found it easy to migrate their risk and post-trade calculations to the cloud.
However, just because they can migrate their calculations to the cloud easily, doesn’t necessarily mean that the Risk app is already on the cloud. In fact, many large firms still have most of their critical applications on-prem which means they require a flexible, performant, and reliable solution to bridge their on-prem environment to their cloud environment. Enter Solace.
A typical deployment for a Risk application to leverage elastic compute on cloud looks something like this:
The Risk application is on-prem and sends requests to the cloud-based Calc Engine deployed on AKS, EKS, or GKE. However, the requests are important messages that need to be sent from an on-prem application to a cloud-native service. We do not want to lose the request/message due to any network issues and want to be able to handle large burst of requests. Similarly, we do not want to lose any replies/results that need to be sent back to the Risk application once the computation job is finished.
To achieve that, several investment banks and hedge funds rely on Solace PubSub+ brokers’s Guaranteed Messaging. Solace PubSub+ brokers help send applications send requests and receive responses without the risk of losing any messages. Risk application can easily publish its calculation requests to Solace where the messages will end up in a queue for the Calc engine to consume and process. Typical architecture for such a deployment would look like this:
Why not use a cloud native messaging service?
An important question that gets frequently asked is why pick Solace over cloud-native messaging service that every cloud provider has. For example, GCP has Pub/Sub and AWS has SQS. Why not use those?
There are two main reasons for that:
- Cloud-native services do not extend to on-prem – Many established companies have a prominent on-prem infrastructure and are most likely, looking to have a hybrid-cloud deployment model going forward. With hybrid-cloud, it’s easier and simpler to rely on one messaging layer that can easily be deployed on-prem and on cloud. Additionally, it needs to be able to create an Event Mesh between the on-prem datacenter and cloud region so that on-prem applications can easily share data with cloud applications and vice versa. A cloud-native messaging solution cannot be extended to on-prem and that’s why Solace excels in hybrid-cloud usecases. Here is a sample deployment:
- Secondly, more and more Capital Markets firm are starting to adopt multi-cloud as their long-term strategy. With AI becoming more prominent and accessible and each cloud provider having their own offering, firms are opening up to multi-cloud approach. For example, a firm might use AWS for S3 and EKS, GCP for BigQuery and Azure for its AI models. Tomorrow, they might use AWS for its AI models and GCP for storage. To allow their teams to cherry pick most suitable service for their usecases, firms choose Solace for its cloud-agnostic approach and to be able to easily route messages to any cloud provider.
For example, a common deployment is to store post-trade data in BigQuery and BigTable for further analytics for usecases such as Research and Transaction Cost Analysis.
In the below architecture diagram, data from on-prem application(s) is sent to a local broker and then forwarded to AWS for risk metrics to be computed. Once computed, those results can be sent back to the Risk application. Additionally, they can also be easily sent to GCP and stored in BigQuery for further analysis via DataFlow (GCP’s managed service for easy integration). This fabric that has been constructed between the three environments to easily distribute real-time data is called an Event Mesh and is a crucial part of any hybrid-cloud or multi-cloud infrastructure in a modern data stack.
Wrapping up, more and more Capital markets firms are migrating their usual computation-intensive jobs to cloud to leverage elastic and scalable compute offered by managed services such as AKS, EKS, and GKE. Additionally, these firms are starting to consider multi-cloud as an option to provide their teams with the flexibility of choosing the most suitable offering for their usecase. To achieve both usecases, these firms are relying on Solace’s Event Mesh which allows them to send calc requests and computed results from one environment to another environment with zero message loss.