Achieving secure, scalable, and cost-effective architectural integration with AWS API Gateway, EKS and MSK

In today’s rapidly evolving technology landscape, businesses are leveraging the power of the cloud to build and scale their applications. This has allowed not only a quicker time-to-market for an application, but also an improvement in the integration process with partner applications; achieving larger and more complex target states in an instant, when time is of the essence.  

GFT is involved in various complex projects, one of them being in the digital-only, or ‘neobanking’ world. A modern bank needs to be a disruptive force in the banking industry, offering an alternative to traditional banking models, with a focus on digital innovation, user experience and cost-effectiveness. But, building a bank without multiple integrations would be impossible. And the only reason I say “impossible” is the fact that a bank needs to be integrated with many other systems, such as being able to have access to external card providers, payment systems, politically exposed persons (PEP) databases and so on. In this blog, I will briefly explore how GFT integrates different types of architectures securely, and in a scalable and cost-effective manner. 

So, how does GFT integrate with other businesses? 

We are a specialist multicloud company that connects and synchronises data, processes and functionalities between two or more applications in a cloud environment. This involves establishing secure communication channels, mapping data structures and implementing API-based interactions, to ensure seamless interoperability and data flow across the integrated applications. 

Amazon Web Services (AWS) is one of the partners we are using to achieve our integrations.  

AWS offers a wide range of services to facilitate this process, but three popular ones for integrating different architectures are Elastic Kubernetes Service (EKS), Managed Streaming Kafka (MSK) and the AWS API Gateway.  

What is AWS API Gateway? 

API Gateway acts as a central entry point, allowing you to expose APIs and manage their lifecycle. With API Gateway, you can securely connect and route requests between disparate architectures, such as microservices, serverless functions and legacy systems. It provides authentication and authorisation mechanisms, including OAuth and IAM roles, ensuring only authorised access. API Gateway also enables scalable and efficient handling of API traffic by automatically scaling resources based on demand, thereby optimising cost efficiency. By leveraging API Gateway, you can seamlessly integrate diverse architectures whilst maintaining security, scalability and cost efficiency in your AWS environment. 

Understanding Elastic Kubernetes Service (EKS):  

Elastic Kubernetes Service is a fully managed container orchestration service that simplifies the deployment, management and scaling of containerised applications using Kubernetes. EKS provides a highly available, secure and scalable platform for running containers, making it an ideal choice for integrating different architectural components. 

Leveraging EKS for architecture integration 

We can utilise EKS for either: 

• Microservices architecture 

EKS allows you to deploy microservices as containers, which can be individually scaled and managed. By containerising each microservice, you achieve better isolation and fault tolerance, making it easier to integrate multiple microservices into a cohesive architecture. 

• Serverless architecture 

EKS supports serverless workloads through AWS Fargate, which eliminates the need to manage infrastructure. You can leverage Fargate to run containers without provisioning or managing the underlying EC2 instances, ensuring scalability and reducing operational overhead. 

Incorporating Managed Streaming Kafka (MSK):  

Managed Streaming Kafka is a fully managed, highly available and durable Apache Kafka service in AWS. Kafka is a popular choice for building real-time streaming architectures and enables scalable and fault-tolerant data processing. 

• Event-driven architecture  

MSK enables the integration of event-driven architectures with EKS. You can use Kafka as a communication layer between different microservices or serverless components, allowing for reliable, asynchronous message passing and decoupling of services. 

• Real-time data processing 

By combining EKS and MSK, you can process and analyse streaming data in real-time. EKS provides the computational power and scalability to process the data, whilst MSK ensures reliable ingestion and delivery of streaming events. 

Security considerations

As always, we must ensure security is considered and reviewed as part of the whole architecture: 

• Network isolation 

Use Amazon Virtual Private Cloud (VPC) to create isolated network environments for EKS and MSK clusters. Implement security groups, network access control lists (ACLs) and VPC peering to control network traffic and restrict access to sensitive resources. 

• Encryption 

Enable encryption in transit and at rest, for data flowing between EKS and MSK clusters. Use AWS Key Management Service (KMS) to manage encryption keys and ensure data privacy and integrity. 

Cost optimisation 

With budgets under constant pressure, cost optimisation is always a consideration: 

• Autoscaling 

Leverage the autoscaling capabilities of EKS and MSK to scale resources up or down based on workload demands. Autoscaling ensures you only pay for the resources you need, optimising cost efficiency. 

• Reserved instances 

Utilise AWS reserved instances to save costs on long-term, predictable workloads. By reserving capacity, you can achieve significant cost-savings compared to on-demand pricing. 

Conclusions 

In conclusion, integrating different types of architectures in AWS requires careful planning to ensure security, scalability and cost efficiency. AWS API Gateway, Elastic Kubernetes Service (EKS) and Managed Streaming Kafka (MSK) provide powerful tools for achieving these goals.  

By leveraging EKS and MSK, businesses can build secure, scalable and cost-effective architectures that can handle the demands of modern applications. As you embark on your architectural integration journey, remember to consider the specific needs of your application, monitor costs closely, and leverage AWS’s extensive documentation and community support for best practices. And, if you need any custom help, GFT of course has a vast array of experience in integrations across various domains, working through the discovery, design, build and run phases. 

Architecture diagram example: