Getting real on data – smarter decisions in real time
Always online, permanently chatting, reviewing and commenting Facebook and Instagram posts – that’s become a reality in our daily life. But when it comes to financial transactions we have to slow down. You want to buy a new smart phone or TV? You can find a good offer online – but then you will have to go through a clumsy payment process. Things are even worse when you have to apply for credit. You receive your annual bonus and start looking for investment options – there are zillions. Why couldn’t your bank look at what your peers or people with similar profiles are doing and provide you with a personalised offer? That´s when Real Time Decisions come into play. Financial institutions sit on a goldmine of data, which they can now leverage by using big data technologies. As a result, banks can make personalised offers in real time, which go beyond ex-post analysis and which truly meet the needs of their customers.
There are many reasons that explain why financial institutions have fallen behind in their ability to respond in real time to their customer’s needs compared to Internet startups. The most important ones are not even of a technical nature. Indeed, banks realised early on the need to provide relevant reactions to a high volume stream of business events. This drove the emergence of complex event processing platforms during the beginning of the 2000’s. The first implementations of these platforms were very successful in certain clearly delimited fields, such as algorithmic trading, in which the data events being processed were well defined, understood, and of good quality.
The emergence of the Big Data phenomenon, however, has created a new sort of global knowledge warehouse which organisations need to be able to exploit in real time in order to remain competitive. This is the challenge posed by “Real Time Decisions” in the age of Big Data.
Today, individual consumers are demanding more ubiquitous digital services, with increased levels of sophistication and customisation and with almost instantaneous responsiveness. This requires the ability to automatically access all relevant information across a variety of data sources, and deliver precisely targeted Business Intelligence in form of actionable information (e. g. effective product recommendations). This must happen as the events occur: in the precious seconds in which the interaction takes place. This is the essence of Real Time Decisions systems in the age of Big Data.
How to harness new Big Data opportunities?
Harnessing and exploiting these new opportunities to improve sales, increase operational efficiency or gain new insights are challenging tasks. In order to be fully effective, Real Time Decision systems need to integrate and process all the data available to an organisation, both from internal and external sources. This means that the silo mentality, so pervasive in many large organisations, must be broken down in order to upload the full enterprise data set into a shared repository. Early adopters of the big data paradigm have been building these data lakes over the past few years, and are now starting to reap the benefits from the insights extracted by applying sophisticated techniques based on machine learning algorithms against these data sets. At the same time, data governance on the data lake must be carefully implemented. ‘Chinese Walls’ must still be erected as required by legal or regulatory compliance, data access policies carefully crafted, and audit records kept ensuring full traceability of any changes.
Additionally, in order to successfully deploy systems which support Real Time Decisions on Big Data streams, careful consideration must be given to designing and implementing a rigorous adoption plan of the emerging Big Data technologies. These technologies have a very different set of requirements and capabilities from traditional BI infrastructure, whether we consider their initial adoption and integration into existing IT landscapes, or a purely operational point of view.
Emerging Big Data technologies reshaping the Real Time Decision landscape
Originating from Internet tech giants, bolstered by a very active open source community and effectively curated by a series of highly innovative firms, a whole new ecosystem of technologies are already reaching enterprise-level maturity: highly scalable and redundant disk storage in industry standard servers like the Hadoop File System (HDFS), distributed and resilient parallel computing frameworks like Hadoop Map-Reduce, for batch disk based processing, or completely in-memory platforms like Spark.
Another major component of the Big Data technical ecosystem is resilient and scalable NoSQL databases, which do not have the rigid constrains of the relational model (but also lack some of its advantages). They range from fast key-value (HBase) to document stores (MongoDB, CouchDB). Besides that, a whole plethora of technologies for fast data ingestion (Flume, Storm, Kafka) and query analysis “a la SQL” (Hive or Impala) that allow the connection between long established Business Intelligence and data visualization tools are also rapidly reaching enterprise level.
Finally, leading edge analytic algorithmic libraries are being developed in languages like the widely used Java, Python or the functional programming language Scala.
A successful model to embrace Big Data Real Time Decisions?
Given our proven track record implementing Big Data based systems in production for some of our main clients, along with GFT’s dynamic innovation lab approach to evaluate new technologies, we are in a unique position to support pioneering companies which want to employ Big Data in the processing of Real Time Decisions.
To successfully embrace the technologies which support Big Data Real Time Decision systems while minimising risks, financial institutions need to combine a deep expertise of the core banking IT technologies with an active set of innovation laboratories and skilled professionals.
GFT has collaborated with some of our clients on the definition and execution of a number of projects which leverage these technologies:
- Detection in real time of anomalies on trading data streams, using machine learning algorithms to continuously monitor the data stream and flag any anomalies detected. The platform is built to scale linearly leveraging the Spark framework.
- Sentiment analysis in real time of retail banking client’s interactions with social media, in order to provide personalised recommendations for promotions of associated companies (tie-ins).
- Real time categorisation of a client’s bank account transactions, in order to support full text search capabilities of each client’s full transaction history on an online banking platform.
- Implementation of decision support systems for Anti-Money Laundering departments based on statistical analysis and natural language processing of social data activity, including the automated gathering and analysis of relevant adverse information.
- Definition of architectures for Hadoop-based data lakes on which to perform sophisticated Data Analytics activities.
In essence, there is no way around Real Time Decisions. The fintech community already provides advanced services and takes away high-margin areas from the banks. The volume might still be small, but the growth rates are impressive. Traditional banks have a much bigger pool of data that they can draw from. Leveraging this data goldmine and combining it with their brand reputation can pave the way into the age of the digital customer while at the same time ensuring profitable business for the banks.