Kafka trading dashboard as a competitive advantage

The traders of a Swiss energy trading company had the idea to combine data from different sources, which would enable prompt reactions and swift decision making based on immediate market changes.

In Short

CHALLENGE

Differing data from two different source systems need to be combined in a real-time manner.

SOLUTION

Consolidating the separate data streams, combining them and visualizing the data in a real-time dashboard using Kafka Streams with the possibility to persist historical events.

BENEFIT

Increased efficiency of the strategy development process thanks to a real-time view on market changes and their impact on the company’s financial position.

Our solution

The traders of an internationally active energy group relied on multiple sets of data to make the right decisions at the right time: some of these data sets gave them a real-time overview of current market prices, the other sets of data provided an overview of their current positions. The two types of data sets originated from various sources and differed in format and structure. Hence, they could not be presented in a consolidated and combined manner, which inconveniently slowed down the decision-making process. Therefore, the client wished to combine the two data streams in a real-time manner in the backend and create a unified visualization of the combined data through a dashboard. This idea was to be realized with the potential of streaming, more precisely with the data processing software Kafka.

Competent and committed: Customer chooses Trivadis team

Among a couple of suppliers having experience with the Kafka technology, Trivadis were asked to answer a Request for Proposal (RFP). In the end, the clear and structured delivery approach, the broader set-up of a highly experienced and versatile team (instead of a single person), the prompt answers during the RFP phase and the demonstrated expertise and high commitment led the company to select the Trivadis team as their preferred partner.

With the customer's confidence gained, the team set about tackling the challenges of the project: combining two different data streams (one of them representing the real-time market situation) and visualizing the result in a real-time fashion. The traders obtain the data on specific real-time product prices in separate markets from a market platform. The overview of open positions (exposed to market price volatility) is provided by the company’s internal trading and risk management system.

Our traders have always been looking for the possibility of calculating and visualizing the financial value of their open positions in a real-time fashion. After some years of discussions and workshops, we wanted to try out and to justify the potentials of a streaming technology. By leveraging the benefits of Kafka, we have managed to combine our internal market-exposed positions with real-life market prices in a real-time manner and visualizing the back-end results on a dashboard. The project has greatly benefited from both, the conceptual Kafka knowledge as well as the hands-on Kafka experience of the Trivadis team.

Project manager at the IT department of the client company

Kafka-based solution creates competitive advantage

The solution for this project was developed by means of the Kafka data processing software. For the external market data platform, a mapping function was used to transform its product names into their corresponding product names of the internal system. Through this alignment, the concluded external market data stream could be merged with the data produced by the internal system.

The data from the internal trading system was reused in a new Kafka stream: the information of the external market data stream (representing the real-time market prices) was combined with the position data of the internal system in order to calculate the real-time financial value of exposed positions. Persisting these data streams in a database secures a certain fault-tolerance of the solution and provides the possibility to visualize the historical trends on the dashboard. Additionally, an advanced splitting of the Kafka topics ensures the solution’s scalability.

The real-time trading dashboard enables the client's traders to get a comprehensive overview of market movements as well as the impact of such changes on their financial positions at any point in time, which leads to fast decision-making processes and thus to a competitive advantage. Due to the great success of this project, multiple other use-cases emerged, and more departments were keen on taking advantage of the potentials of this technology.

Technologies used

Kafka
Kafka Streams
Confluent
Java

.NET
Python