Developing a cost-efficient data platform for energy supplier EKZ

For the Swiss energy supplier EKZ, Eraneos not only developed a new data platform to handle a constant stream of large-scale data but also provided on-the-job training for employees. In this article, Azure Data Architect Corné Versloot proudly shares insights on the tendering process, the key challenges, and, of course, the results of the collaboration. With a focus on scalability, minimal maintenance, and the integration of real-time data analysis, this project demonstrates how strategic cooperation and the use of modern technologies like Azure Data Explorer can lead to successful outcomes.

“Looking at the Total Cost of Ownership, this is by far the most cost-efficient IT solution I’ve ever developed. Every day, we process around 30 million events, amounting to about 15 GB of raw data per day, and the system is already prepared for future growth as more smart energy meters are rolled out. I’m proud that the client has had to spend virtually no time on maintenance over the past year, which really demonstrates how robust and efficient our solution is.”

EKZ, or Elektrizitätswerke des Kantons Zürich, is a key energy supplier for the Zurich region. The company was facing three core challenges that impacted the foundation of its operations. First, it lacked deep insight into its own network, which occasionally made it difficult to anticipate issues caused by maintenance or outages. Second, it wanted to improve customer interaction by providing them with near real-time insight into their energy consumption. Finally, regulatory requirements demanded that customers be able to request five years of historical data on their energy use. While this was already technically possible, it was impractical due to the significant amount of manual effort required. These challenges led EKZ to seek an external partner to help create a robust solution.

For Corné Versloot, Azure Data Architect at Eraneos, this project was particularly challenging and inspiring. “We’re tech enthusiasts, and we love the challenge of solving complex puzzles,” he explains. “The scalability required for 450,000 smart meters, each generating data every 15 minutes, combined with peak loads from portals and queries from data scientists, made this a fascinating problem. It wasn’t just about storing data but also keeping the solution cost-effective and scalable. On top of that, the ambition to provide EKZ’s customers with easy and fast access to their energy consumption data really appealed to us.”

The tendering process and the selection of Eraneos

“Our Swiss branch already had a relationship with EKZ, but the project exceeded the threshold for direct contracting, so we had to go through the formal tendering process. But our existing relationship gave us confidence to go the extra mile. For our tender, we had already built part of what they wanted using publicly available data. This allowed us to demonstrate exactly what it would cost to store the data and run queries through the portals. The MVP (Minimum Viable Product), along with other factors like pricing and availability, of course, was a key reason why the project was awarded to us.”

Joint team: advantages and disadvantages

During the tendering process, EKZ had already hired several people who would ultimately be operationally responsible for the solution. It was, therefore, a logical step to make these people part of the project team. This approach allowed EKZ’s employees to be directly involved in the technical decisions and implementations throughout the development process, which not only improved the quality of the solution but also ensured a smoother handover and faster adoption within the organization. By working together in an integrated team, knowledge and skills could be transferred directly, contributing to a more sustainable and manageable solution.

“Having the client as part of the project team proved highly valuable, especially since we also had to collaborate with the smart meter supplier. The EKZ team simply had an easier time getting things done with them, which helped us enormously. But you also let someone into the kitchen. When mistakes are made—such as in this project with a colleague who wasn’t meeting expectations—the client sees it firsthand. In these situations, openness and transparency become even more crucial. Fortunately, we were able to resolve it through open communication and collaboration. Ultimately, this allowed us to respond more quickly to the client’s needs and to tap into their internal knowledge directly. This led to a solution that was not only technically robust but also perfectly aligned with the client’s operational realities. It’s one of the key reasons this project was delivered on time and practically within budget.”

Key challenges

The project team faced two major challenges: firstly, the system that was the source of most of the data was undergoing migration that year. This caused delays, as Eraneos had to wait for the new system to be in place before determining the structure of the data. Secondly, EKZ’s SAP system underwent an upgrade, which also impacted the data flow. All in all, this resulted in several months of delay and additional unforeseen costs, as both systems, independently of each other, were unavailable to the project team.

Despite these challenges, Eraneos was also able to simplify the project. Certain cloud components that were originally part of the design were removed to reduce complexity. Their added value proved insufficient and ultimately detracted from the solution’s manageability and clarity. This resulted in a product that requires minimal maintenance:

The entire solution is now essentially maintained by one person, and he’s had virtually no maintenance work to do over the past year,” Versloot explains. “The system is designed to automatically scale and handle peak loads, keeping operational overhead to a minimum. For example, when the system providing data from the smart meters encountered issues and we temporarily received no data, the system processed a full day’s backlog without a hitch once the problem was resolved, and it did so within an hour without any human intervention. This not only makes the solution cost-efficient but also allows EKZ’s staff to focus on delivering new insights rather than constantly troubleshooting.

Based on this foundation, EKZ is now working to further develop the product, enrich its data landscape, and enable more detailed analysis. They are using Power BI dashboards to provide various departments with deeper insights and are considering the addition of historical data to reveal seasonal patterns. This presents a new challenge that I would be eager to tackle.”

Lessons for the future

“What I take away from this project for future ones is, firstly, the importance of building an MVP during a tender process. It’s incredibly valuable to show a concrete solution because it builds trust and demonstrates that you understand the challenges. It also gives the project a running start because the design is already in place and has been partially validated. This shortens the project duration and minimizes risks.

Secondly, you shouldn’t be afraid to collaborate as an integrated team with the client. This ensures that everyone is moving in the same direction and makes it easier to resolve problems quickly. You’re truly working towards a shared goal, and it strengthens trust and engagement on both sides.

Finally, in this project, we used a new technology: Azure Data Explorer. This technology turned out to be a perfect fit for our needs when processing large amounts of data in real-time. It’s a powerful tool that is not only scalable and cost-effective but also well-suited for IoT-like projects with high volumes and a constant stream of data. These kinds of specific technologies can really elevate a project to the next level. I expect to use this technology in future projects, for instance, in an industrial production environment.”