A powerful database in the cloud

Country: Poland

Sector: Public Sector

Services: Cloud services

Area: Azure, Power BI, Snowflake

A powerful database in the cloud

We have designed and implemented a highly efficient solution based on Microsoft Azure and Snowflake for our client, a green energy supplier. The combination of these cloud technologies allowed us to shorten data operations from a few hours to mere minutes and reduce costs by almost 30%.

About the client

The client offers green energy generated from 100% renewable sources and works with a group of over 450 energy producers in Poland.

The need

In order to obtain key data for making decisions at the operational and strategic level, the client obtained data from many sources, which then went to Cassandra, a Key Value Store database. This was not a satisfactory solution, as it did not allow for full control of the costs of data operations (the client paid for the maintenance of unused virtual machines, ineffective queries sent to the database, etc.). Furthermore, they used the Digital Ocean cloud, which did not meet expectations and did not offer all the services required.

The database was inadequate in terms of queries and was slow, and the overload generated additional costs. Therefore, the client needed an efficient Business Intelligence tool in a cloud environment that would meet their expectations and allow for cost optimization. At the same time, they had previously used a number of different tools and preferred to avoid retraining the team.

Challenges in the project

  • Checking the possibilities of available cloud solutions.
  • Organization of infrastructure – transferring the entire database to
    the cloud.
  • Creation of a data warehouse enabling the integration of data from
    various levels and sources.
  • The continued use of previously used tools and scripts without the need
    to retrain the client’s specialists.

Proof of Concept for a cloud-based data warehouse architecture

We wanted the client to be able to regain control over costs, which is why we decided to present the possibilities of cloud-based architecture. As part of PoC, we recreated some of the data and reports from the Cassandra database. Based on our previous experience, and at the same time encouraged by the positive attitude of the client, we proposed solution architecture based on the tried and tested Microsoft Azure cloud.

We confirmed that Azure was a suitable environment for embedding the Snowflake database technology as well as the possibility of using the existing scripts and tools in the Azure cloud. Subsequently we conducted an audit and performance tests, confirming Snowflake’s superior performance over Cassandra.

A high-performance Business Intelligence tool

With data security foremost in our minds, we transferred further parts of the Cassandra database and built the foundations of a data warehouse which will ultimately integrate data from various sources.

The client has gained modern reporting and analytical tools which allow for the optimization of costs. The solution allows the user to create in-depth analyses through the use of Machine Learning algorithms. The new tools do not require employees to be retrained. The client is now able to use components such as Docker or Kubernetes independently, which leads to higher efficiency while maintaining an optimal level of costs.

Benefits for the client

Reduction of the process time from several hours to mere
minutes.

  • Replacement of the Cassandra database with a tool from
    a similar price range, but with much greater efficiency.
  • An ordered data structure and having a data warehouse
    that enables the integration of data from many other
    systems.
  • Support and knowledge of the official Microsoft partner,
    Snowflake, and a leader in Business Intelligence
    implementations in Poland

Technologies

Snowflake, Microsoft Azure, Docker, Kubernetes, Jupyter, Microservices,
Airflow, Microsoft Power BI.

Azure cloud services used in the project

Data Factory, Functions, Logic Apps, DevOps, SQL Server, Virtual Machines,
Blob Storage, Key Vault.