One of the paradigms we stand for as we work for the client is to always cater the solution exactly to her needs. As obvious as this sounds, we stretch that approach to include not only the to the business case. Instead, we stay flexible even when it comes to the implementation of specific tools. In February this year we decided to revise and update what we know about the current state-of-the art data tools. Today, we look closer at the one we decided to go further within one of our projects.

Increasing cloud-based options

Since our first years in the business, a lot has changed as of the options for handling the data and running the ETL process. The market shifted from legacy systems, on-premise “heavy machinery” for computation or warehousing and virtualizations to cloud-based, SaaS or PaaS (Software-as-a-Service or Platform-as-a-Service) solutions.

Recently, the options for extracting - transforming and loading the data with cloud technologies became even broader. After some research we have identified four different providers within this field, with which we wanted to continue some further discussions:

• Fivetran - A California-based extractor platform, claiming 5-minute setup times. As others, offering tenths of extractors for the main sources you can possibly find in your stack.
• Matillion - A full-service ETL coming from Manchester, UK with a sweet drag-and-drop environment.
• Querona - Platform with “ETL-free” consolidation on-the-fly with origins in Poland.
• Xplenty - A San Francisco's fast growing tool offering fast data integration.

As mentioned in the beginning, one of the immediate reasons to conduct this exploration was the aim to check for new possible solutions within one of our projects. The criteria then comprised of:

1. Complete ETL process availability,
2. Flexibility of available back-end warehouses (not fixed to just one of the providers),
3. User-friendly for business, non-technical users (no to basic SQL knowledge).

Several hours on video-calls and numerous demos later, our decision was to continue further with the British rising star - Matillion. Here is why.

Drag-and-drop with transparent pricing

Matillion offers integration with the main three players in the cloud-warehousing field as its' backend technology: Google BigQuery, Snowflake and Amazon Redshift. While Matillion for the later two is available through the AWS Marketplace, a BQ version is to be found on the Google Cloud Launcher site.

In both cases, the platform provides two pricing options - on hourly basis and a yearly fee. The former is great for proof-of-concepts or projects with low and irregular loads, annual subscription, on the other hand, favors bigger-scale projects with repetitive data-orchestrations. The price starts at $1.4 per hour. With online billing breakdowns within both of the providers' consoles you have 24/7 overview of the current cost of your service.

What persuaded us to continue with Matilllion for further work, was their Alteryx-like, user-friendly interface. As the picture below shows, the screen is divided into several sections. The main of them is a work-desk, which serves as a board for putting together an ETL flow from components located on the left-hand-side. The the screen also offers a direct view into the projects' warehouse enabling an immediate control over the changes inside the database. The bottom bar then provides for a space to parametrize the flow components - without the need to script in an SQL-like language. Although the the tool seems to include majority of functions which we might need in considerably complex project, should we need any custom components, the tool offers a nice bash/Python integration console for developing own stuff.

What's next

Overall, Matillion seems to give us yet another nice alternative to our current favourite tool. We are planning to play around with it a bit further. Then we definitely include it as one of the  go-to suggestions for an upcoming client project.