Sunday , 23 February 2020

The Advantages Commercial ETLs Have Over Writing Your Own SQL

ETL tools have come a long way since their creation. Whereas custom SQL tools had the advantage in the early days because of their versatility and flexibility, ETLs have become far more sophisticated in recent years. Now, ETL tools have surpassed manual SQL in a lot of ways, making them an attractive option for many who need an effective database solution.

While using a commercial ETL tool like Domo won’t be the correct choice for every use case (there will always be organizations that will need a custom-made solution), an increasing number of businesses are finding the benefits that come with using an ETL tool. In an effort to illustrate some of those advantages, we’ve put together this list, intended for anyone who’s still trying to decide whether to hire an SQL developer or just use a commercial option.

It Can Save Time and Money

Like any prebuilt solution in any industry, one of the most important benefits of an ETL tool is that you don’t have to build your own. With manual SQL, you have to hire one or more developers, then give them time to build the system, test it, and implement it, before you have a working solution. Like building your own website, app, or program, coding takes time and expertise, both of which cost money, all of which has to be expended before you have your tool in place to do what you need it to.

When you choose an ETL tool, you get a complete, plug-and-play solution that’s ready to start doing what you need it to do. While there is always a little bit of IT computer training necessary to know how to use the tool, the time investment required is much less severe. And since you’re up and running faster, you get your results faster. You spend less time trying to get your data, and more time putting that data to use.

It has Better Performance and Reliability

Just as building a tool takes time, making the tool reliable takes longer. Coding something that can perform its task quickly and reliably takes multiple iterations, with lots of testing to find the bugs in the code. All that effort, as mentioned above, requires capital.

Often, to avoid the cost and to get the tool online faster, stable and speedy performance is set aside in favor of speedy programming. The tool is assembled quickly, only included the necessary base functionality, leaving more advanced functions as either inaccessible or requiring a separately-coded workaround. This “code as you go” method leaves much to be desired in the realm of reliability, and if multiple developers are working the project, spaghetti code can result. Then, should anything break, fixing it becomes a veritable quest to Mount Doom.

Commercial tools have already front-loaded all of that work for you, creating a tool that’s stable, and has been tested in a variety of use cases by actual users. It’s already been taken through multiple iterations to deal with bugs, and is likely still being patched occasionally to fix new ones as they pop up. And best of all, if it does break…

Support is More Readily Available

One of the biggest advantages to using an established tool is that other people are using it, too. When something breaks, or you need to know how to execute a particular function, there are places you can go for the best tech support. First, there’s the customer support for tool. If that’s not enough, you can get help from the community forums. Often, your problem has been seen before, and there’s already an answer for it on the forums.

Meanwhile, those who build their own SQL tools have to troubleshoot issues by themselves for the most part. It’s their tool, and no one else has one built just like it, so the solutions will be specific to that tool. That could mean a lot of wasted time in trying to find a bug-free solution.

It’s Easier to Use

Commercial tools are also frequently built with non-developers in mind. Often operated via a GUI, much of the process is “drag-and-drop,” meaning almost anyone can be taught how to run it. Most useful functions—like “add constants,” “collapse columns,” “rank and window,” and so forth—will be built in, and can be done with the click of a button.

Compare that to a custom-built SQL tool, which likely won’t be optimized for user experience. Typically, an in-house tool will require at least a basic understanding of SQL code to operate, meaning that the people who need the data most can’t use it. Developers often find themselves being used as data gophers as a result, sinking inordinate amounts of time serving as middlemen just to make the tool usable.

It Makes Management Easier

Perhaps best of all, ETL tools have built-in management features that make supervising the process easier. For instance, there’s usually a visual representation provide that illustrates large portions of the ETL process. You can see what data is being pulled from where, what’s being done to it, and where it’s being sent. While those who bleed SQL or free database software will likely balk at a visual representation, everyone else will be grateful for it.

What’s more, ETL tools track an awful lot of metadata that management will certainly find useful. The list of collected metadata often includes things like which database merged datasets come from, how many columns have been modified, how long functions will take to process, and more. A lot of basic, manually-coded tools will lack that kind of functionality, and requesting it after the fact will take more time.

At the end of the day, when you need a database solution, you need to ask yourself if building something custom is really necessary. For very large organizations (who can likely field a sports team full of devs), or those with truly unique use cases, a custom solution may be the only answer. For many of us, though, using an ETL tool will make our lives significantly easier.

Image from

Leave a Reply

Your email address will not be published. Required fields are marked *


Scroll To Top