Big Data Solutions: Build or Buy
Published on Jun 28, 2016 by Sean Kantrowitz
Companies that have decided to get serious about using big data to inform decision-making and increase their productivity and revenue may find themselves at a pretty substantial fork in the road early on in the process – How will they manage the data? Should they buy a ready-made solution, or build their business intelligence tools themselves from scratch?
There is no “one-size-fits-all” answer to this question. But by identifying the size, scope, priorities, and specific needs and concerns that a business organization has, decision-makers can be confident about which route is the most appropriate. Here’s a breakdown of the pros and cons to be aware of when considering building vs. buying big data systems.
Building: The Pros
No one knows your needs better than you: Working internally provides a great advantage in creating a system that will speak directly to the needs and processes of the business. Business intelligence tools don’t operate in a vacuum, and will need to integrate with other internal systems to fulfill the intended purpose – a custom system can be built with that in mind.
“Larger companies with a very robust IT department will tend to default to the building option,” says Michael J. Taylor, a business intelligence consultant/advisor with nearly two decades of industry experience and a client list that includes Federal Express, Delta Airlines, The Walt Disney Company, and Verizon. “In these type of environments, you’ll find engineers who like to build the best mouse trap, so to speak. It’s sustainable work that also attracts young and talented people to the company, and at the end of the day, can provide a solution that directly caters to the operations of the organization that it was built for.”
A customized solution can offer enhanced security: In a time where the integrity of data is more important than ever, taking extra measures to ensure that a big data system is impenetrable from hackers, and even industry competitors, is a significant factor to consider. “You know what they say – if you want to keep a secret, tell no one,” says Taylor, who also serves a course director in Full Sail University’s Business Intelligence Master of Science program. “Maintaining that integrity and keeping it internal is definitely a major pro when it comes to building your own data systems.”
In many cases, having the most optimal data warehousing structure may be a strong advantage that one company has over another – so keeping those blueprints close to the vest might be a particularly high priority, as well. And although infrastructure and data management are two different beasts, they are ultimately tied to one another because of how closely they work together. To simplify – building your own house increases the chance that no one will know how to break in.
Building: The Cons
Custom solutions take considerable time and money: If a company already has the resources and time to develop their own infrastructure - though few small companies do - than this may not be a huge consideration. A careful cost-benefit analysis comes into play here; If new talent needs to be brought in, or people resources need to be diverted from other current projects, then the value of an in-house system may not outweigh the expense. These are also not one-time expenses. A system needs to be maintained and updated, meaning the need for resources won’t end.
Diverts resources from your real focus: “During my time with GE, one of the things that CEO Jack Welch used to always tell us was ‘stick to your knittings.’ It typically doesn’t make much sense to devote lots of time and finances towards a side endeavor that you’re not even that good at,” says Taylor. “United Airlines is good at making you feel comfortable while you’re on a flight to Hong Kong, and getting you there on time and safely. But they don’t want to start designing jet engines. It’s important to stick to your business.” The important considerations are: where does a company generate their revenue, where are their strengths, and how should they direct their focus.
Technology is always changing: It’s not just about building a system; it’s about maintaining, updating, and supporting it as well. Just trying to keep on the cutting edge of data management can be a full-time endeavor. As Dr. Horn-yeu Shiaw, a software engineer and big data expert with over a decade of experience working and consulting in the BI industry who – along with Taylor – is an instructor in Full Sail University’s Business Intelligence Master’s Degree program, points out, “There are definitely technical factors that should be considered. Building your own data structure will give you the benefits of control and customization, but if a company doesn’t have a strong financial backbone, building a database may cause upgrade concerns when it fails to keep up with ever-changing technology.” The financial outlay and resources are not just an initial commitment, thanks to the fast pace at which technology moves today. A company that can’t keep up will soon be struggling with out-of-date systems.
Buying: The Pros
Leave it to the experts: Businesses that specialize solely in creating data warehousing or data analytics solutions for clients bring with them a wealth of information and experience that can only be amassed after logging countless hours encountering and troubleshooting obstacles - things potential buyers might not even be aware of yet. The experts, specialists in BI solutions, have the advantage of experience. When buying a solution, a company also buys the support that comes along with the product, a great value to consider when weighing expenses.
Someone else has to keep up with the latest and greatest: As noted, when creating an infrastructure for big data, a business not only has to consider their immediate needs, but also think several steps forward into a landscape that may gradually evolve or move rapidly in a completely different direction. Buying a big data system likely comes with solutions that already address these potential hiccups. At the very least, the vendors who sell these systems will be already hard at work forecasting what the next step will be – a full-time effort for their teams – and making the necessary upgrades in order to remain competitive in the field of data management.
Get started right away (maybe): Companies that sell big data systems have already invested in developing their software, so buyers can rest assured that they can jump right in when using these big data solutions. This can be a great advantage over spending time crafting a customized system – an endeavor that takes significantly more time and also requires a great deal of financial investment. If the company’s needs are fairly standard for the industry, this should be a fairly smooth process.
“You don’t have to worry about installation or migration challenges. It’s all already taken care of for you,” says Taylor, “A lot of times these systems have been standardized and are easy to integrate into your current system. APIs have been built upon the knowledge of what professionals in the industry are already using. Even if the sellers themselves don’t necessarily utilize some of those programs, it’s already on the technology roadmap.” Training, onboarding, and troubleshooting are generally all part of the package, providing a great start in implementation.
Buying: The Cons
It’s not always a perfect fit: There are some headaches that businesses might encounter if they go the route of buying their structure but need more customization than the system provides for. Proprietary components and tight, specific coding can sometimes leave companies out of luck if they have specific or unusual needs. Taylor – who also has a background in mechanical engineering and spent many years of his career implementing BI solutions to that industry – draws a comparison to the world of jet engines.
“If you’re United Airlines and you have a fleet of airplanes that have been using Pratt and Whitney engines, it’s very difficult for you when the next fleet of Boeing or Airbus airframes comes out and you decide that you want to put GE engines in your next plane,” he says. “Every one of your maintenance shops has been tooled for the type of engine you’ve used previously, so introducing a new element becomes problematic. That is essentially what it’s like when you buy instead of building – you’ve basically committed to sticking to one route and have forgone being able to make changes, because your system is already pretty solidified.”
You’re putting your eggs in someone else’s basket: Investing in a system from a company specializing in big data management means putting a certain amount of trust in that company: trust that their system will achieve the stated goals, that it will integrate with the necessary systems, that they will provide an acceptable level of support, that they will update and secure the system at the appropriate pace, and even that they will remain in business. Using due diligence in the selection process should ensure that the chosen solution will fulfill all of their obligations, but there are no guarantees. Building a solution offers an element of control that a purchased solution can’t match.
At the end of the day, there’s no right or wrong answer when it comes to the buying vs. building debate, and as technology moves forward there is an ever-increasing array of options available. Assessing the goals and priorities of an organization, along with a close look at the pros and cons, can provide a solid foundation for selecting the big data solutions that are right for each organization’s needs.
Business intelligence technologies is one of the many topics explored by students in the Business Intelligence Master of Science degree program at Full Sail University. Click here to learn more about Full Sail’s accelerated online programs, and get started on your path to a master’s degree today.