Data creation is occurring at a record rate. In fact, IDC’s Digital Universe Study, sponsored by EMC, May 2010, predicts that between 2009 and 2020, digital data will grow 44 fold to 35 Zetabytes per year. It is also important to recognize that much of this data explosion is the result of an explosion in devices located at the periphery of the network including embedded sensors, smartphones and tablet computers. All of this data creates new opportunities for data analytics in areas such as human genomics, healthcare, oil & gas, search, surveillance, finance and many other examples. This is known as the ‘Big Data‘ problem.

This data explosion also means that datasets are becoming increasingly large and difficult to manage via conventional database management tools. As data sets grow in size – typically ranging from several terabytes to multiple petabytes – businesses face the challenge of capturing, managing, and analyzing the data in an acceptable timeframe. Organizations are also struggling to understand the opportunity information provides through advanced analytics. The IDC study suggests that organizations with high rates of change in their business think about business operations differently. These customers often put data analytics to use as part of a wide range of business decisions often using data analysis to develop business strategies. In short these users make data based decisions both more efficiently and at faster speed than peers typical of their industry.

Barriers to big data adoption are generally cultural rather than technological. In particular many organizations fail to implement big data programs because they are unable to appreciate how data analytics can improve their core business. Business executives need to improve their ability to convey complicated insights to the organization and drive an effective action from the data analysis process. When getting started it is helpful to think of the following:

• Identify a problem that business leaders can understand and relate to – one that commands their attention.
• Don’t get too focused on the technical data management challenge. Remember that resources need to be invested to understand the uses for the data inside the business.
• Define the questions needed to meet the business objective and only then focus on discovering the necessary data.
• Understand the tools available to merge the data and the business process so that the result of the data analysis is more actionable.

Traditional business intelligence systems have historically been centrally managed in an enterprise datacenter with the scalable server and high-performance storage infrastructure built around a relational database. Now enterprises are working to extract competitive business value – and ultimately revenue – from a growing sea of data. These big data implementations leverage diverse sets of distributed semi-unstructured and unstructured data types which frequently start with mathematics, statistics and data aggregation efforts. Big data analytic software is increasingly deployed on massively parallel clusters leveraging Apache Hadoop framework, distributed file systems, distributed databases, MapReduce algorithms and cloud infrastructure platforms (for time-to-market and scale needs). These big data applications are becoming a source of competitive value for enterprises as firms monetize information by building data products and services. Despite the uncertainty many organizations face, several things are clear:

• Big data is not an extension of traditional BI and requires new thinking.
• New skills will be required including mathematics, statistics and business capabilities necessary to build revenue models around data.
• New tools and architectures such as Hadoop and MapReduce algorithms will be required to deal with largely unstructured or semi-structured data.
• Many of the resulting business opportunities created through big data will be industry specific (genomics, healthcare, oil & gas, finance, etc.) making the focus on time to revenue more critical.

Because speed is strategically important, it will be tempting for business to move forward without IT support. IT needs to recognize that it needs to think differently (and quickly) and fight for a seat at the table as big data strategies are developed. CIOs need to both understand what their organization is planning around big data and begin developing a big data IT infrastructure strategy. In doing so CIOs need to understand the potential value big data represents to their organization and industry. Skill sets will also need to be evaluated with a focus on training as core analytical skills are not widely available in the market and in many cases will need to be developed internally. IDC believes that building successful business cases around big data can only be accomplished through a tight alignment critical thinking across both IT and the business. This will require out of the box thinking as well as moving outside the traditional IT comfort zone as traditional data warehousing models may not be appropriate in order to effectively monetize the big data opportunity.

Matthew Eastwood is the Group Vice President of Enterprise Platforms for IDC. His postings are his own opinions and may not represent AMD’s positions, strategies or opinions. Links to third party sites, and references to third party trademarks, are provided for convenience and illustrative purposes only. Unless explicitly stated, AMD is not responsible for the contents of such links, and no third party endorsement of AMD or any of its products is implied.