Over the past four blog posts from me and my colleagues at IDC, we’ve tried to alert you to one of the most important new developments in the IT landscape: the emergence of Big Data as a driver of business change, technical innovation, and IT staffing requirements. One major concern that senior IT executives often bring up when we dive deeper into the Big Data issue is the potential impact on their evolving data center plans.

For many enterprises, data center concerns for the past 5 years focused on IT sprawl and inefficiency, power and cooling shortfalls, and inadequate disaster recovery plans. IT organizations are addressing these challenges with a range of solutions:

• New technologies such as server/storage virtualization to consolidate IT assets
• New IT deployment models such as converged IT infrastructure to improve utilization and maximize asset use
• New data center designs to optimize power and cooling while enabling more rapid IT recovery

Customers are increasingly deploying modular computing infrastructures optimized at the chasis and rack level for optimum processing, memory, I/O and storage performance. Capacity will be delivered in partial and full rack increments. Transaction and content delivery environments, used in traditional web serving and on-line transaction processing, are sensitive to latency, response time, and availability variations, so some geographic dispersion in multiple data centers makes sense.

As an organization makes the transition from Big Data as a “junior science project” to Big Data as a core business resource, concerns about the impact of this new, and rather different, workload on current and future data center plans will increase. Today, the IT architectural approach used in clustered environments such as a large Hadoop grid is radically different from the converged and virtualized IT environments driving most organizations’ data center transformation strategies. They have different server/storage configurations, different environmental (power and HVAC) profiles, and different data ingest/migration patterns.

In a number of larger enterprises (especially those with global needs like major financial institutions and retail organizations), IDC expects to see the emergence of separate data centers designed specifically for Big Data workloads. Concentration of data streams and compute resources makes more sense for both performance and telecommunications cost reasons.  Basically, the data center becomes the Big Data system.

This concentration requirement will also make “the Cloud” a critical part of the “Big Data” picture. Conceptually, “The Cloud” is about dispersion of computing and storage resources, but in reality, the organizations building the underlying cloud infrastructure are concentrating IT and telecommunications resources to more efficiently deliver applications and information. Those organizations are also playing a role in every facet of the Big Data space:

• They are one of the most important generators/collectors of data inputs
• They are among the most important aggregators/forwarders of data streams and content
• They are the most aggressive users/innovators in the development of Big Data hardware and software solutions for use in their own businesses

Given this combination, IDC expects cloud service providers to play a critical role (through simple, temporary provisioning of large processor, memory, and data pools) in the early and rapid adoption of Big Data use by technically savvy, but resource constrained, organizations. Already, cloud-based Big Data platforms are making it practical for smaller engineering and architectural firms to access massive compute resources for short, semi-predictable time periods without having to build their own Big Data IT farms.  Look for solutions targeting other sectors such as healthcare, consumer products, and logistics.

As an IT leader, you need to both understand what your organization is planning around Big Data and begin developing a Big Data IT infrastructure strategy (be it part of your own data center or with a cloud partner). In doing so, you must understand the potential value Big Data represents for your organization and industry. IDC believes that building successful business cases around Big Data can only be accomplished through a tight alignment of critical thinking across both IT and the business. This will require out-of-the-box business thinking as well as moving outside traditional IT comfort zones like traditional data warehousing.

Big Data deployments will also require you to rethink and realign IT administration and application developer skill sets. In this fast evolving area people with these skills are likely to be in short supply for quite a while. You may be able to retrain some existing team members, but once you do they will be highly sought after by competitors. Cloud service providers, making their own considerable investments in Big Data solutions will be among your best “friends” when it comes to getting started, getting serious, and getting to be a leader in Big Data.

Richard Villars is Vice President of Storage and IT Executive Strategies with IDC. His postings are his own opinions and may not represent AMD’s positions, strategies or opinions. Links to third party sites, and references to third party trademarks, are provided for convenience and illustrative purposes only. Unless explicitly stated, AMD is not responsible for the contents of such links, and no third party endorsement of AMD or any of its products is implied.