Last week, I attended the TDWI Cloud BI Solution Summit as a speaker. I was fortunate enough to sit through some of the other presentations and I learned something interesting about the market place for Big Data and Cloud: not everyone is ready for it. This may sound like a given; American organisations are way ahead of Canadian organisations, but I found that there are still some fairly big fish out there that are still grappling with these concepts in the US.
Initially, I figured that the audience, composed mainly of senior IT people, middle managers, and executives, was up to speed with cloud computing and that its use was a forgone conclusion. This was clearly not the case.
There were questions from attendees such as:
"What is cloud BI?"
"Where is the data?"
And, a response to informal polls showed that:
- The majority of organisations represented did not have a "cloud BI solution".
- Few were considering moving their BI to the cloud.
- Licensing is still a concern.
- Performance is a concern.
- Data lifecycle management in the cloud is a concern.
So, it would seem that, while Cloud Computing (well, at least IaaS and SaaS; PaaS is coming into its own) has entered the vernacular of IT departments and some executive level suites, Big Data is still not there. Gartner has Big Data *just* entering the phase they call the "Peak of Inflated Expectations" which means that we are now looking to Big Data as a means of increasing our competitiveness without necessarily having the skills and knowledge of how to do so.
This leaves us in an interesting situation: if we can't do it ourselves, we have to subcontract to those who can. What are the costs? How is this agreement governed? If there are no stellar insights, whose fault is it? Is it even someone's fault? The fact that these questions exist is interesting because we're at the same stage we were at when the web came to prominence, when e-commerce was new, when the concept of outsourcing caught on, and more recently, as Cloud Computing became a new IT service delivery model.
The evolution of Big Data and its adoption will be interesting to follow. But one thing is for sure: look for Big Data to leverage public, low cost, commodity compute resources, and for more and more public cloud service providers to offer such services. After all, AWS has already launched such a service...
Netapp's Val Bercovici, AKA Big Data Czar, says that, in the cloud hierarchy, data is foundational and most valuable and that we can reach it and share more easily than ever before. It is because of this that we are able to use it in ever more creative ways such as Big Data (yes, big B, big D) to create value for our organisations.
He also said that the amount of data stored tends to exceed the capability of the network to carry it and why analytics and Big Data are done locally to the data rather than remotely. Cisco and telecom companies will be happy to note that uploading and downloading data is not simple and subject to the contraints of the network forcing any organisation or person who wants to move these quantities of data to buy more bandwidth.
Ultimately, the infrastructure planning and architecture should depend on where data will be stored, your governance, minimizing data movement, and maximizing data use.
So far, we've discussed what a community cloud looks like at a high level. In this post, we'll see a few of the benefits and take a look at a real-life example.
A community cloud, like any communal resource, is shared among stakeholders that have something in common such as regulatory requirements. This means that the cost of standing up a communal cloud versus individual private clouds can be significantly cheaper due to the division of costs among all participants. Think of it this way:
- If organizations A, B, and C each implement their own cloud computing environment to meet regulatory requirements, and assuming that these costs are roughly the same, we have $A + $B + $C = 3x the cost.
- However, if these organizations represent a community with common interests, then they can deploy a single cloud computing environment that meets all of their needs at some reduced cost; thus $(A, B, C) = 2x the cost of individual cloud environments (instead of 3x).
For this to work, the community needs to have some interest in its success, some skin in the game, so to speak. Joint ventures are a good example of such an arrangement. Each participant in the community contributes some consideration such as resources, funds, other assets, or some combination thereof, and ideally in equal proportions.
Another acceptable arrangement could be to outsource the management of the community cloud to a cloud provider. The advantage here is that the provider would be an impartial third party that is bound by contract and that has no preference to any of the customers involved other than what is contractually mandated.
Yet another arrangement would be for one of the participants to provide the cloud based services to its peers as a service provider. We already mentioned that the New York Stock Exchange (NYSE) is doing just that and will begin billing in a utility based billing model sometime in the near future.
One last thing that bears mention in this posting is the fact that community clouds can foster innovation. Typically, we think of community clouds as being horizontal in nature and encompassing similar organizations such as hospitals or government, for example. In fact, community clouds are not limited to horizontal integration, they can be vertically integrated like supply chains.
For example, a manufacturer produces a widget that is transported to a warehouse and distributed to retailers who then sell it to consumers. Tools residing in the community cloud can be used to leverage the information stored to serve customers and the supply chain, such as return tracking and just-in-time production and distribution. It is at this intersection of big data and tools that community clouds can really contribute to innovation.
This week, at Interop, I attended Private and Public Cloud Days in which speakers discussed the relative merits, drawbacks, and case studies. I've compiled a few ideas that I thought were interesting:
- Regarding private clouds, we have commonly referred to them in the context of dev/test, but in reality, their value is much more important as a way to deliver "IT as a service".
- Since clouds are commonly built on hardware that is not designed with redundancy in mind, cloud computing does not automatically imply redundancy. High availability is included in the architecture of your instance footprint (e.g., no IP hardcoding, no DB source hardcoding, etc.) and application(s) running on it.
- Redundancy costs money. Because of this, your organization needs to understand its tolerance to risk (and recovery point and time objectives (RPO/RTO)) and then design to meet this tolerance level.
- "Big data" should be actionable. Instead of thinking in terms of data, think in terms of business problems and solving those.
- Fear of lock-in from platforms (PaaS) is irrational since programmers have been locking themselves in by the simple act of choosing a programming language.