I see the current state of the Internet as the ultimate success … You can deploy a wildly successful Web 2.0 application that serves millions of users and never know how a router, switch or load-balancer works. Even network security and firewalls that were making headline news not more than a few years ago are considered perfunctory. The success of these networking devices and technologies has enabled them to become part of the technology landscape that exists for all to use as they see fit, similar to the microprocessor or electricity.

It is always odd to see the once-glamorous jobs of your youth thrown onto the scrap heap of history (think about the differences in perception between the masons of the middle ages and your local bricklayer). Network Engineers were once the masters of a difficult and arcane field, literally bringing information from chaos. Now, the wizards have been trapped in tiny control panels for now, until they can be embedded in silicon for all time.

This has really got me to thinking about my own field, and its future. What specialties are going to dissapear if data becomes as reliable as electricity? For one thing, I think we would see ETL and Business Analysis become a single career path that is much more abstract and tools-based. With the advent of good BPM, I could see a lot of the scheduling and other mechanics pushed off towards the DBA’s and Systems Administrators. Also, I think that a lot of the hardware could be appliance based, or outsourced completely. Of course, this leaves a great opportunity for open source BI and for nimble players to attack the market and take advantage of the innovators dilemma.

A brave (an infinitely more useful) new world!

I work in ETL, which means I need to get a hold of big iron to crunch on big data. Machines are expensive, licenses are expensive, storage and networks are cheap. Scalability is important, but measured.AWS would be perfect for sourcing ETL jobs that are one-offs or are particularly large or complex. However, the major vendors are very particular about making sure that their products are only installed on authorized machines. They make it pretty difficult for you to cluster easily, especially if you are a little guy just starting out.

This is antithetical to the AWS approach to problems. Here, machines and storage are dirt cheap, networks are pretty cheap, and scalability is paramount. The most difficult thing is arranging a problem so that it can be worked on by your infinite monkeys, in the form of EC2 instances. The biggest problem then becomes licensing.

In a highly scalable environment, it is incredibly compelling it becomes to use easily licensed software. Compelling to the point where it becomes worth it to build your own tools instead of purchasing off the shelf. For example, for a web server I could use Apache or Websphere. Apache is free, and I can install it on my instance with absolutely no problems (as a matter of fact, it is pre-installed). With Websphere I am going to have to purchase a license (or more), then monkey with the fact that it will be installed on a new machine with a new hostname each time. You can make the same argument for MySQL vs. Oracle, or Python vs. .NET.

Now this isn’t an anti-corporate rant, not by a long shot. But, I think it is a valid way to look at how licensing will be a competitive advantage in the future. Software vendors should start looking at their products in terms of AWS and other compute farms, especially at the enterprise level. Those who don’t get out in front of this are going to find their lunches eaten, and quickly. There is quite a hype around Web 2.0 companies these days, this could be a great way for someone to get their foot in the door of the Fortune 1000.

Perhaps Richard Stallman should send a Christmas card from the bazaar to Jeff Bezos over at the cathedral this year …

While this is worth discussing just for the schadenfreude, I actually have a reason for talking about this. I was reading a post about running an entire business on SaaS and another switching businesses to a Web 2.0 environment when it hit me like a ton of bricks. Web 2.0 is fulfilling the dream of internet appliances.

A stretch … perhaps. But bear with me. The idea behind IA’s was to provide super simple access to internet services and take away the complications of managing the desktop. Tell me that Google hasn’t taken a quantum leap in that direction with Search, GMail, Calendar, Docs and Spreadsheets, and Maps. Tell me that Yahoo hasn’t done the same with Flikr (I don’t use Yahoo much so I don’t know their products as well).

Web 2.0 is effectively turning the computer into an internet appliance where it makes sense, but without sacrificing the advantages of running a fully functioning machine. It is making it possible to actually run a business from the web. This is going to radically change the business environment, at least as much as the transition from the mainframe to the PC. I can’t wait to see how everything will shake out …

An applicance is the complete encapsulation of a business process, freeing the user to step back and look at the bigger picture. For a product like Netezza, this it pretty obvious that it is a complete RDBMS encapsulated in hardware black box. But, as hardware becomes more of a commodity, operating system configuration becomes more complicated, and virtualization becomes widespread, it makes sense that the definition of an appliance could migrate out of silicon.

Consider that Amazon Web Services are working on some very interesting projects, namely namely S3 (online storage) and EC2 (grid for rent). These projects have essentially abstracted the operation and maintenance of processing and storage. Icebreaker sounds ideal for this kind of enivronment. Thinking of the metaphor of the kitchen, it changes the role of the server from appliance to countertop.

OK, OK, this is just a jumble of random thoughts.  I have been trying all week to get these thoughts out and on to paper, pretty unsuccessfully.  The one thing I think I think is that there are some radical changes just over the horizon, based on the trends in web services, appliance-ization, and outsourcing.  Something big, something both familiar and new and unexpected.  I will have to think about this a bit more …