I have been working with Google Docs and Sheets lately, in order to avoid the portability problem when working at different machines and locations. While it isn’t as fully featured as Excel, it does just about everything I need it to do, and then some. Plus, it adds in the collaboration features that are almost more useful to an internet-oriented business.

It would be incredibly boring for Google simply to replicate Excel and Word in a web format, unless you are an HTML groupie. However, there are some very, very interesting features that I think really turn the traditional office application on its ear. The first thing that caught my eye was the Google Lookup function, which allows one to incorporate search information dynamically into documents. The second thing was the Google Finance function, which allows financial information to be leveraged as well.  The third thing was the ability to embed portions or entire spreadsheets into a blog or web page.
Very cool stuff, and very interesting results. One could imagine this type of thing being leveraged with Froogle, Maps, or other service, within a document, presentation, or spreadsheet. Low cost, high reward stuff. However, there are some ramifications with using this type of information. For example, the spiffy new spreadsheet you put together for your boss could be modified by outside influences (like a Google Bomb).

Worth a look, at least.

I have been working with Amazon Web Services (and EC2) a lot lately, and have made some observations that really fly in the face of conventional wisdom.

I work in ETL, which means I need to get a hold of big iron to crunch on big data. Machines are expensive, licenses are expensive, storage and networks are cheap. Scalability is important, but measured.AWS would be perfect for sourcing ETL jobs that are one-offs or are particularly large or complex. However, the major vendors are very particular about making sure that their products are only installed on authorized machines. They make it pretty difficult for you to cluster easily, especially if you are a little guy just starting out.

This is antithetical to the AWS approach to problems. Here, machines and storage are dirt cheap, networks are pretty cheap, and scalability is paramount. The most difficult thing is arranging a problem so that it can be worked on by your infinite monkeys, in the form of EC2 instances. The biggest problem then becomes licensing.

In a highly scalable environment, it is incredibly compelling it becomes to use easily licensed software. Compelling to the point where it becomes worth it to build your own tools instead of purchasing off the shelf. For example, for a web server I could use Apache or Websphere. Apache is free, and I can install it on my instance with absolutely no problems (as a matter of fact, it is pre-installed). With Websphere I am going to have to purchase a license (or more), then monkey with the fact that it will be installed on a new machine with a new hostname each time. You can make the same argument for MySQL vs. Oracle, or Python vs. .NET.

Now this isn’t an anti-corporate rant, not by a long shot. But, I think it is a valid way to look at how licensing will be a competitive advantage in the future. Software vendors should start looking at their products in terms of AWS and other compute farms, especially at the enterprise level. Those who don’t get out in front of this are going to find their lunches eaten, and quickly. There is quite a hype around Web 2.0 companies these days, this could be a great way for someone to get their foot in the door of the Fortune 1000.

Perhaps Richard Stallman should send a Christmas card from the bazaar to Jeff Bezos over at the cathedral this year …

A Problem

I have been working more and more with AWS and EC2 and one of the challenges in working with EC2 is dealing with the fact that each instance gets a dynamic IP address upon creation. This makes it easy easy to crank out a large number of instances, which is a key feature of the system. At the same time, it makes it difficult to find and manage those instances in an automated, systematic way. So, there is a disconnect here.

A Solution

A decent solution is Dynamic DNS. That is, to have your instance be assigned a easily recognized hostname as it is being started, but still keeps its dynamic IP address and creation. To me, this seems like the best solution, as it lets AWS folks be good at what they are good at (providing cool technology infrastructures) and allows its users to be good what they are good at (making cool applications that use the infrastructure).

How can this be done? Well, it takes a couple of steps:

1. Establish an account with one of the myriad DDNS providers.
2. Configure your instance to use the DDNS software upon boot-up.
3. Profit!

I have gotten some feedback about possibly using SQS to do something similar to this.  I actually thought of this, but there are issues here around cost (cost per message) and configuration (duplicates and ordering).  Because I would like to maximize the system’s reliability and scalability, I would probably rule these out.Which Brings Another Problem …

There is a caveat here, and it isn’t a small one if you don’t want to write code. The big problem is that every instance you create is going to behave identically. So, if you open up multiple instances each thinking they are ‘dynamic-name-1..com’ then you will not get the results you are looking for. Instead, most likely that name will be assigned to the last instance that started and the others will run around headless.

Which Requires a Hack …

This leaves a couple of options. First, you can write a script to go through your pool of potential host names, pick one that isn’t being used, and then request that name from the DDNS provider. Better yet is to pass the Dynamic DNS value you want to the instance through the keypair.

Of course, the best solution would be for someone to write a script that would make each node self-configure itself to get a dynamic hostname.  I would think this would be something that would be attractive to most of the DDNS providers, perhaps one of them will read this and get cracking.  If I don’t spot anything in the near future I will probably write a simple KSH to do this.
Conclusion

I think that at some point the folks at AWS are going to allow for some type of host identification, either through passing parameters to instances or by renting out static IP addresses or subdomain ranges. However, for now it will take some DIY in order to make this happen.

To be honest, probably the best solution here is to approach the problem as if you will never have a static IP address or host name and go from there. It will probably force you to think about your solution differently and challenge you to come up with a more flexible, scalable solution. It isn’t going to fit for every type of problem, but I think it works for the types of things that AWS is inherently good at.

Computer World has a new article titled “Gartner: Age, attitude matter in IT“, that raises a lot of the issues that I have been discussing about the changing of the guard in IT and the ensuing culture wars. This has been brewing for a long time, and I think it will be a hot issue in virtually every organization over the next 5-10 years.  Let’s face it, it’s never too late to be a genius, but it’s also never to early to be self-centered.

I found an interesting article from Daniel @ benzadryne.cx attacking the spam issue from an economic perspective. This is not going to be a technical discussion; instead it will focus on the creative approach to solving the problem at hand.

The Problem

Spam is a complex problem, stemming from the following issues:

• Spam is anonymous as well as being fast, easy, and cheap to send.
• Blocking spam is slow, difficult, expensive, and difficult to organize on a large scale.
• Any effective spam defense can be overcome with enough time and energy.
• 99% of spam is unwanted by its recipients, who have to pay for it.
• 1% of spam is wanted, and this response rate is high enough to justify more spam.
• It is not uncommon for regular email to be accidentally blocked as spam, which aggravates customers to no end.
• People who send spam make a lot of money.

Yuck.

The Current Solution

This is quite a concern for email providers, as they have to make their customers happy. So, an entire industry (one even might say a subculture) has sprung up to try and solve the problem. So far, all the approaches have been around trying to block spam, either at the server or the client level. This has worked to make sure customers don’t see spam, but they still have to pay for it indirectly, through the systems that do the actual filtering.

Now, this isn’t bad for a first try, the thing is we are just treating the symptoms and not the disease. We don’t need to make our customers happy by making spam invisible, we need to do it by making spam go away. Nice thought, you say. How do we actually do that? If any effective counter measure can be overcome with time, then that route is just a short term fix.

The only other way to attack the problem is to make spam unprofitable.

The Creative Solution

What Daniel has suggested is to make it so that it takes hours (or more) for known spammers to try and send a single piece of mail. Thus, they can send less mail and make less money, making that activity less attractive to the unsavory elements. The technical details aren’t as important as the fresh approach to the problem. He has looked at the root cause of the problem and attacked it with simple and innovative approach that looks very promising.

Why You Should Care

Once again, this is a great example of how we can learn from the hacking community . Every organization has a problem like this one that seems insurmountable. A problem that is complicated, technical and personal, and every solution that has been tried so far has failed.

The way to start trying to solve a difficult problem is to:

1. Determine the root cause(s).
2. Attack the problem at the root cause.
3. Find a solution that works on the smallest possible scale.
4. Make the solution work on that scale.
5. Demonstrate the solution to others.
6. Help make the solution work on a larger scale.
7. Share the credit.