A Problem

I have been working more and more with AWS and EC2 and one of the challenges in working with EC2 is dealing with the fact that each instance gets a dynamic IP address upon creation. This makes it easy easy to crank out a large number of instances, which is a key feature of the system. At the same time, it makes it difficult to find and manage those instances in an automated, systematic way. So, there is a disconnect here.

A Solution

A decent solution is Dynamic DNS. That is, to have your instance be assigned a easily recognized hostname as it is being started, but still keeps its dynamic IP address and creation. To me, this seems like the best solution, as it lets AWS folks be good at what they are good at (providing cool technology infrastructures) and allows its users to be good what they are good at (making cool applications that use the infrastructure).

How can this be done? Well, it takes a couple of steps:

1. Establish an account with one of the myriad DDNS providers.
2. Configure your instance to use the DDNS software upon boot-up.
3. Profit!

I have gotten some feedback about possibly using SQS to do something similar to this.  I actually thought of this, but there are issues here around cost (cost per message) and configuration (duplicates and ordering).  Because I would like to maximize the system’s reliability and scalability, I would probably rule these out.Which Brings Another Problem …

There is a caveat here, and it isn’t a small one if you don’t want to write code. The big problem is that every instance you create is going to behave identically. So, if you open up multiple instances each thinking they are ‘dynamic-name-1..com’ then you will not get the results you are looking for. Instead, most likely that name will be assigned to the last instance that started and the others will run around headless.

Which Requires a Hack …

This leaves a couple of options. First, you can write a script to go through your pool of potential host names, pick one that isn’t being used, and then request that name from the DDNS provider. Better yet is to pass the Dynamic DNS value you want to the instance through the keypair.

Of course, the best solution would be for someone to write a script that would make each node self-configure itself to get a dynamic hostname.  I would think this would be something that would be attractive to most of the DDNS providers, perhaps one of them will read this and get cracking.  If I don’t spot anything in the near future I will probably write a simple KSH to do this.
Conclusion

I think that at some point the folks at AWS are going to allow for some type of host identification, either through passing parameters to instances or by renting out static IP addresses or subdomain ranges. However, for now it will take some DIY in order to make this happen.

To be honest, probably the best solution here is to approach the problem as if you will never have a static IP address or host name and go from there. It will probably force you to think about your solution differently and challenge you to come up with a more flexible, scalable solution. It isn’t going to fit for every type of problem, but I think it works for the types of things that AWS is inherently good at.

This entry was posted on Friday, December 1st, 2006 at 1:57 pm and is filed under Information Architecture, Systems Integration, Automation, Over the Horizon, AWS. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.