OK, once you have done the steps mentioned in the article “Getting Started with AWS” you should be able to actually do something. Probably the easiest service to start with is S3 (short for Simple Storage Service).

What is S3?

Good Question. The AWS folks describe S3 as:

… a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.

OK, so that is about half marketing-speak and about a third gobbledy gook. Imagine S3 as a very large, very simple place to store files. It isn’t a hierarchical file system, instead everything is stored in buckets which are containers for objects stored on S3. Also, each object has a key which uniquely identifies it. You use a bucket like a directory, an object like a file, and its key like the file name.

The documentation says that:

Every object in Amazon S3 can be uniquely addressed through the combination of the Service endpoint, bucket name, and key, as in http://s3.amazonaws.com/doc/2006-03-01/AmazonS3.wsdl, where “doc” is the name of the bucket, and “2006-03-01/AmazonS3.wsdl” is the key.

Actually Doing Something

This all sounds great, but in reality we want to actually do something with this giant file system. Now, here is the tricky part. S3 is really set up as a developers system, and the tools are in a bit of a rough form.

At this point, you have two choices:

1. Buy a tool that has S3 support in it.
2. Pick a language and code it yourself.
3. Use ready made tools.

There are already lists of tools that support S3, and examples of #2 and #3 in the S3 Code Samples, and they are worth checking out. Each of them have varying degrees of configuration required, and what you use is really up to you. I would recommend #1 if you are using Windows or aren’t either an experienced developer or willing to learn the hard way.

I am a fortunate enough to use Mac OS X, so I really have my run of things I can use. I picked s3curl because I am lazy and don’t want to have to make, configure, or compile anything and am comfortable on the command line. This didn’t work (missing packages in PERL, methinks) so I switched to s3-shell, although I did have to install ant to do it. It worked fine, but it is missing some pretty basic things, like the ability to set access to the buckets and objects you have created. This won’t do!
The best solution I could find was jSh3ll. It is a ready made Java tool comes compiled and has a lot of nice command line options, including the ability to run scripted commands from an input file. So, for now that is that I am going to be giving descriptions with.

The Nitty Gritty

Now, start jSh3ll, and enter in the following commands (substituting in your own access_key_id and secret_access_id)

$ java -jar dist/jSh3ll.jar

Welcome to jSh3ll (Amazon S3 command shell for Java) (c) 2006 SilvaSoft, Inc.
Type ‘help’ for command list.

jSh3ll> host s3.amazonaws.com
jSh3ll> user access_key_id
jSh3ll> pass secret_access_id
jSh3ll>

At this point, you should be logged in. To see what commands are available, you can use help:

jSh3ll> help
bucket [bucketname]
count [prefix]
createbucket
delete
deleteall [prefix]
deletebucket
exit
get
getacl [’bucket’|'item’]
getfile
getfilez
gettorrent
head [’bucket’|'item’]
host [hostname]
list [prefix] [max]
listatom [prefix] [max]
listrss [prefix] [max]
listbuckets
pass [password]
put
putfile
putfilez
putfilewacl [’private’|'public-read’|'public-read-write’|'authenticated-read’]
putfilezwacl [’private’|'public-read’|'public-read-write’|'authenticated-read’]
quit
setacl [’bucket’|'item’] [’private’|'public-read’|'public-read-write’|'authenticated-read’]
time [’none’|'long’|'all’]
threads [num]
user [username]
jSh3ll>

Next, create a bucket (substitute its name for bucket-name below) and upload a file to that location.

jSh3ll> bucket bucket-name
Bucket set to ‘bucket-name‘
jSh3ll> createbucket
Created bucket ‘bucket-name‘
[runtime: 2.614s]
jSh3ll> list
Item list for bucket ‘bucket-name‘
jSh3ll>

This shouldn’t return anything, as the bucket should be empty. Now, let’s try uploading something (substitute your file’s name for file-name below):

jSh3ll> putfile key-name file-name
Stored item ‘bucket-name/key-name‘
jSh3ll> list
Item list for bucket ‘bucket-name‘
key=key-name, owner=user-name, size=XXXX bytes, last modified=Fri Sep 29 23:45:36 CDT 2006
[runtime: 2.077s]
jSh3ll>

At this point, you should see key-name, as this is the name for the file on the remote system. Also, at this point this exercise is costing you money! Just to really prove to yourself that it is there, you can look at it through your web browser. First, we have to open up the Access Control List (or ACL) so that it the bucket and object can be seen:

jSh3ll> setacl bucket bucket-name public-read
Set ACL for bucket ‘bucket-name‘ to public-read
jSh3ll> setacl item key-name public-read
Set ACL for item ‘bucket-name/key-name‘ to public-read

At this point, your file should be available at http://bucket-name.s3.amazonaws.com/key-name. If you wanted to be really clever and give your file a key-name like docs/critical/README then it would appear like a truly hierarchical file system.

Cleaning Up

The last thing you need to do is to delete the file and bucket to make sure you don’t incur any extra cost.

jSh3ll> delete key-name
Deleted ‘bucket-name/key-name‘
jSh3ll> deletebucket
Deleted bucket ‘bucket-name‘

To verify this you can check the URL again and make sure that you get back an error message.

This entry was posted on Friday, September 29th, 2006 at 9:15 pm and is filed under Systems Integration, Over the Horizon, AWS. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.