phps3tk - Amazon S3 PHP Toolkit

This is yet another set of PHP scripts for S3. It's primary difference from other packages is the us of the CURL functions to reduce memory usage when PUTing or GETing large files. Mime type detection is from the /usr/bin/file command which is only opened once, there were issues when exec()ing more than 1000 times. Some other packages also used a bucket:object_key convention we do not. Choice of delimiter is '/', there is not option to change it.

Command Interface

The primary command is s3.php, make a symlink as /usr/bin/s3 to make life simple. Currently there is also a command titled s3sync too. This script will shortly be merged with the s3.php script.

auth

Creates a "REST Query String Authentication" url to allow access to objects

s3 auth /bucket/prefix-00/object.001
http://s3.amazonaws.com/bucket/prefix-00/object.001?AWSAccessKeyId=123456789&Expires=141889120&Signature=edoceoybdZaNmGa%2ByT272YEAiv4%3D

ls

Listing Buckets

s3 ls ( or s3 ls /)
b--  07/13/06 03:53          0b bucket
b--  07/13/06 03:53          0b bucket.test
b--  07/13/06 03:53          0b bucket.real

List bucket contents

s3 ls /bucket/
-p-  12/31/69 04:00          0b prefix-00/
-p-  12/31/69 04:00          0b prefix-01/
-p-  12/31/69 04:00          0b prefix-02/

List buckets recursively

s3 ls -r some.bucket:
--o  04/16/07 02:01     3799b  prefix-00/object.one
--o  09/15/06 04:00 48923197b  prefix-01/object.two
--o  04/20/07 16:20  2457600b  prefix-02/object.nnn

get

s3 get /bucket/prefix-00/prefix-01/object.key /path/to/local

put

s3 put /path/to/local /bucket/prefix-00/prefix-01/object.key

API Interface

There are three objects provided by this library S3Cache, S3Connection and S3Response. The S3Connection object provides some ftp like methods to operate, get, ls, put, rm, stat. These methods return S3Response objects which provide a bunch of information about the request. The S3Cache is used by the sync feature but could be used by any consumer.

$s3c = new S3Connection($key,$secret,$config);
$s3r = $s3c->ls('/bucket/object/key',null,'/');
$s3r = $s3c->get('/bucket/object/key','/path/to/local/file');
$s3r = $s3c->put('/path/to/local/file','/some/object/key');
$s3r = $s3c->stat('/bucket/object/key');

Progress Meter

When uploading large files a display can be forced by settings the curl_progress parameter. The value set here will trigger the CURL progress meter when files are larger than this value. A null or zero value will disable progress meters.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 85  109M    0     0   85 93.2M      0  67540  0:28:21  0:24:07  0:04:14 70389

Resource Usage

This was when putting a file and processing a cache of about 8700 objects. Cache file size was 1M MB and the Cache is fully memory resident in this software.

UID   C    SZ   RSS PSR STIME TTY        TIME CMD
you   6 14486 17216   0 22:43 pts/0  00:00:04 /usr/bin/php s3sync bucket /full/local/path /prefix/in/bucket/

This is from a 35424 object chache, the cache file was 2.7 MB

UID   C    SZ   RSS PSR STIME TTY        TIME CMD
you  48 12770 44608   0 00:31 pts/0  00:00:45 /usr/bin/php s3sync bucket /full/local/path/ /prefix/in/bucket/

Dependencies

This package requires the PEAR package Crypt_HMAC and the SimpleXMLElement extensions for PHP. Install Crypt_HMAC as indicated below.

pear install --onlyreqdeps Crypt_HMAC

SimpleXMLElement should be included when building PHP. From source:

# ./configure [whatever] --with-simplexml [whatever]

Gentoo USE flags:

dev-lang/php +simplexml

Downloads

phps3tk.tgz

Support

Available from Edoceo or your local PHP developer, please start with our contact page. If the volume of requests increases we'll add direct email request or forums/email-list or ???.

History

28 Apr 2007 - Add auth() routines /djb
27 Apr 2007 - Cleaned S3 toolkit, now using single instance of file command /djb
17 Apr 2007 - Released /djb