How to use Amazon S3 for persistent file storage on AppFog

Right now, we’re working hard on providing persistent file storage for AppFog. This kind of service is nice to have with a variety of languages and frameworks, but for PHP it is essential, and we know that. We’ll make a big announcement as soon as AppFog persistent file storage is both battle-tested and production-ready.

For the time being, we suggest looking at Amazon’s Simple Storage Service (S3) file storage system not only as a temporary but also as a potentially permanent file storage solution (if you wish). S3 is inexpensive, mature, easy to use, and reliable.

Initial setup for the S3 service

If you already have Amazon’s S3 service set up, feel free to skip down to the next section.

If not, then beginning S3 integration with your PHP application will require you to sign up for an account with Amazon Web Services (AWS). From the account/console page, click on the “Amazon Simple Storage Service (S3)” link under “Infrastructure Services” and then on the “Sign Up Now” button on the main S3 page.

At this point, you’ll need to enter your credit card information to continue. But don’t worry: there are no fees up front and you’ll only be charged for the services that you actually use on S3. Once you’ve done that, you’ll be asked to verify your account via email.

Managing buckets in the AWS console

Once your S3 service is set up, head to your AWS console and click on S3 in the “Storage & Content Delivery” section:

In the S3 management console, create a new storage bucket:

I’m going to not-so-creatively name mine rainyinpdx in honor of the atrocious Portland weather today and store it in Amazon’s Virginia data center (all buckets share the same namespace, so make sure and choose a different name for yours!). Upon creation, you’ll be taken to the console for your bucket:

Here, you can manually upload files, separate them into folders, copy and paste files, and so on, all according to a very simple UI.

The next step is to give my PHP application to the files stored in my rainyinpdx bucket.

How to access your bucket using PHP

Luckily, the S3 console makes it incredibly easy to manage your file system from a centralized location. Now, it’s time to integrate our application with S3 on the coding side.

A variety of PHP frameworks offer their own ways of managing S3 integration, including Drupal, Zend, CodeIgniter, WordPress, and many others. If you’re comfortable working in one of those frameworks, there are a lot of good resources, plug-ins, and tutorials currently available. This tutorial, however, is for those who’d like to use S3 outside of an established PHP framework or to use S3 without using framework-specific plug-ins.

In order to interact with my rainyinpdx S3 bucket via PHP, I’ll be using this S3 module. First, we need to download the GitHub repo via Git:

git clone https://github.com/tpyo/amazon-s3-php-class.git

Once we’ve done that, we need to move the S3.php file into our app’s directory and we’ll be ready to interact with the module.

Now, let’s create a sample my_s3.php file that will enable us to access our bucket and start by loading up the module and defining our AWS security credentials, which we can always easily access via the “Security Credentials” page in the AWS console:

<?php

require_once './S3.php';

define('awsAccessKey', 'MY_AWS_ACCESS_KEY');
define('awsSecretKey', 'MY_AWS_SECRET_KEY');

One way to set these two variables is to simply copy and paste your AWS credentials directly into your application code. But this is not in line with best practices. It is far better to set variables such as this as environment variables for your app. This can be done either in the console, found under the “Env Variables” section of the console for a given app, or in the command line. I’ll show you how to set the MY_AWS_ACCESS_KEY variable through the command line:

af env-add rainyinpdx MY_AWS_ACCESS_KEY=<insert-key-here>

Now, you need to modify your variable definition code to access environment variables:

define('awsAccessKey', getenv('AWS_ACCESS_KEY'));
define('awsSecretKey', getenv('AWS_SECRET_KEY'));

Once you’ve done this with both your MY_AWS_ACCESS_KEY and MY_AWS_SECRET_KEY variables, you’ll be able to access your S3 account via your application without having to embed that information directly in your application.

Now, let’s create a $s3 object that will enable us to interact with our S3 bucket:

$s3 = new S3(awsAccessKey, awsSecretKey);

Now, let’s see if our $s3 object is properly set up. We’ll test this by grabbing a list of our currently available buckets:

print_r($s3->listBuckets());

The print_r command will print the response from the AWS server (in the console) when asked for a list of buckets corresponding to the access credentials I’ve entered. If I run php my_s3.php in the CLI, here’s what my response looks like:

Array
(
    [0] => rainyinpdx
)

So that confirms that we can interact with our S3 account via PHP. Now, let’s try to work with actual files in our rainyinpdx bucket.

Managing files in our S3 bucket

One way of managing files in S3 is of course to do so manually in the console. But very often, we need to be able to do so dynamically and to embed S3-related functionality into our working PHP apps. Fortunately, the S3 module that I have used thus far provides a very simple API for doing so.

At this point, we no longer need to verify that interaction is working, so let’s remove the print_r($s3->listBuckets()); line from our my_s3.php file. Now, let’s add a new file to our bucket. In the same directory where our my_s3.php file is, let’s create a simple new text file called hello.txt using a simple command in the CLI:

echo "wake up, sheeple!" >> hello.txt

Now, let’s add a line to our PHP file that will put our hello.txt file in our rainyinpdx bucket:

$s3->putObject($s3->inputFile('hello.txt', false), 'rainyinpdx', 'hello.txt', S3::ACL_PUBLIC_READ);

If I navigate to my AWS S3 console and hit “Refresh” in my browser, I’ll see my hello.txt file there and ready to go:

Now, let’s say that our hello.txt file has outlived its usefulness and we want to get rid of it. Let’s go back and modify our PHP script to delete it. Let’s start by removing the line we just entered that uploaded the file, and replacing it with this:

$s3->deleteObject('rainyinpdx', 'hello.txt');

If I run the script again, navigate to my S3 console, and click “Refresh” in the browser, I’ll see that my hello.txt file has indeed been deleted.

Simply uploading and deleting specific files is of course one of a much wider variety of tasks that you can perform via this S3 module. With it, you can also create and delete buckets, find out in which AWS infrastructure a bucket is located, copy objects between buckets, and so on.

I strongly suggest checking out the module’s very readable docs for information on how to use all of these commands, which are all fortunately quite syntactically similar to ones that I have used thus far.

Integrating S3 functionality via HTML forms

Let’s imagine that we want to build an app that enables users to upload files to our S3 bucket. This is actually surprisingly simple once we’ve set up our connection to S3 in our PHP app. This tutorial on NetTuts is the best that I have encountered, and I will rely on it heavily, although with appropriate modifications.

First, let’s set up an HTML form that enables users to post files to our PHP server:

<form action="" method="post" enctype="multipart/form-data">
    <input name="ourFile" type="file" />
    <input name="Submit" type="submit" value="upload" />
</form>

This will enable users to click on a “Browse” button that allows them select a file from their local machine. When they click the “Submit” button, the file will be sent to our server.

Now, we need to set up our server to handle this input. Let’s set up a form handler to handle the HTTP request sent by our file form and then upload it to S3 using the same putObject function that we used before. Let’s assume that users will only be uploading text files:

<?php

if(isset($_POST['Submit'])) {
    $fileName = $_FILES['ourFile']['name'];
    $s3->putObject($s3->inputFile($fileName, false), 'rainyinpdx', $fileName, S3::ACL_PUBLIC_READ);
}

Now, when we click on the “Submit,” our server will save the file directly to our S3 bucket.

All in all, this demonstrates only a small fraction of the functionality available to you using PHP and S3. But I hope that it suffices to provide some basic orientation on how to set up an S3 account, interact with buckets via PHP, and then to integrate some of that functionality into a dynamic PHP application.

Like I said, persistent file storage is coming to AppFog very soon. In the meantime, S3 is a powerful service that we strongly suggest using, be it in the short or long term. If you have any issues connecting to S3, please get in touch with our support team.

Share this post
Facebook Twitter Google
Try AppFog: The new PaaS Hackers love
  • 学峰 陈

    好复杂啊,晕了

  • http://forresto.com/ Forrest O.

    How soon is very soon? :-)

  • Gus Shotz

    Could you add Ruby Sinatra instructions on how to access an S3 bucket?

    • http://twitter.com/lucperkins Luc Perkins

      For Ruby, I highly recommend the AWS/S3 gem: http://amazon.rubyforge.org/. Lots and lots of functionality comes included.

      • Anonymous

        Also check out fog http://fog.io The S3 gem had thread-safety issues, not sure if those are resolved, but Fog is free and clear of thread-safety issues.

        • http://twitter.com/lucperkins Luc Perkins

          Thanks a lot, Jesse! I had heard good things about Fog, but now I’ll definitely have a dedicated look. I’m already impressed by how much it puts on the table.

  • Ben

    Really not happy that you’re making it hard if not impossible to switch a working WordPress install to a platform where I supposedly can’t even use half of the WordPress features like uploading – I don’t want to write a program just for your plaftorm

  • Danny

    MongoDB will store files via GridFS. I’m looking into setting up WordPress to upload to a MongoDB store.

  • adi

    Hi, Can you give tutorial how to use RackSpace Cloud Files as persistent storage?
    Thanks

  • http://www.facebook.com/leonardo.majowka Leonardo Majowka

    I saw that the persistent file system is been on top appfog top priority since june, so very soon can be too late

    • http://www.facebook.com/profile.php?id=61401254 Kevin Coyle

      It’s now March 2013. It’s not coming soon by any definition!

      • http://williamrobertson.com/ Will

        And now its 2014…

  • http://www.facebook.com/henriquebf Henrique Ferreira

    I use asset_sync to upload the assets after precompiling to S3. It works really well. I also upload paperclip images to S3. But it requires image magick and I didnt find any reference if appfog already supports it. I only find old messages, saying that appfog was working on it, when I google it. Any news about the imagemagick support?

  • http://www.bradezone.com/ Brade

    “Like I said, persistent file storage is coming to AppFog very soon.”

    Soooooooo… how’s this coming along?

  • Rain Walker

    I never found a updated Script with Amazons latest sdk. i have made it by myself. it woks as a php commandline interpreter script. give it a try : https://github.com/arizawan/aiss3clientphp

  • John
Powered by Olark