Amazon S3 and Cloudfront cache, how to clear cache or synchronize their cache

I have a cron job that runs every 10 minutes and updates the content-type and x-amz-meta. But since yesterday it seems like after the cron job run, Amazon is not picking up the changes made (refreshing his cache).

I even went and made the changes manually but no change...

When a video is uploaded it has a application/x-mp4 content-type and the cron job changes it to video/mp4.

Although S3 has the right content type video/mp4 cloudfront shows application/x-mp4(old content-type) ....

The cron job has been working for the last 6 months without a problem.

What is wrong with amazon caching? How can i synchronize the caching?

Use Invalidations to clear the cache, you can put the path to the files you want to clear, or simply use wild cards to clear everything.

http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html#invalidating-objects-api

This can also be done using the API! http://docs.aws.amazon.com/cloudfront/latest/APIReference/API_CreateInvalidation.html

The AWS PHP SDK now has the methods but if you want to use something lighter check out this library: http://www.subchild.com/2010/09/17/amazon-cloudfront-php-invalidator/

user3305600's solution doesn't work as setting it to zero is the equivalent of Using the Origin Cache Headers.

As to the actual code

get your CloudFront distribution id

Invalidate all files in the distribution, so CloudFront fetches fresh ones

My actual full release script is

#!/usr/bin/env bash

BUCKET=mysite.com
SOURCE_DIR=dist/

export AWS_ACCESS_KEY_ID=xxxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxxxxx
export AWS_DEFAULT_REGION=eu-west-1

   
echo "Building production"
if npm run build:prod ; then
   echo "Build Successful"
else
  echo "exiting.."
  exit 1
fi


echo "Removing all files on bucket"
aws s3 rm s3://${BUCKET} --recursive


echo "Attempting to upload site .."
echo "Command:  aws s3  sync $SOURCE_DIR s3://$BUCKET/"
aws s3  sync ${SOURCE_DIR} s3://${BUCKET}/
echo "S3 Upload complete"

echo "Invalidating cloudfrond distribution to get fresh cache"
aws cloudfront create-invalidation --distribution-id=S11A16G5KZMEQD --paths / --profile=myawsprofile

echo "Deployment complete"  

#References

http://docs.aws.amazon.com/cli/latest/reference/cloudfront/get-invalidation.html

http://docs.aws.amazon.com/cli/latest/reference/cloudfront/create-invalidation.html

Here is a manual way to invalidate the cache for all files on CloudFront via AWS

1. Open your CloudFront Distributions list, and click the desired distribution ID (circled in red in screenshot below) you want to clear it's cache.
2. Click 'Invalidations' tab (see selected tab highlighted in blue in the screenshot below).
3. Click 'Create invalidation' button (circled in red in the screenshot below)
4. Insert /* in the object paths input in order to clear cache of all files.
5. Click 'Create invalidation' button.

S3 is not used for real time development but if you really want to test your freshly deployed website use

http://yourdomain.com/index.html?v=2
http://yourdomain.com/init.js?v=2

Adding a version parameter in the end will stop using the cached version of the file and the browser will get a fresh copy of the file from the server bucket

Cloudfront will cache a file/object until the cache expiry. By default it is 24 hrs. If you have changed this to a large value, then it takes longer.

If you anytime needs to force clear the cache, use the invalidation. It is charged separately.

Another option is to change the URL (object key), so it fetches the new object always.

If you're looking for a minimal solution that invalidates the cache, this edited version of Dr Manhattan's solution should be sufficient. Note that I'm specifying the root / directory to indicate I want the whole site refreshed.

export AWS_ACCESS_KEY_ID=<Key>
export AWS_SECRET_ACCESS_KEY=<Secret>
export AWS_DEFAULT_REGION=eu-west-1

echo "Invalidating cloudfrond distribution to get fresh cache"
aws cloudfront create-invalidation --distribution-id=<distributionId> --paths / --profile=<awsprofile>

Region Codes can be found here

You'll also need to create a profile using the aws cli. Use the aws configure --profile option. Below is an example snippet from Amazon.

$ aws configure --profile user2
AWS Access Key ID [None]: AKIAI44QH8DHBEXAMPLE
AWS Secret Access Key [None]: je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
Default region name [None]: us-east-1
Default output format [None]: text

(edit: Does not work)As of 2014, You can clear your cache whenever you want, Please go thorough the Documentation or just go to your distribution settings>Behaviors>Edit

Object Caching Use (Origin Cache Headers) Customize

Minimum TTL = 0

http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html

I believe using * invalidate the entire cache in the distribution. I am trying at the moment, I would update it further

invalidate request screenshot

Update:

It worked as expected. Please note that you can invalidate the object you would like by specifying the object path.

Don't use invalidations. They cannot be reverted and you will be charged. They only way it works for me is reducing the TTL and wait.

Regards