Syncing GitHub Repo to AWS S3 Location Using Webhook: A Step-by-Step Guide

Are you tired of manually uploading your GitHub repository changes to your AWS S3 bucket? Do you want to automate the process and save time? Look no further! In this article, we’ll show you how to sync your GitHub repo to an AWS S3 location using a webhook. By the end of this guide, you’ll be able to set up a seamless integration between your GitHub repository and AWS S3 bucket, ensuring that your changes are reflected in real-time.

What is a Webhook?

Before we dive into the tutorial, let’s quickly understand what a webhook is. A webhook is an API callback that notifies a specific URL when a particular event occurs. In our case, we’ll use a webhook to notify AWS S3 whenever there’s a push event in our GitHub repository. This notification will trigger the syncing process, ensuring that our S3 bucket is always up-to-date.

Prerequisites

Before we begin, make sure you have the following:

• A GitHub account with a repository containing the files you want to sync.
• An AWS account with an S3 bucket created.
• A basic understanding of GitHub and AWS S3.

Step 1: Create an AWS S3 Bucket

If you haven’t already, create an S3 bucket in your AWS account. Follow these steps:

1. Log in to your AWS account and navigate to the S3 dashboard.
2. Click on “Create bucket” and enter a unique name for your bucket.
3. Choose a region and click “Create bucket” to create the bucket.

Make a note of your bucket name, as we’ll need it later.

Step 2: Create an IAM User and Policy

To sync your GitHub repository with your S3 bucket, we need to create an IAM user with the necessary permissions.

Follow these steps:

1. Log in to your AWS account and navigate to the IAM dashboard.
2. Click on “Users” and then “Create user.”
3. Enter a username and select “Programmatic access” as the access type.
4. Click “Next: Permissions” and then “Attach policy.”
5. Click “Create policy” and then “Custom policy.”
6. In the policy editor, add the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowS3 BucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::YourBucketName/*"
    }
  ]
}

Replace “YourBucketName” with the name of your S3 bucket.

Step 3: Generate an Access Key

Once you’ve created the IAM user and policy, generate an access key pair for the user.

Follow these steps:

1. Navigate to the IAM dashboard and click on “Users.”
2. Find the IAM user you created and click on the “Security credentials” tab.
3. Click on “Create access key” and then “Create access key pair.”
4. Download the .csv file containing the access key ID and secret access key.

Make a note of the access key ID and secret access key, as we’ll need them later.

Step 4: Create a GitHub Webhook

Now, let’s create a webhook in GitHub that will notify AWS S3 whenever there’s a push event in our repository.

Follow these steps:

1. Log in to your GitHub account and navigate to your repository.
2. Click on “Settings” and then “Webhooks.”
3. Click on “Add webhook” and enter the following details:

Field	Value
Payload URL	https://your-aws-lambda-function.execute-api.us-east-1.amazonaws.com/production
Content type	application/json
Events	Push

Replace “your-aws-lambda-function” with the actual name of your AWS Lambda function, which we’ll create in the next step.

Step 5: Create an AWS Lambda Function

Now, let’s create an AWS Lambda function that will handle the webhook notification from GitHub and sync our repository with our S3 bucket.

Follow these steps:

1. Log in to your AWS account and navigate to the Lambda dashboard.
2. Click on “Create function” and choose “Author from scratch.”
3. Choose Node.js as the runtime and give your function a name.
4. In the “Handler” field, enter “index.handler.”
5. In the “Environment variables” section, add the following variables:

Variable	Value
GITHUB_TOKEN	Your GitHub token
AWS_ACCESS_KEY_ID	Your AWS access key ID
AWS_SECRET_ACCESS_KEY	Your AWS secret access key
S3_BUCKET_NAME	Your S3 bucket name

Replace the placeholders with the actual values.

Step 6: Write the Lambda Function Code

In this step, we’ll write the code for our AWS Lambda function.

exports.handler = async (event) => {
  const githubToken = process.env.GITHUB_TOKEN;
  const awsAccessKeyId = process.env.AWS_ACCESS_KEY_ID;
  const awsSecretAccessKey = process.env.AWS_SECRET_ACCESS_KEY;
  const s3BucketName = process.env.S3_BUCKET_NAME;

  const s3 = new AWS.S3({
    accessKeyId: awsAccessKeyId,
    secretAccessKey: awsSecretAccessKey
  });

  constgithubApi = new (require('github-base'))({
    token: githubToken
  });

  const repo = githubApi.repos.get(event.repository.full_name);
  const files = repo.tree(event.sha);

  files.forEach((file) => {
    const params = {
      Bucket: s3BucketName,
      Key: file.path,
      Body: file.content
    };

    s3.putObject(params, (err, data) => {
      if (err) {
        console.log(err);
      } else {
        console.log(`File uploaded: ${file.path}`);
      }
    });
  });

  return {
    statusCode: 200,
    body: 'Synced successfully!'
  };
};

This code uses the `github-base` library to interact with the GitHub API and the `aws-sdk` library to interact with AWS S3. It fetches the files from the repository and uploads them to the S3 bucket.

Step 7: Test the Webhook

Now, let’s test the webhook by making a push event in our GitHub repository.

Follow these steps:

1. Make a change to your GitHub repository (e.g., create a new file).
2. Commit the change and push it to your repository.
3. Wait for the webhook to trigger the Lambda function.
4. Check your S3 bucket to see if the file has been uploaded.

If everything is set up correctly, you should see the file in your S3 bucket.

Conclusion

In this article, we’ve shown you how to sync your GitHub repository with an AWS S3 bucket using a webhook. By following these steps, you can automate the process of uploading your repository changes to your S3 bucket, ensuring that your bucket is always up-to-date. Remember to replace the placeholders with your actual values and test the webhook thoroughly to ensure that it’s working correctly.

Happy coding!

Note: This article is for educational purposes only and should not be used without proper testing and validation in a production environment.

Frequently Asked Questions

Get answers to your burning questions about syncing GitHub repo to AWS S3 location using webhook!