Getting your Twitter Timeline as an Email Digest using AWS

I normally avoid social media, but I make an exception for Twitter because I follow some interesting people there who do not publish anywhere else. However, I find scrolling through the site incredibly distracting. To avoid doing that as much as possible, I decided to explore ways to have the tweets delivered to me via email instead.

I started by exploring automation tools like IFTTT and Zapier. IFTTT has a lot of Twitter Integrations, but none of them provide a way to store tweets from your Home timeline. Zapier has a neat Digest app, which can be integrated with their Twitter app’s Tweet in List trigger to collect tweets from a Twitter list and email them periodically (see screenshot below). This met most of my requirements, but there were still two problems — I would have to move all my Twitter follows to a list, and I would have to upgrade to a starter plan for $20/month because the free version only supports 2-step Zaps (and mine would have been a 3-step Zap).

I finally decided to do it the hard way using custom code running in the cloud using AWS. I created a serverless system using Lambda and Simple Email Service that I could run for free and customize exactly as I wanted.

I am documenting all the steps below. It is a bit technical, and requires familiarity with Amazon Web Services(AWS) and using a command-line terminal. I have tried to include a lot of detail so that anybody with enough commitment can set this up on their own. All the code is available on Github. If you have any questions, require help or are encountering problems, feel free to open an issue.

Prerequisites

Install the following software on your computer.

• AWS Command Line Interface (CLI). You will also need to create or use an existing AWS account to set up the software in the cloud. (brew install awscli on Mac)
• Go programming language to compile the code (brew install golang on Mac)
• Terraform to perform deployments (brew install terraform on Mac)
• Git to check out code (brew install git on Mac)

Create credentials for Twitter API

You will need some secret credentials from Twitter that the code will provide to the Twitter API to retrieve tweets. Follow the steps below.

• Go to https://developer.twitter.com
• Under Apps, select Create an App
• Once the app is created, go to the Keys and tokens tab (see screenshot below), and copy 4 items: Consumer API key, Consumer API secret key, Access token, and Access token secret. We will need these credentials below when we create a configuration file to go along with the code.

Setup AWS Command Line Interface

Configure the AWS CLI by doing aws configure in the terminal:

> aws configure
AWS Access Key ID [None]: <redacted>
AWS Secret Access Key [None]: <redacted>
Default region name [None]: us-east-1
Default output format [None]:

Make sure you set a default region to avoid having to specify the region everywhere. It is also recommended that you set the AWS_REGION environment variable to the default region (us-east-1 above), unless you know what you are doing.

Verify email with SES

Use the CLI tools to verify the email address at which you want to receive the email digest (for e.g. user@example.com below):

> aws --region us-east-1 ses verify-email-identity --email-address user@example.com

Note that we explicitly specified us-east-1 as the region in CLI because SES is not available in all AWS regions. The Go code that we have also assumes the email is set up in us-east-1. So unless you know what you are doing, use the explicit --region flag above when running the command.

This will trigger an email to be sent to the address, which will contain a link to verify the email address.

Once you have clicked on the link, the email will appear as verified in the SES Console.

Create S3 Bucket to store tweets and deployment configuration

Next, create an S3 bucket to store the tweets as you retrieve them periodically from the Twitter API, before emailing them out as a digest. Once again, you will use the CLI for this.

> aws s3 mb s3://twitter-to-email-example

Configure Lambda function using a JSON file

Now, you could put together a configuration file that the Lambda function will use when it runs. Create a file named config.json, which will contain the credentials, email address, and bucket name. Here is an example:

{
  "bucket": "twitter-to-email-example",
  "consumer-api-key":"<redacted>",
  "consumer-api-secret-key":"<redacted>",
  "access-token":"<redacted>",
  "access-token-secret": "<redacted>",
  "email": "user@example.com"
}

You will need this file when we compile, test and package the code below.

Code to retrieve, store and email tweets

Check out the code from Github

> git clone https://github.com/deepakjois/twitter-to-email.git

Change into the directory, and either copy the config.json file created above into the folder, or use the config.json.example file as a template to populate the information as shown above

> cd twitter-to-email
> cp config.json.example config.json…(populate config.json with credentials and other information)…

The code for the project is contained in a single Go file twitter-to-email.go. If you are familiar with Go you can read through the code. The code is intended to be run every 5min. The core logic of the code is as follows:

1. Fetch today’s stored tweets from previous runs.
2. Get the latest tweet ID that was fetched and retrieve any newer tweets using the Twitter API, and add it to the list of stored tweets.
3. If running for the first time today:

• Fetch all of yesterday’s tweets, email them out and get the latest tweet ID
• Use the latest tweet ID and run Step 2 above.

It is possible to test the code locally before deploying it in the cloud:

> AWS_REGION=us-east-1 go test
Getting tweets from: s3://twitter-to-email-example/tweets/2019-10-08/tweets.json
tweets/2019-10-08/tweets.json not found. Trying to retrieve yesterday’s tweets
Getting tweets from: s3://twitter-to-email-example/tweets/2019-10-07/tweets.json
tweets/2019-10-07/tweets.json not found.
Uploading an empty array to tweets/2019-10-08/tweets.json
Uploading 0 tweets to s3://twitter-to-email-example/tweets/2019-10-08/tweets.json
192 New Tweets Found
Uploading 192 tweets to s3://twitter-to-email-example/tweets/2019-10-08/tweets.json
PASS
ok  	github.com/deepakjois/twitter-to-email	1.094s

Create a Lambda package

Before you create the AWS Lambda function and related resources, you need to package your code into a zip file. There is a script named build-lambda-package.sh in the repo to do that.

> ./build-lambda-package.sh
updating: twitter-to-email (deflated 58%)
updating: config.json (deflated 25%)

This will create a zip file named twitter-to-email.zipin the same folder.

Deploy using Terraform

With the Lambda function code packaged inside a zip file, you could now create the AWS Lambda function in AWS and upload code to it. You could create the function and related resources using the AWS Management Console, but that can be a time consuming and error-prone process. It is also difficult to track the changes that you make to your infrastructure.

Instead, you could use an Infrastructure as Code (IaC) tool like Terraform to declare all the resources in a file and create them all at once via the command line. There is a Terraform file named deploy.tf (Link) which contains all the resources needed for the deployment. You just need to make some small changes to use it for your setup.

There are a couple of places where the name of the S3 bucket I am using is hardcoded. You need to replace those instances in deploy.tf with the name of your bucket. Here is the relevant portion of the file:

# Terraform state in S3
terraform {
  backend "s3" {
    bucket = "twitter-to-email-example"
    key    = "terraform/terraform.tfstate"
  }
}# S3 bucket to store tweets
# FIXME: Change this to a unique value
locals {
  twitter_to_email_bucket = "twitter-to-email-example"
}…<rest of file>…

Once you have made the changes, run terraform plan to check the resources that will be created.

> terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.data.aws_iam_policy.AmazonS3FullAccess: Refreshing state...
data.aws_iam_policy.AWSLambdaBasicExecutionRole: Refreshing state...
data.aws_iam_policy.AmazonSESFullAccess: Refreshing state...------------------------------------------------------------------------An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create<…snip a lot of text…>Plan: 8 to add, 0 to change, 0 to destroy.------------------------------------------------------------------------Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

The output above is telling us that 8 new resources will be created in AWS, which sounds right because we haven’t created anything yet. We are now ready to proceed with the creation by running terraform apply

> terraform apply<…snip a lot of text…>Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.  Enter a value: yes<…snip a lot of text…>Apply complete! Resources: 8 added, 0 changed, 0 destroyed.

Now if you go into the AWS Management Console, you should see the Lambda function.

If you click the function, you can see the detailed configuration:

If everything has gone according to plan, you should start receiving tweets in your inbox every day at around 0000 UTC.

Troubleshooting and Monitoring

If you encounter any issues, first go to the Monitoring tab and check the logs for the service to make sure there isn’t any error. If you cannot figure out what is wrong, open an issue in the Github repo, and I will try my best to help you.

Additional Customization

You can customize the behavior of the function according to your needs by editing the Go code. You could change the frequency of the emails from 24 hours to every hour for example, or customize the email that gets sent out. Of course, that would require some familiarity with programming in Go.

Once you have tested and packaged the code (as shown in the previous sections), you can do a terraform apply to upload your changes to the function.

Good luck!