How to build an Event Pipeline within 1 hour and minimum lines of code in AWS

Kinesis Stream PUT Record

So far we have created a general endpoint for

• /Streams Listing all the available Streams
• /Streams/{stream-name} Describe the given stream

The step will be to create a endpoint to accept data. This time when we are creating the child resource, we will select the Enable API Gateway CORS so that we can send requests from browsers

The most crucial part of the setup is the following piece of JSON data that will structure the data called Integration Request (remove the comments)

which will transform the input data into a version that Kinesis would receive

Enter event-pipe into the text field (stream-name) and add the exact same request body to test the functionality. And once you click the test button you should get a similar response body from the Kinesis

Kinesis Firehose

Once we go back to the Kinesis dashbord you will be able to see the Data Streams and Firehose streams. What we need to do next is to create a “Delivery Stream”.

On the left side, Kinesis Firehose is created to write data into S3. In the first step of the creation we select which stream the Firehouse should consume from.

Next up is to choose whether we want to transform our incoming data into another format. For the brevity of this article, we won’t get into Lambda transformations, thus pick Disabled and move to the next step.

Firehose give us the opportunity to load the Kinesis Data Stream into different sources such as;

• AWS S3 ( Choose this )
• Redshift
• ElasticSearch Service
• Splunk

Finally we select how we would like our data to be batched which the default settings as follows;

• Every 5 MB of data
• Every 300 seconds

I wish there was an option to also batch with # of records however there isn’t. For now leave the S3 compression as disabled but once we go to production, we would also better to select the GZIP S3 Compression for the finalized data since it’s pointless to pay extra for the storage.

The final page of the wizard shows you the Review and that’s it. It’s time to deploy our API Gateway.

Now lets head over to our API. Mine is at the following address, and yours will be provided on the top part of the screen. As you can see, the end url contains the word “test”

https://m2v3xe5shl.execute-api.us-east-1.amazonaws.com/test/Streams

{"HasMoreStreams":false,"StreamNames":["event-pipe"]}

https://m2v3xe5shl.execute-api.us-east-1.amazonaws.com/test/Streams/event-pipe

will return you the details of the stream

and finally it’s time to put a record into our pipe.