Amazon Web Services Streaming Data Solutions on AWS
8
encrypt the data before loading, minimizing the amount of storage used at the
destination and increasing security. It can also transform the source data using AWS
Lambda and deliver the transformed data to destinations. You configure your data
producers to send data to Kinesis Data Firehose, which automatically delivers the data
to the destination that you specify.
Sending data to a Firehose delivery stream
To send data to your delivery stream, there are several options. AWS offers SDKs for
many popular programming languages, each of which provides APIs for Amazon
Kinesis Data Firehose. AWS has a utility to help send data to your delivery stream.
Kinesis Data Firehose has been integrated with other AWS services to send data
directly from those services into your delivery stream.
Using Amazon Kinesis agent
Amazon Kinesis agent is a standalone software application that continuously monitors a
set of log files for new data to be sent to the delivery stream. The agent automatically
handles file rotation, checkpointing, retries upon failures, and emits Amazon
CloudWatch metrics for monitoring and troubleshooting of the delivery stream.
Additional configurations, such data pre-processing, monitoring multiple file directories,
and writing to multiple delivery streams, can be applied to the agent.
The agent can be installed on Linux or Windows-based servers such as web servers,
log servers, and database servers. Once the agent is installed, you simply specify the
log files it will monitor and the delivery stream it will send to. The agent will durably and
reliably send new data to the delivery stream.
Using API with AWS SDK and AWS services as a source
The Kinesis Data Firehose API offers two operations for sending data to your delivery
stream. PutRecord sends one data record within one call. PutRecordBatch can send
multiple data records within one call, and can achieve higher throughput per producer.
In each method, you must specify the name of the delivery stream and the data record,
or array of data records, when using this method. For more information and sample
code for the Kinesis Data Firehose API operations, see Writing to a Firehose Delivery
Stream Using the AWS SDK.
Kinesis Data Firehose also runs with Kinesis Data Streams, CloudWatch Logs,
CloudWatch Events, Amazon Simple Notification Service (Amazon SNS), Amazon API