Streaming QLDB Journal data to Lambda
The best time to establish protocols with your clients is when you onboard them.
Heading
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.
In this article we’ll discuss how to stream QLDB (Quantum Ledger Database) journal data to lambda using AWS kinesis Data streaming service.
Creating a Kinesis Data Stream
Go to Kinesis service and click on Create data stream.
Enter the stream name, capacity mode as On-Demand/Provisioned(select based on the requirement) and proceed with creating the data stream.
Once the stream has been created successfully, we’ll proceed with the second step.
Creating an IAM role that enables QLDB to assume write permissions for the Kinesis data stream.
- Create a permission policy:
- Open the IAM console
- In the navigation column on the left, choose Policies
- At the top of the page, choose Create policy.
- Choose the JSON tab.
- Enter a JSON policy document (Please refer to the below example). To use the below policy, replace us-east-1, 123456789012, and kinesis-stream-name in the example with your own information.
- Click on Next Tags.
- Click on Review.
- On the Review policy page, enter a Name and an optional Description for the policy that you are creating. Review the policy Summary to see the permissions that are granted by your policy. Then choose Create policy to save your work.
- Create an IAM role:
- In the navigation pane of the IAM console, choose Roles, and then choose Create role.
- Choose the AWS service role type, and then choose QLDB.
- Choose the QLDB use case. Then, choose Next.
- Select the box next to the policy that you created in the previous steps
- Choose Next.
- For Description, enter a description for the new role.
- Review the role and then choose Create role.
Now, we’ll proceed with creation of QLDB journal stream
QLDB Stream Creation
- Sign in to the AWS Management Console, and open the Amazon QLDB console.
- In the navigation pane, choose Streams
- Choose Create QLDB stream
On the Create QLDB stream page, enter the following settings:
- Stream name — The name that you want to assign to the QLDB stream.
- Ledger — The ledger whose journal data you want to stream.
- Start date and time — The inclusive timestamp in Coordinated Universal Time (UTC) from which to start streaming journal data. This timestamp defaults to the current date and time. It can’t be in the future and must be earlier than the End date and time.
- End date and time — (Optional) The exclusive timestamp (UTC) that specifies when the stream ends. If you keep this parameter blank, the stream runs indefinitely until you cancel it.
- Destination stream — The Kinesis Data Streams target resource to which your stream writes the data records. Use the following ARN format.
The following is an example: arn:aws:kinesis:us-east-1:123456789012:stream/stream-for-qldb - Enable record aggregation in Kinesis Data Streams — (Enabled by default) Lets QLDB publish multiple data records in a single Kinesis Data Streams record, increasing the number of records sent per API call.
- Service access — For Service access choose “use an existing service role” and select the role that we created in the above steps.
- When the settings are as you want them, choose Create QLDB stream.
If your request submission is successful, the console returns to the main Streams page and lists your QLDB streams with their current status. - After your stream is active, use Kinesis to process your stream data with a Consumer application.
For information about the format of the stream data records, visit the below url: https://docs.aws.amazon.com/qldb/latest/developerguide/streams.records.html
Now, we have come to the final section
Creating a consumer application and consuming the Data Stream in a Lambda
- Create a lambda function
- Create a lambda function with the below code and perform a serverless deploy App.js File
- Open the roles page in the IAM console.
- Choose Create role.
- Create a role with the following properties.
Trusted entity — AWS Lambda.
Permissions — AWSLambdaKinesisExecutionRole.
Role name — lambda-kinesis-role.(Kindly refer the below Image)
The AWSLambdaKinesisExecutionRole policy has the permissions that the function needs to read items from Kinesis and write logs to CloudWatch Logs.
- After creating both lambda and above role, now we can attach the above role to the lambda we created
- I have attached the role we created to the Lambda (Please refer below image)
- Now, I will add Kinesis stream that we created as a trigger to my lambda function
- Click on add to create a Kinesis trigger.
- Now, for the ledger what we have chosen to stream, create a table ‘transactiondetails’ and insert the document as below:
insert into transactiondetails
{
‘accountNo’ : ‘0002’,
‘amount’ : ‘10000’,
‘transactionId’ : ‘TXN001’,
‘transactionType’ : ‘DEBIT’,
‘timeStamp’ : ‘12:00:00’
}
- After running the above command in the query editor, if we check the CloudWatch logs of our lambda, we can see the streamed data has been printed out (Please refer to the below image).
.webp)
.webp)
