Join Transform 2021 for the most important themes in enterprise AI & Data. Learn more.


Apps in the cloud each have their own requirements and sometimes need a different view of data in a dataset. For example, an ecommerce database might include personally identifiable information that isn’t required when the data is processed for analytics. On the other side, if the ecommerce dataset is used for a marketing campaign, it might need to be enriched with additional details like information from a customer loyalty database.

To address this modality challenge, Amazon today launched S3 Object Lambda, a new capability that allows developers to add their own code to process data retrieved from Amazon Web Services (AWS) Simple Storage Service (S3) before returning it to an app. Amazon says that S3 Object Lambda works with existing apps, leveraging AWS Lambda functions to automatically transform data as it’s being retrieved from an S3 bucket.

Prior to the rollout of S3 Object Lambda, AWS offered two primary ways to provide different views of data to multiple applications. Enterprises could create, store, and maintain additional derivative copies of the data, so that each app had its own custom dataset tailored to its needs. Alternatively, they could build and manage infrastructure as a proxy layer in front of S3 to intercept and process data as it was requested.

But both options added complexity and costs.

AWS S3 Object Lambda

With S3 Object Lambda, Amazon says that companies can more easily automate tasks like converting across data formats, compressing or decompressing files as they’re being downloaded, resizing and watermarking images, and implementing custom authorization rules to access databases. Using S3 Object Lambda, developers can present multiple views from the same dataset and update the necessary Lambda functions to modify these views at any time.

“When retrieving an object using S3 Object Lambda, there is no need for an object with the same name to exist in the S3 bucket. The Lambda function can use information in the name of the file or in the HTTP headers to generate a custom object,” AWS chief evangelist Danilo Poccia explained in a blog post. “This new capability makes it much easier to share and convert data across multiple applications.”

S3 Object Lambda is available today in all AWS Regions excepting Asia Pacific (Osaka), AWS GovCloud (US-East), AWS GovCloud (US-West), China (Beijing), and China (Ningxia) Regions. Customers can use S3 Object Lambda with the AWS Management Console, AWS Command Line Interface, and AWS SDKs. They pay for the AWS Lambda compute and request charges required to process the data, the data S3 Object Lambda returns to their applications, and S3 requests invoked by Lambda functions.

To begin using S3 Object Lambda, follow these steps suggested by Amazon:

  1. Create a Lambda Function to transform data for your use case.
  2. Create an S3 Object Lambda Access Point from the S3 Management Console.
  3. Select the Lambda function that you created above.
  4. Provide a supporting S3 Access Point to give S3 Object Lambda access to the original object.
  5. Update your application configuration to use the new S3 Object Lambda Access Point to retrieve data from S3.

VentureBeat

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.

Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:

  • up-to-date information on the subjects of interest to you
  • our newsletters
  • gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
  • networking features, and more

Become a member