Loading

Bulk Data Insertion API Quick 10-minute recipe using POSTMAN

Authored by Ishan Sinha, Lead Technical Support Consultant for Adobe Analytics at Adobe on 07th June, 2021. Feedbacks/suggestions welcome at isinha@adobe.com.

Contents

  • Introduction
  • Prepare the file
  • Setup authentication
  • File upload

Introduction

In this article, I'm going to list down the steps to set up Bulk Data Insertion API and upload test server-side data to Adobe Analytics in 10 minutes using Postman.

Prepare the file

Minimum requirements:

  • Required columns:
  1. reportSuiteID (one rsid or comma separated list of rsids)
  2. timestamp (Epoch time (1486769029) or ISO format (2021-02-10T16:23:49-07:00))
  3. userAgent
  4. Any one of pageURL, pageName, pe, queryString
  5. Note: If only "queryString" is used, at least one of pageURL, pageName, or pe must be specified in the "queryString" as a query parameter
  6. Any one of marketingCloudVisitorID, ipAddress, customerID.[customerIDType].id with customerID.[customerIDType].isMCSeed set to TRUE (customerID is used when customer IDs are set along with the visitor IDs - help doc).
  • Size requirements: Compressed files (.gz format) can be up to 100 MB. The uncompressed file size is limited to 1 GB.
  • Report suite must be set to timestamp required or timestamp optional.
  • All rows in a batch file for any given visitor must be sorted in chronological order by timestamp, from earliest to latest (not mandatory for timestamp optional).

Sample file format:

When queryString is used -

Each row correspond to one server call

When column-based row is used -

Each row correspond to one server call
  • Both the above files will populate the same data.
  • Populate the required data and columns. For more information on the supporting columns, refer to this section of the documentation.
  • Save the file in the CSV format. Compress it in .gz format.

With that, the file is ready for upload.

Setup authentication

Adobe's IMS is used to facilitate authentication for Bulk Data Insertion API. Currently, only JWT based authentication is supported.

Create integration:

  1. Go to console.adobe.io and sign in using your Adobe ID or Enterprise/Federated ID.
  2. Click "Create new project" or open any existing project. Click "Add API" icon.
  3. Select "Adobe Analytics" under "Experience Cloud" and click "Next" button.
  4. Select "Service Account (JWT)" integration and click "Next" button.
  5. Select "Generate a key pair" (Option 1) and click "Generate keypair" button.
  6. Save the generated zip file on your machine and extract its content to find the "private.key" file.
  7. Click "Next" button in the Adobe I/O console.
  8. Select all the relevant product profiles. These product profiles should grant access to the "Web Service Access" and the required report suite, dimension, metrices, etc.
  9. Click "Save Configured API" button. Select "Service Account (JWT)" from the left nav menu.
  10. You will see the details and credentials of your application integration. Take note of two tabs: "Credential details" and "Generate JWT".

Generate access token:

  1. Open the saved "private.key" file in any text editor, like Notepad++ and copy all its content - the private key. Ensure that you also copy "-----BEGIN PRIVATE KEY-----" and "-----END PRIVATE KEY-----".
  2. Go to your integration in Adobe I/O console and select "Generate JWT" tab.
  3. Paste the copied private key in the textbox with "Paste private key" placeholder.
  4. Click "Generate Token" button. Copy the generated JWT and save it as it will be used shortly.
  5. Next, open Postman and send a POST request to: https://ims-na1.adobelogin.com/ims/exchange/jwt
  6. The body of the request should contain URL-encoded parameters with your Client ID (API Key), Client Secret, and JWT: client_id={api_key_value}&client_secret={client_secret_value}&jwt_token={base64_encoded_JWT}
  7. The value of client_id, client_secret can be obtained from the "Credential details" tab of the Adobe I/O console integration. The value of jwt_token is the value saved in step 4 above.
Sample request to obtain the access token

Click on "Send". The response will contain an "access_token". Copy and save the value to be used in the next step.

File upload

Validate file before uploading to check for errors:

It's one of the best practices to validate the Bulk Data Insertion API file before uploading it to avoid any errors, in advance. Following steps can help in validating the file -

  1. Open Postman, go to Import > Raw text and paste the following Curl command: curl -X POST -H "Authorization: Bearer <IMS_ACCESS_TOKEN>" -H "x-api-key:<CLIENT_ID>" -F file=@/tmp/ingest_file.gz "https://<BDIA_HOST>/aa/collect/v1/events/validate"
  2. Click "Continue" and then "Import".
  3. In the generated request, replace <ASSIGNED HOST> in the request URL with the corresponding host. You may want to use the global endpoint: https://analytics-collection.adobe.io and it will auto-route.
  4. For anyone preferring to use regional endpoints, the value is https://analytics-collection-va7.adobe.io for the North America region and https://analytics-collection-nld2.adobe.io for Europe/Asia region.
  5. The request URL would look like https://analytics-collection.adobe.io/aa/collect/v1/events for sending data to the global endpoint.
  6. Note that, the Bulk Data Insertion API has an explicit validation endpoint - https://analytics-collection.adobe.io/aa/collect/v1/events/validate.
  7. Go to the "Headers" tab, find "Authorization" and replace <IMS_ACCESS_TOKEN> in the value with the access token generated in the previous section. Find "x-api-key" and populate its value with "CLIENT ID" obtained from the "Credential details" tab of the Adobe I/O console integration.
  8. Go to the "Body" tab, hover over the "file" item under "KEY" and choose "File" from the drop-down. Under "VALUE", click "Select Files" and choose the saved Bulk Data Insertion API file from the first section.
  9. Click "Send". You will see "file is valid" success response in case there are no issues.
  10. Refer to this section in the documentation for common issues/errors.

Upload file to send data to Analytics:

  1. Open Postman, go to Import > Raw text and paste the following Curl command: curl -X POST -H "x-adobe-vgid:prod-18" -H "Authorization: Bearer <IMS_ACCESS_TOKEN>" -H "x-api-key:<CLIENT_ID>" -F file=@/tmp/ingest_file.gz "https://<BDIA_HOST>/aa/collect/v1/events"
  2. Click "Continue" and then "Import".
  3. In the generated request, replace <ASSIGNED HOST> in the request URL with the corresponding host. You may want to use the global endpoint: https://analytics-collection.adobe.io and it will auto-route.
  4. For anyone preferring to use regional endpoints, the value is https://analytics-collection-va7.adobe.io for the North America region and https://analytics-collection-nld2.adobe.io for Europe/Asia region.
  5. The request URL would look like https://analytics-collection.adobe.io/aa/collect/v1/events for sending data to the global endpoint.
  6. Go to the "Headers" tab, find "Authorization" and replace <IMS_ACCESS_TOKEN> in the value with the access token generated in the previous section. Find "x-adobe-vgid" and give any visitor group name, for example, group1. To know more about Visitor Groups, refer to this section of the document. Find "x-api-key" and populate its value with "CLIENT ID" obtained from the "Credential details" tab of the Adobe I/O console integration.
  7. Go to the "Body" tab, hover over the "file" item under "KEY" and choose "File" from the drop-down. Under "VALUE", click "Select Files" and choose the saved Bulk Data Insertion API file from the first section.
  8. Click "Send". You will see the below successful response.
  9. Refer to this section in the documentation for common issues/errors.

I hope this was helpful in familiarizing you with the Bulk Data Insertion API setup and process. I'd recommend that you go through our detailed documentation for more information: https://www.adobe.io/apis/experiencecloud/analytics/docs.html#!AdobeDocs/analytics-2.0-apis/master/bdia.md.

Created By
Ishan Sinha
Appreciate

Credits:

Created with an image by xresch - "tech circle technology"