Providing User Data
Introduction
In addition to ad network data about the ads you serve, AlgoLift requires data generated by your product that is indicative of user behavior and performance.
require data on Attribution, Revenue, and User Engagement in a standardized format that is described below. This data is aggregated and delivered daily, with each daily delivery landing at a unique location for that day's data. Instructions on delivering this data are likewise detailed below.
Note: Please exclude test or QA user data from the files in order to ensure accuracy and optimal results. You can also provide a list of users to blacklist as well.
Data History
AlgoLift provides the most accurate LTV model when we have 2 years+ of data. This enables us to observe temporal changes in revenue and user behavior and to fully understand how cohorts mature.
We can train our LTV model with less historical data but this restricts the projection windows based on the amount of data available to train our models. Below is a schedule for how AlgoLift releases new projection windows to clients based on the historical data available for modeling:
| Days of Historical Data | Projection Window |
|---|---|
| 45 | D30 |
| 70 | D60 |
| 95 | D90 / D180 |
| 105 | D365 |
Data Shape
Attribution Data
Attributions, which are closely tied to installs (which represent the beginning of a user's lifetime within the app or product), are the events when a given user can be attributed to a specific ad in a campaign if they were acquired through advertising, or otherwise can be labeled 'organic.' When preparing Attribution data for export to AlgoLift, please format it according to the below schema.
All columns present in the schema should be present in the corresponding file. Any columns for which data is unavailable should be left blank, but not omitted.
| Field | Data Type | Required | Description |
|---|---|---|---|
| user_id* | varchar(255) | Yes | A common user identifier used across all data tables |
| impression_dt | datetime | No | The timestamp of the ad impression event the user is attributed to (ISO-8601 compliant datetime string) |
| install_dt | datetime | Yes | The timestamp of the user's install event (ISO-8601 compliant datetime string) |
| country | varchar(2) | Yes | The country in which the user resides (ISO-3166 alpha-2 country code) |
| platform | varchar(255) | Yes | The platform of the user's device |
| source | varchar(255) | Yes | The ad network to which the user has been attributed. Should be Unattributed Facebook for Facebook VTA users who are not attributed to any specific campaign. |
| subpub_id | varchar(255) | No | A unique anonymous identifier for a sub-publisher provided by your MML |
| subpub_bundle_id | varchar(255) | No | The app store bundle id (com.app.pub, or 1234534534) |
| subpub_name | varchar(255) | No | The proper name for a sub-publisher (i.e., Bubbles Game) |
| campaign_id | varchar(255) | No | The id of the campaign to which the user has been attributed |
| campaign_name | varchar(255) | No | The name of the campaign to which the user has been attributed |
| adgroup_id | varchar(255) | No | The id of the ad group to which the user has been attributed |
| adgroup_name | varchar(255) | No | The name of the ad group to which the user has been attributed |
| ad_id | varchar(255) | No | The id of the ad to which the user has been attributed |
| ad_name | varchar(255) | No | The name of the ad to which the user has been attributed |
| ad_type | varchar(255) | No | The type of creative user has been attributed valid types: text - an ad unit containing only text, e.g. a search result banner - a basic format that appears at the top or bottom of the device screen interstitial - a full-page ad that appears during breaks in the current experience video - a standard video, i.e. non-rewarded rewarded_video - an ad unit offering in-app rewards in exchange for watching a video playable - an ad unit containing an interactive preview of the app experience sponsored_content - a link included in a piece of sponsored content, like an advertorial article audio - an audio ad |
| keyword_id | varchar(255) | No | The id of the keyword to which the user has been attributed |
| keyword_name | varchar(255) | No | The name of the keyword to which the user has been attributed |
| device_brand | varchar(255) | No | The brand of the user's device |
| device_model | varchar(255) | No | The model of the user's device |
| os_version | varchar(255) | No | The version of the operating system of the user's device |
| custom1 | varchar(255) | No | An arbitrary, client-defined, field. Will be carried through the pipeline as a dimension and will be present in the portal as a segmentable field. |
| custom2 | varchar(255) | No | An arbitrary, client-defined, field. Will be carried through the pipeline as a dimension and will be present in the portal as a segmentable field. |
| custom3 | varchar(255) | No | An arbitrary, client-defined, field. Will be carried through the pipeline as a dimension and will be present in the portal as a segmentable field. |
*user_id is a non-PII (ie not IDFA/GAID) identifier that uniquely identifies a given user of the app or product. This can be an identifier generated by the client, or a third party identifier provided by a third party such as an MMP or Analytics provider. Any identifier that follows the following rules can be used:
- The identifier is available at the time of attribution.
- All subsequent revenue and engagement events for a given user are provided to AlgoLift using the same identifier as that initially provided in that user’s attribution event. Stated another way, all revenue and engagement events should contain a user_id that has previously been included in an attribution event.
- The identifier is the one AlgoLift will return in any output data.
Not all ad networks use the same nomenclature. The below table shows the names under which you can find certain fields for a given ad network.
AlgoLift Apple Search Ads Google Ads Campaign Campaign Campaign Campaign Ad Group Ad Set Ad Group Ad Group Ad Ad Creative Set Ad Keyword N/A Keyword N/A
Revenue Data
Revenue represents revenue generated from the user, either from in-app purchases, ongoing-subscriptions, or payments derived from ads presented to the user. When formatting Revenue data for export to AlgoLift, please format it according to the following schema:
| Field | Data Type | Required | Description |
|---|---|---|---|
| user_id | varchar(255) | Yes | A common user identifier used across all data tables |
| revenue_type | varchar(255) | Yes | The type of the revenue source valid types: [adrev, iap, subscription] |
| revenue | double | Yes | Aggregate revenue amount |
| transaction_count | int | Yes | Number of transactions |
| parcel_name | varchar(255) | Yes* | Unique identifier of the transaction. For iAP this should be the item(s) sold, for AdRev this should be the ad_type of the ad viewed (banner, rewarded_video, etc). |
* Only for revenue_types iap or subscription
User Engagement Data
User Engagements represent user actions within the app that meaningfully distinguish a user's relationship with the app from other users. Many of these actions are common across many or all apps, such as starting a session or utilizing a social sharing feature. Others may be highly specific to your app alone. You are encouraged to include whatever actions you feel are most important; if it is unclear to you which actions may be most useful, please contact us and we can help consult on the question. When formatting User Engagement data for export to AlgoLift, please format it according to the following schema:
| Field | Data Type | Required | Description |
|---|---|---|---|
| user_id | varchar(255) | Yes | A common user identifier used across all data tables |
| engagement_type | varchar(255) | Yes | The type of the engagement event |
| engagement_count | int | Yes | The number of occurances of the engagement event |
Data Delivery
The volume of the above described data can be significant. Having a reliable way to regularly deliver data at scale is imperative for the functioning of AlgoLift's prediction and optimization products. We currently rely on reading and writing to Amazon S3 in order to transfer data to and from clients. The instructions for how to do so are detailed below, along with technical formatting information.
Write to AlgoLift's S3
When you begin with AlgoLift, we will create an S3 bucket specifically for housing your data. The convention we use for bucket naming is algolift-<client>. Sending us data is as simple as writing that data to the appropriate location in the S3 bucket. In order to enable this transfer method send a message to integrations@algolift.com with the AWS role that you will be using for writing to the bucket. We will grant the appropriate bucket permissions to the role (don't forget to grant the appropriate user or role permissions on your end as well), and you may begin writing data directly to it.
Data should be written to the following locations:
- Attribution:
s3://algolift-<client>/<app>/ingestion/daily_data/attribution/date=<date>/ - Revenue:
s3://algolift-<client>/<app>/ingestion/daily_data/revenue/date=<date>/ - Engagement:
s3://algolift-<client>/<app>/ingestion/daily_data/user_engagement/date=<date>/
Data should be delivered daily and should be delivered as soon as possible after 00:00:00 UTC. All dates should be in the format of
YYYY-MM-DD. If you only operate a single app, the stringappis sufficient for the app parameter in the paths above.
Data Formatting
- All data should be formatted in one or more CSV files. These files can be named as you like, but should have the
.csvextension and be placed in the appropriate S3 bucket and prefix for the app, data type, and date they represent. - The first row of each CSV file should be a header row that contains the column names for the ensuing rows.
- All field entries should be double quoted according to the CSV spec.
- Text fields without data should be sent as an empty field, not as the string
NULL - Required numeric fields should never be
NULL. If there is no data available, pass0as the value for the field. Other numeric fields can be passed asNULL(empty fields, as before). - Data should be deduplicated prior to transfer.
- Aggregating the data by day prior to upload will reduce data size and therefore transfer times and resulting storage space.