Twitter recently announced the new Twitter API v2, rebuilt from the ground up to deliver new features faster to better serve all developers. The latest release includes new endpoints to listen to and analyze the public conversation. This new API allows for a new way of requesting the Tweet payload, giving you greater control by allowing you to request specific fields that you might be interested in. The new Tweet payload also includes additional new fields that are not part of the v1.1 payload (that you get from endpoints such as statuses/filter and statuses/show).
In this article, we will take a close look at the v1.1 Tweet payload (from v.1.1) and compare it with the new v2 Tweet payload (v2).
Comparison of the old Tweet payload and the new Tweet payload
If you have used endpoints that are part of the v.1.1 Twitter API, then you may have seen the Tweet payload that is part of the API response. The new Twitter API provides a new way to build the Tweet payload that you want returned, giving you more control over the specific data that you need. By default, the new endpoints only return the Tweet ID and text. Any additional fields can be explicitly specified and requested using expansions and fields. You can learn more about how to use expansions and fields from our documentation. Here is an example of the new Tweet payload when fully hydrated:
{
"data": {
"author_id": "2244994945",
"context_annotations": [
{
"domain": {
"id": "46",
"name": "Brand Category",
"description": "Categories within Brand Verticals that narrow down the scope of Brands"
},
"entity": {
"id": "781974596752842752",
"name": "Services"
}
},
{
"domain": {
"id": "47",
"name": "Brand",
"description": "Brands and Companies"
},
"entity": {
"id": "10045225402",
"name": "Twitter"
}
},
{
"domain": {
"id": "65",
"name": "Interests and Hobbies Vertical",
"description": "Top level interests and hobbies groupings, like Food or Travel"
},
"entity": {
"id": "848920371311001600",
"name": "Technology",
"description": "Technology and computing"
}
},
{
"domain": {
"id": "66",
"name": "Interests and Hobbies Category",
"description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations"
},
"entity": {
"id": "848921413196984320",
"name": "Computer programming",
"description": "Computer programming"
}
}
],
"created_at": "2020-07-02T17:48:57.000Z",
"entities": {
"annotations": [
{
"start": 23,
"end": 29,
"probability": 0.4636,
"type": "Organization",
"normalized_text": "Twitter"
}
],
"hashtags": [
{
"start": 106,
"end": 117,
"tag": "TwitterAPI"
}
],
"urls": [
{
"start": 188,
"end": 211,
"url": "https://t.co/WvjuEWCa6G",
"expanded_url": "https://blog.twitter.com/developer/en_us/topics/tools/2020/a-year-with-twitter-developer-labs.html",
"display_url": "blog.twitter.com/developer/en_u…",
"images": [
{
"url": "https://pbs.twimg.com/news_img/1278747527043362816/7HQRkQeV?format=jpg&name=orig",
"width": 1600,
"height": 600
},
{
"url": "https://pbs.twimg.com/news_img/1278747527043362816/7HQRkQeV?format=jpg&name=150x150",
"width": 150,
"height": 150
}
],
"status": 200,
"title": "A year with Twitter Developer Labs: What we've learned and changed",
"description": "Labs has been invaluable in helping us understand what works well and what doesn’t, what you liked and what you didn’t.",
"unwound_url": "https://blog.twitter.com/developer/en_us/topics/tools/2020/a-year-with-twitter-developer-labs.html"
}
]
},
"id": "1278747501642657792",
"lang": "en",
"possibly_sensitive": false,
"public_metrics": {
"retweet_count": 64,
"reply_count": 13,
"like_count": 148,
"quote_count": 19
},
"source": "Twitter Web App",
"text": "It's been a year since Twitter's Developer Labs launched.\n\nAs we build towards the next generation of the #TwitterAPI (coming VERY soon), see what we've learned and changed along the way. https://t.co/WvjuEWCa6G"
},
"includes": {
"users": [
{
"created_at": "2013-12-14T04:35:55.000Z",
"description": "The voice of Twitter's #DevRel team, and your official source for updates, news, & events about Twitter's API. \n\n#BlackLivesMatter",
"entities": {
"url": {
"urls": [
{
"start": 0,
"end": 23,
"url": "https://t.co/3ZX3TNiZCY",
"expanded_url": "https://developer.twitter.com/en/community",
"display_url": "developer.twitter.com/en/community"
}
]
},
"description": {
"hashtags": [
{
"start": 23,
"end": 30,
"tag": "DevRel"
},
{
"start": 113,
"end": 130,
"tag": "BlackLivesMatter"
}
]
}
},
"id": "2244994945",
"location": "127.0.0.1",
"name": "Twitter Dev",
"pinned_tweet_id": "1255542774432063488",
"profile_image_url": "https://pbs.twimg.com/profile_images/1267175364003901441/tBZNFAgA_normal.jpg",
"protected": false,
"public_metrics": {
"followers_count": 507830,
"following_count": 1867,
"tweet_count": 3565,
"listed_count": 1562
},
"url": "https://t.co/3ZX3TNiZCY",
"username": "TwitterDev",
"verified": true
}
]
}
}
Updates to existing fields
Tweet Text
The v1.1 payload returned the truncated Tweet text (for Tweets over 140 characters) by default, and also returned a Boolean field to identify whether the Tweet is truncated or not. In order to get the full Tweet text, you needed to use the tweet_mode = extended parameter and then do a conditional check on the truncated field. With the new v2 Tweet payload, you do not need to do this check. The Tweet text for all Tweets (including 140+ characters) is returned in the 'text' field and can be used directly.
Tweet Id
| Old Tweet Payload | New Tweet Payload |
|---|---|
| Returns the Tweet ID in two fields: 1. 'id' which is an integer 2. 'id_str' which is the string form of the ID |
Returns a single 'id' field as a string |
Entities
If a Tweet contains a URL, you can request information about it in the new v2 Tweet payload using 'entities' as a value for the tweet.fields parameter. This will provide a 'urls' object on your Tweet object (nested in the 'data' object). For a URL in the Tweet, you will be able to get the URL, the title, description, and unwounded_url as shown below:
{
"entities": {
"mentions": [
{
"start": 117,
"end": 129,
"username": "suhemparack"
}
],
"urls": [
{
"start": 253,
"end": 276,
"url": "https://t.co/qwVOgw0zSV",
"expanded_url": "https://twittercommunity.com/t/using-twitter-developer-labs-filtered-stream-to-study-covid-19/138613",
"display_url": "twittercommunity.com/t/using-twitte…",
"images": [
{
"url": "https://pbs.twimg.com/news_img/1270417574266900490/bIXurp5s?format=jpg&name=orig",
"width": 480,
"height": 360
},
{
"url": "https://pbs.twimg.com/news_img/1270417574266900490/bIXurp5s?format=jpg&name=150x150",
"width": 150,
"height": 150
}
],
"status": 200,
"title": "Using Twitter Developer Labs filtered stream to study COVID-19",
"description": "If you don’t have access to the covid-19 stream endpoint, you can still study some of this data using the Twitter Developer Labs filtered stream endpoint and the covid-19 annotation. Check out the video tutorial below, that teaches you how to get a sample of the data about covid-19 using the filtered stream endpoint and the covid-19 annotation. Watch this video here: Note: The filtered stream endpoint only provides ~1% of the conversation that matches your filtering rule, so the volume is o...",
"unwound_url": "https://twittercommunity.com/t/using-twitter-developer-labs-filtered-stream-to-study-covid-19/138613"
}
]
}
}
Updates to field names
Updates have been made to naming of certain fields as well. These include:
| Old Payload | New Payload |
|---|---|
| favorite_count | like_count |
| screen_name | username |
| statuses_count | tweet_count |
| friends_count | following_count |
Updated Tweet payload structure
As you can see in the example above, the overall structure of both the payloads is different. The v1.1 payload has one single object that contains information about the Tweet such as the text and id, as well the user object, with information such as the id and name. The new v2 payload is structured differently. It primarily contains two objects: 'data' and 'includes.'
The sample below showcases the structural differences between the old and new payload, highlighting the breakdown of the new payload by 'data' and 'includes:'
| Sample Tweet (old payload) structure | Sample Tweet (new payload) structure |
|---|---|
{
"id": 1270417572001976322,
"id_str": "Sample Tweet ID",
"text": "Sample Tweet Text",
"user": {
...
},
"place": {
...
}
}
|
{
"data": {
"id": "1270417572001976322",
"text": "Sample Tweet Text"
},
"includes": {
"users": [
{
...
}
],
"places": [
{
...
}
]
}
}
|
User
Unlike the v1.1 Tweet payload, the User object is not included directly in the main payload, and has to be requested separately using the expansions parameter when using the Twitter API v2. The requested User object, along with the requested fields is returned nested in the 'includes' object.
Place
Similarly, in the v2 of the API, the information for the Place object can also be requested separately, and if present, is also returned nested in the 'includes' object of the new Tweet payload. An example of this object is shown below:
{
"places": [
{
"country": "United States",
"country_code": "US",
"full_name": "Manhattan, NY",
"geo": {
"type": "Feature",
"bbox": [
-74.026675,
40.683935,
-73.910408,
40.877483
],
"properties": {}
},
"id": "01a9a39529b27f36",
"name": "Manhattan",
"place_type": "city"
}
]
}
Quoted Tweet
In the v1.1 payload, a 'quoted status' object was returned in the Tweet payload for a Tweet object that was a Quote Tweet. In the new v2 Tweet payload, it can be requested separately, and is returned nested in the 'includes' object of the new Tweet payload.
Public Metrics
In the v1.1 Tweet payload, you can get metrics about the Tweet such as favorite_count and retweet_count from the Tweet object itself. In the new v2 Tweet payload, you can request a 'public_metrics'; object.' In addition to retweet_count and like_count (renamed from favorite_count), this will include additional fields such as quote_count and reply_count. See the example below:
{
"public_metrics": {
"retweet_count": 13,
"reply_count": 3,
"like_count": 39,
"quote_count": 1
}
}
Note : the quote_count field in the public_metrics object is a new field that identifies the number of Quote Tweets. More information on this new field can be found in our forums.
Similarly, the user object in the v1.1 payload contains fields like followers_count and friends_count. With the new v2 payload, you can request the public_metrics object on the user object that will include fields such as:
{
"public_metrics": {
"followers_count": 507944,
"following_count": 1865,
"tweet_count": 3560,
"listed_count": 1546
}
}
New objects introduced with Twitter API v2
Non-public & organic metrics
The new v2 Tweet payload includes the 'non_public_metrics,' which includes 'metrics' totals such as number of impressions, video view quartiles, 'organic_metrics' and 'promoted_metrics' objects (all of which require OAuth 1.0a User Context authentication). Learn more about available metrics here.
Polls
The new Tweet payload includes a 'polls' object where applicable, which includes information about Tweets with polls, such as the number of voters per option, and duration of the poll. This information is not provided in the v1.1 Tweet payload.
Context annotations
The new Tweet payload includes a new object called 'context_annotations' that can be requested using the 'context_annotations' value for the tweet.fields parameter. This provides contextual information to help you understand what the Tweet is about without needing to do custom entity recognition.
For example, a Tweet payload about the NFL may include the following object for context_annotations:
{
"context_annotations": [
{
"domain": {
"id": "47",
"name": "Brand",
"description": "Brands and Companies"
},
"entity": {
"id": "10042758256",
"name": "NFL"
}
}
]
}
Each object within context_annotations contains a 'domain' object and an 'entity' object, and each of those have an id and name property. The domain indicates the high level category or topic under which the Tweet falls, and the entity indicates the person, place, or organization that is recognized from the Tweet text. More details on available annotations can be found on our documentation page.
Getting the original Tweet for 'in_reply_to'
One major difference between the v1.1 and new v2 Tweet payloads is the payload structure for Tweets that are 'in reply to' another Tweet.
- The v1.1 Tweet payload does not include the Tweet information for the original Tweet in reply to which this Tweet was posted. So, if you wanted to get the Tweet information for both the Tweet and the Tweet in reply to it, you would have to make another API call, using the in_reply_to_status_id
- With the new v2 Tweet payload, you can request the original Tweet and the reply, as both objects will be part of the returned Tweet payload, with one single call, using the referenced_tweets.id expansions.
Below is an example of the new Tweet payload for a Tweet that was in reply to another Tweet:
{
"data": {
"author_id": "2244994945",
"context_annotations": [
{
"domain": {
"id": "123",
"name": "Ongoing News Story",
"description": "Ongoing News Stories like 'Brexit'"
},
"entity": {
"id": "1220701888179359745",
"name": "COVID-19"
}
},
{
"domain": {
"id": "65",
"name": "Interests and Hobbies Vertical",
"description": "Top level interests and hobbies groupings, like Food or Travel"
},
"entity": {
"id": "848920371311001600",
"name": "Technology",
"description": "Technology and computing"
}
},
{
"domain": {
"id": "66",
"name": "Interests and Hobbies Category",
"description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations"
},
"entity": {
"id": "848921413196984320",
"name": "Computer programming",
"description": "Computer programming"
}
}
],
"created_at": "2020-04-29T17:01:40.000Z",
"entities": {
"urls": [
{
"start": 243,
"end": 266,
"url": "https://t.co/j725kKXuXs",
"expanded_url": "https://docs.google.com/forms/d/e/1FAIpQLSdX3znRD87_j1E3aOUHAZtxzJfHwXzpFPKZTBiB90_9vJLscA/viewform",
"display_url": "docs.google.com/forms/d/e/1FAI…",
"images": [
{
"url": "https://pbs.twimg.com/news_img/1269516220459515905/TZm5wJPQ?format=jpg&name=orig",
"width": 1200,
"height": 630
},
{
"url": "https://pbs.twimg.com/news_img/1269516220459515905/TZm5wJPQ?format=jpg&name=150x150",
"width": 150,
"height": 150
}
],
"status": 200,
"title": "Twitter COVID-19 data application",
"description": "Please complete this form to apply for access to the COVID-19 stream endpoint. As the conversation around this topic has surged in recent weeks, this dataset is notably large and requires experience working with Twitter data and substantial infrastructure to ingest, process, store, and analyze the volume. Please provide as much detail in English as you can, as this will help speed our review process. Note that we are unable to process applications that do not contain complete and detailed answers to each question listed. Requirements: * You have a Twitter developer account (https://developer.twitter.com/en/portal/petition/essential/basic-info) * You have demonstrable experience working with Twitter data in JSON format * You have a clear plan and ability to support the ingestion, processing, storage, and analysis of streaming data at a scale of tens of millions of Tweets per day * Your project is adherent with our Developer Terms and Restricted Use Cases (https://developer.twitter.com/en/developer-terms) * ",
"unwound_url": "https://docs.google.com/forms/d/e/1FAIpQLSdX3znRD87_j1E3aOUHAZtxzJfHwXzpFPKZTBiB90_9vJLscA/viewform"
}
]
},
"id": "1255542782099255296",
"in_reply_to_user_id": "2244994945",
"lang": "en",
"possibly_sensitive": false,
"public_metrics": {
"retweet_count": 35,
"reply_count": 17,
"like_count": 66,
"quote_count": 6
},
"referenced_tweets": [
{
"type": "replied_to",
"id": "1255542774432063488"
}
],
"source": "Twitter Web App",
"text": "The volume of conversation about COVID-19 is tremendous, which means it requires expertise and computational resources to process. \n\nDevelopers and researchers with that capability and intent to support the public good can apply for access. \n\nhttps://t.co/j725kKXuXs"
},
"includes": {
"users": [
{
"created_at": "2013-12-14T04:35:55.000Z",
"description": "The voice of Twitter's #DevRel team, and your official source for updates, news, & events about Twitter's API. \n\n#BlackLivesMatter",
"entities": {
"url": {
"urls": [
{
"start": 0,
"end": 23,
"url": "https://t.co/3ZX3TNiZCY",
"expanded_url": "https://developer.twitter.com/en/community",
"display_url": "developer.twitter.com/en/community"
}
]
},
"description": {
"hashtags": [
{
"start": 23,
"end": 30,
"tag": "DevRel"
},
{
"start": 113,
"end": 130,
"tag": "BlackLivesMatter"
}
]
}
},
"id": "2244994945",
"location": "127.0.0.1",
"name": "Twitter Dev",
"pinned_tweet_id": "1255542774432063488",
"profile_image_url": "https://pbs.twimg.com/profile_images/1267175364003901441/tBZNFAgA_normal.jpg",
"protected": false,
"public_metrics": {
"followers_count": 507676,
"following_count": 1863,
"tweet_count": 3563,
"listed_count": 1552
},
"url": "https://t.co/3ZX3TNiZCY",
"username": "TwitterDev",
"verified": true
}
],
"tweets": [
{
"author_id": "2244994945",
"context_annotations": [
{
"domain": {
"id": "123",
"name": "Ongoing News Story",
"description": "Ongoing News Stories like 'Brexit'"
},
"entity": {
"id": "1220701888179359745",
"name": "COVID-19"
}
},
{
"domain": {
"id": "46",
"name": "Brand Category",
"description": "Categories within Brand Verticals that narrow down the scope of Brands"
},
"entity": {
"id": "781974596752842752",
"name": "Services"
}
},
{
"domain": {
"id": "47",
"name": "Brand",
"description": "Brands and Companies"
},
"entity": {
"id": "10045225402",
"name": "Twitter"
}
}
],
"created_at": "2020-04-29T17:01:38.000Z",
"entities": {
"annotations": [
{
"start": 54,
"end": 60,
"probability": 0.6094,
"type": "Product",
"normalized_text": "Twitter"
}
],
"urls": [
{
"start": 264,
"end": 287,
"url": "https://t.co/BPqMcQzhId",
"expanded_url": "https://blog.twitter.com/developer/en_us/topics/tools/2020/covid19_public_conversation_data.html",
"display_url": "blog.twitter.com/developer/en_u…",
"images": [
{
"url": "https://pbs.twimg.com/news_img/1274730688919613440/SEsqlE16?format=jpg&name=orig",
"width": 1600,
"height": 600
},
{
"url": "https://pbs.twimg.com/news_img/1274730688919613440/SEsqlE16?format=jpg&name=150x150",
"width": 150,
"height": 150
}
],
"status": 200,
"title": "Enabling study of the public conversation in a time of crisis",
"description": "To further support our company’s ongoing efforts, we want to enable developers and researchers to study the public conversation on COVID-19 in real-time.",
"unwound_url": "https://blog.twitter.com/developer/en_us/topics/tools/2020/covid19_public_conversation_data.html"
}
]
},
"id": "1255542774432063488",
"lang": "en",
"possibly_sensitive": false,
"public_metrics": {
"retweet_count": 327,
"reply_count": 308,
"like_count": 710,
"quote_count": 120
},
"source": "Twitter Web App",
"text": "During these unprecedented times, what’s happening on Twitter can help the world better understand & respond to the pandemic. \n\nWe're launching a free COVID-19 stream endpoint so qualified devs & researchers can study the public conversation in real-time. https://t.co/BPqMcQzhId"
}
]
}
}
Summary of the contrast between fields and objects of the v1.1 and new v2 Tweet payloads
Below is a summary of the comparison of objects and fields in the v1.1 Tweet payload and the new v2 Tweet payload:
| Objects/Fields | v1.1 Tweet Payload | New Tweet Payload |
|---|---|---|
| user | Part of the returned Tweet object | Requested with expansions and is nested in 'includes' object of the returned Tweet object |
| places | Part of the returned Tweet object | Requested with expansions and is nested in 'includes' object of the returned Tweet object |
| quoted_status | Part of the returned Tweet object | Requested with expansions and is nested in 'includes' object of the returned Tweet object |
| polls | Not supported | Requested with expansions and is nested in 'includes' object of the returned Tweet object |
| Original Tweet (in reply to) | Not supported | Requested with expansions and is nested in 'includes' object of the returned Tweet object |
| context_annotations | Not supported | Provided in the 'data' object of the returned Tweet object |
| public_metrics | Some available as individual fields as part of the returned Tweet object | A new public_metrics object is provided in the 'data' object of the returned Tweet object |
| non_public_metrics & organic_metrics | Not supported | A new non_public_metrics object is provided in the 'data' object of the returned Tweet object |
These are some of the key differences between the v1.1 Tweet payload and the new payload available with v2. With the introduction of expansion and fields, we aim to provide developers a higher degree of customization to scope and limit the fields you need based on your use case. Below are some resources to help you get started with exploring the new v2 Tweet payload. Reach out to me on Twitter @suhemparack or @TwitterDev or on our community forums with feedback or questions.
Resources to help you get started
- Check out the video tutorial explaining the differences between the v1.1 Tweet payload and the new v2 Tweet payload.
- A complete list of all the fields that are part of the new v2 Tweet payload and what each of those can be used for can be found on our documentation for the data dictionary for the object model.
- Learn more about how to use fields and expansions to get the data you want returned in the Tweet payload.
- A list of code samples for getting started with the new Twitter API (that return the new Tweet payload) can be found on Github.
- Check out this hydrator tool on Glitch that retrieves the new v2 Tweet payload for a given Tweet ID.
Top comments (0)