Introduction
Twitter data is used by developers, students, and researchers to study various topics. You may want to study the conversation around a topic using hashtags, set of keywords, etc. from the last few days. For example, maybe you want to study the sentiment of Tweets about a football game over the weekend. You can get data for such use cases using Twitter’s recent search endpoint. This endpoint gives you conversations from Twitter for the last 7 days.
In this guide, you will learn how to:
- Identify past conversation you wish to study
- Connect and authenticate to the recent search endpoint to receive Tweets
- Analyze the data for past conversations
Prerequisites
- You must have a developer account. Access is available with active keys and tokens for a developer App that is attached to a Project created in the developer portal.
- A Bearer Token from your App in the Twitter developer portal
Steps to consider
Step 1: Identify which past conversation you wish to study
As a first step, you need to identify the type of conversation you wish to study. For example, you may decide to study all Tweets about the climate for the last few days. In order to do this, you might be looking for Tweets with a certain hashtag or words. Once you have identified these words, you will need to convert these into a query (also referred to as rule or filter). You can define the rule to include/exclude keywords, images, etc. You can also specify certain account names to get Tweets from. Check out this guide that shows you how to use the operators that are supported by the recent search endpoint that lets you get the data you need.
Step 2: Authenticate and connect to the recent search endpoint to receive Tweets
Once you have decided what data you want and what operators you will use, you can connect to the recent search endpoint. The example below shows how you can use cURL to do so. To authenticate, replace $BEARER_TOKEN with the bearer token from your App in the developer portal and $QUERY using your query.
curl -X GET -H "Authorization: Bearer $BEARER_TOKEN" "https://api.twitter.com/2/tweets/search?tweet.fields=created_at,author_id,lang&query=$QUERY"
So for example, if you wish to get all Tweets with images about the hashtag #Caturday, excluding retweets your query will look like:
“#caturday has:images -is:retweet”
The '-' before a operator indicates negation, so the rule above excludes Tweets that are retweets.
And thus, your cURL call will be as follows:
curl -X GET -H "Authorization: Bearer $BEARER_TOKEN" "https://api.twitter.com/labs/2/tweets/search?query=caturday%20has:images%20-is:retweet&tweet.fields=created_at,author_id,lang"
You can also connect to the recent search endpoint using one of our code samples:
The cURL call will return 10 Tweets by default. You can also specify a maximum number of results you want to receive per API call. To do so, you can use the max_results query parameter with your API call. In order to get additional Tweets, you need to handle pagination. To do so, you will need to access the next_token from the object called meta and connect to, get the additional Tweet objects back for the next 10 Tweets (by default, unless you specify a number with the max_results parameter, which can be up to 100) until there is no longer a next_token available.
By default, you will get back the id and text of each Tweet. However, you can customize what additional fields you want to return as part of the Tweet payload. You can include these additional fields in the payload by adding additional fields and expansions to your query.
Instead of making curl calls from your terminal, you can use a REST client such as Postman or Insomnia to get data from the recent search endpoint. Our Postman collection will help you get started quickly with calling the recent search endpoint.
Step 3: Analyzing the data for past conversations
The Twitter API provides a good opportunity to study historical data. Examples of analysis that you might be interested in doing with Twitter data include:
- Mapping Tweets by location
- Sentiment analysis on Tweets about past events
- Identifying influencers on Twitter
There are various libraries available in programming languages such as Python, R, etc. that let you analyze the data obtained from Twitter API.
In Python, you can use libraries such as pandas, numpy, etc. that let you do data analysis and wrangling. You can also use libraries such as matplotlib to build visualizations from Twitter data.
In R, you can use something like Tidyverse (which is a collection of R packages for data science) for data analysis. You can use ggplot2, which is part of Tidyverse to build data visualization using Twitter data.
There are a variety of visualizations that you can build using these libraries mentioned above. Some examples of such visualizations that can be built using Twitter data include:
- Histograms for displaying Tweet frequency
- Map for displaying Tweets by Geo-Location
- Time series analysis of Trends
Check out this sample app that demonstrates how to use the recent search endpoint to analyze sentiments of your own Tweets.
Resources
- Learn more about the recent search endpoint.
- Get started building with our quick start guide.
- Check out our API reference to learn more about what’s available.
- Learn more about building search queries.
- Get inspired by reading our other tutorials.
Top comments (0)