Listen for important events using the new Twitter API v2

Introduction

Twitter APIs provide a way for developers to access data about what's happening around the world. Developers use our APIs to listen for events of interest and build applications that trigger actions based on those events. For example, you may want to listen for Tweets containing certain words (breaking news) from specific user accounts and then get notified via SMS anytime those accounts Tweet with those keywords.

Using the filtered stream and sampled stream endpoints, you can listen for important events from public Tweets as those events unfold in real-time. The filtered stream endpoint returns a specific set of Tweets matching the criteria that you set while the sampled stream endpoint returns about a 1% random sample of all public Tweets in real-time.

So, using the filtered stream endpoint, you can pre-define the criteria for the Tweets that you are looking for. As for the sampled stream endpoint, you can start consuming the random 1% sample and then from the response identify the Tweets of interest to you. Once you have Tweets about those events, you can trigger appropriate actions or notifications based on those.

In this guide, you will learn how to:

• Identify the kinds of Tweets you would like to listen to
• Authenticate and connect to the appropriate endpoint to receive Tweets
• Trigger actions or notifications based on the Tweets of interest that you listen for

Prerequisites

• You must have an a developer account. Access is available with active keys and tokens for a developer App that is attached to a Project created in the developer portal.
• A Bearer Token from your App in the Twitter developer portal

Steps to consider

Identify the kinds of Tweets you would like to listen to

First, you will have to identify the kinds of events you want to listen for. In order to do so you should consider some basic questions such as:

• Are you looking for Tweets based on certain hashtags or keywords?
• Do you want to include/exclude Tweets that are retweets?
• Are you specifically looking for Tweets that have media?
• Do you want Tweets in certain languages?

Once you have the answers to some of these questions, you can get the data you need using the following approaches:

• Using the sample stream endpoint, you can listen for a 1% sample of all activity happening on Twitter. Once you are connected to this stream, you can identify trends for popular topics in this stream of Tweets. Then, you can notify channels of your choice.
• Using the filtered Stream, you can explicitly create search queries and define what data you are looking for. For example, you can specify that you want all Tweets with the hashtag #Caturday and filter for only those Tweets that contain images. Once you start receiving Tweets that meet this criteria, you can send links of those Tweets via email, sms etc.

Authenticate & Connect to the appropriate endpoint to receive Tweets

Authentication

In order to start receiving data from the sampled stream or filtered stream endpoint, you need to authenticate. In order to do so, replace the $BEARER_TOKEN (including the $ symbol) in cURL calls below, with the bearer token from your app in the developer portal

Staying informed on a topic of interest

In order to stay informed about a specific topic of interest, you can use the filtered Stream endpoint. You can add rules for the filtered stream endpoint to specify the data you are looking for. In order to do so, you will have to make a POST call to the filtered stream’s create rules endpoint. An example cURL call to create a rule is as follows (replace $REPLACE_WITH_RULE with your own set of terms and operators):

curl -X POST 'https://api.twitter.com/2/tweets/search/stream/rules' -H "Content-type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{
  "add": [
    {"value": "from:twitterdev from:twitterapi has:links"}
  ]
}'

For example, a rule to get all Tweets with images about the hashtag ‘#Caturday’ excluding retweets will look like:

“#caturday has:images -is:retweet”

More information on the available operators and on building rules can be found here.

Once the rule has been added, you can connect to the filtered stream endpoint to start streaming data to your client. The example below shows how you can use cURL to do so:

curl -X GET -H "Authorization: Bearer $BEARER_TOKEN" "https://api.twitter.com/2/tweets/search/stream"

You can also connect to the filtered stream endpoint using one of our code samples:

• Python
• Ruby
• Node.js
• Java

Once you establish a successful connection to the stream, you will receive Tweets delivered in JSON format through a persistent HTTP streaming connection. You will only receive Tweets matching the rules you defined earlier while connected to the stream. If there are currently no matching Tweets, you will receive a carriage return line feed every twenty seconds to prevent your app from timing out.

Detecting current trends

In order to learn about popular topics at a given time, you can use the Sampled stream API. To connect to the Sample Stream API, issue a GET request to the endpoint. The example below shows how you can use cURL to do so.

curl -X GET -H "Authorization: Bearer $BEARER_TOKEN" "https://api.twitter.com/2/tweets/stream/sample"

You can also connect to the sampled stream endpoint using one of our code samples

• Python
• Ruby
• Node.js
• Java

Once connected, you will continually receive Tweets as they are posted. As Tweets are received on the endpoint, you can programmatically scan for and keep count of repeat occurrences of hashtags, user mentions, keywords and Annotations in a key-value store or another data store.

Trigger actions or notifications based on the Tweets of interest that you listen for

Once you have the Tweets of interest, you can decide what to do next with them. You may want to notify using a certain channel (e.g. sending an email, sending an SMS, or sending a notification to a communication app etc.) with links to the Tweets of interest.

Sending an email with the Tweet data

In order to programmatically send links to Tweets in an email, you will need to use an email service. Some examples of such services include the SendGrid Email API, AWS Simple Email Service (SES) etc. These services generally provide SDKs that make it easy for you to get started. They generally require you to verify your email addresses (to & from) and you can pass the URLs to the Tweets in the body.

Sending an SMS with the Tweet data

Your use-case may require sending the links to Tweets using SMS. Some services that let you send SMS programmatically include Twilio, AWS Simple Notification Service (SNS) etc. These services also provide SDKs that you can use in your code to get started. Twilio provides you with a phone number that you can send SMS from and you can use the URLs to the Tweets as the message body.

Sending a notification to a communication tools

You may want to send the links of Tweets to a communication tool that you might be using. Communication tools such as Slack, Discord etc. have APIs that let you send messages to channels on those apps. You may first have to create an app on those communication tools and obtain webhook urls to send messages to. Once you have the app created (and have the webhooks etc.) you can send the links to Tweets to the channels of your choice.

Note: If for your use-case, you do not want to be notified for events instantaneously as they happen, you can also use the Recent Search endpoint and call it periodically. Then, after you get data from this endpoint, you can decide on how to notify the right channels.

For an example of building a service that listens for breaking news using the filtered stream endpoint and sends notifications via Slack or SMS, check out this tutorial. This app does not require any custom coding and can be deployed to AWS.

Next steps

• Get started building with the filtered stream endpoint quick start guide
• Check out the filtered stream endpoint API reference to learn more about what’s available
• Learn how to create rules using the filtered stream endpoint's create rules endpoint
• Get started building with the sampled stream quick start guide
• Check out the sampled stream endpoint API reference to learn more about what’s available
• Read our other use case guides