BotSlayer

botslayer logo

BotSlayer is an application that helps track and detect potential manipulation of information spreading on Twitter. The tool is developed by the Observatory on Social Media at Indiana University --- the same lab that brought to you Botometer and Hoaxy.

Check out our Election 2020 public instance of BotSlayer.

BotSlayer is not a tool to detect and remove likely social bots from your list of Twitter followers or friends. For that purpose, check out Botometer. If you just want to visualize the spread of some piece of information, consider Hoaxy.

BotSlayer can be used, for example, by journalists, corporations, and political candidates to discover in real-time new coordinated campaigns in their domains of interest, without any prior knowledge of these campaigns. The system is easily installed and configured in the cloud to monitor bot activity around a standing user-defined query. All you need is a Twitter developer app key to fetch data from the Twitter streaming API.

BotSlayer uses an anomaly detection algorithm to flag hashtags, links, accounts, and media that are trending and amplified in a coordinated fashion by likely bots. A Web dashboard lets users explore the tweets and accounts associated with suspicious campaigns via Twitter, visualize their spread via Hoaxy, and search related images and content on Google.

BotSlayer is designed as a crowdsourcing platform: in exchange for the free service, the system provides anonymous data back to our lab for research, in a way that is compliant with Twitter’s terms and guidelines. The data will aid in the study and early detection of social media manipulation phenomena.

We gratefully acknowledge generous support for this project from Craig Newmark Philanthropies.

Getting the Software (Currently in Beta-Testing)

BotSlayer is released under this End User License Agreement (EULA). The tools bundled with BotSlayer use these open source licenses. In addition, the following operating systems are aggregated and provided in conjunction with BotSlayer: Debian (license) and Ubuntu (license).

If you would like to install and run BotSlayer, submit this form. You must be logged into a Google account to verify your identity, and agree to the EULA. You will then receive details needed to follow the installation instructions below.

You have three installation options:

  1. The easiest is to use Amazon Web Services (go to instructions). Another advantage is that you can set up a low-power server for free. A disadvantage is that a free-tier server can only handle low-volume data streams. For example, tracking breaking news or trending hashtags will probably crash a free-tier server. You can of course pay for a more powerful server.
  2. We have a Mac installer, which makes it easy to set up BotSlayer on your laptop or desktop (go to instructions). Disadvantages: BotSlayer will use a lot of CPU and disk resources, and you must keep your Mac running and connected to the Internet 24/7.
  3. To install the software on your own server you can use Docker (go to instructions). This requires some technical knowledge about Docker.

Once you install BotSlayer and access the Web dashboard, click "Config" in the menu, enter a password of your choosing, and then provide your Twitter developer app keys and a standing query. See the Help page for further instructions and pointers about Twitter keys and query format.

Usage

After installation, BotSlayer requires user to set up their query, and input their Twitter app keys. You can obtain your app keys by creating a personal app at Twitter's developer platform (register here).

The most common type of query is to track a set of keywords and hashtags. To do so, select the track query option in the dropdown list on the config page, and put a comma-separated list of keywords and hashtags in the text box. For more complicated queries that include logical combinations and URLs, please be referred to Twitter's documentation here.

You can also set up BotSlayer to send DM alerts to you on Twitter. To do so, BotSlayer requires the Twitter app, from which you obtain your app keys, to have specific options enabled and set:

  1. Enable "Read, write, and Direct Messages" permissions; if you see '3-legged OAuth' under Authentication settings, enable it
  2. Set "Website URL" to "https://osome.iu.edu"
  3. Add "https://osome.iu.edu" to the list of "Callback URLs"
  4. Regenerate your keys
Then you can setup your own rules for how the alerts should be triggerred. Remember to save your new keys and rules .

Warnings

The dashboard is accessible via the Web using the IP address of the server. DO NOT SHARE THE IP OR URL with anyone outside your organization or anyone you do not trust. They could make changes to the system or access data in violation of Twitter terms of service. To prevent potential security issue and terms violation, BotSlayer blocks search engine indexing by default.

When accessing the Config and Alerts pages, you have to enter a password that you set when you first use your instance. WARNING: this password is not sent securely over the network. Someone who intercepts communication between your browser and server could access the password and gain control of your instance configuration. They could also change the password and lock you out. They could also steal your Twitter keys. Therefore DO NOT SHARE THE IP ADDRESS OR URL with anyone you do not trust, and only share them securely with trusted parties.

BotSlayer-CE

BotSlayer-CE (Community Edition) is the open source version of BotSlayer. Please be aware that the "CE" version uses simple heuristics to calculate bot scores, which may not be suitable for research. We strongly recommend the "Pro" version of BotSlayer introduced here, which uses proprietary BotometerLite software and has many other improvements.

Publication and Credit

If you use BotSlayer for research, please cite the following publication: Hui et al., (2019). BotSlayer: real-time detection of bot amplification on Twitter. Journal of Open Source Software, 4(42), 1706, DOI: 10.21105/joss.01706

Installation Instructions for AWS Users (Easiest)

BotSlayer uses Amazon Web Services (AWS) via an Amazon Machine Image (AMI) to streamline the setup process for non-technical users.

  • Step 1. Go to AWS. If you do not have an account, you will have to create one. This is free but requires a credit card. You can select the Free Tier. Sign in to the Console. In the AWS Management Console, Click on "All services", then "EC2".
    step 1
  • Step 2. Click the button to launch an EC2 instance.
    step 2
  • Step 3. [Warning: Please set your AWS service region to Ohio in the top right corner of the page before searching] After you requested the software and agreed to the EULA, you should have received instructions that include a long secret string like 0e...f2. Type this in the search box at the top. Click "Community AMIs" tab on the left. Select the one result under Community AMIs.
    step 3
  • Step 4. Select the correct image.
    step 4
  • Step 5. You can select the instance type marked "free tier eligible" for free. However, this configuration is unable to handle queries that generate a non-trivial volume of tweets. For example, it might work okay for tracking specific topics like #vaccines, but it is likely to crash for election-related content. We recommend you select "t2.xlarge" or more powerful configurations for high-volume topics. This is not free; it may cost $0.20/hour or more. Click "Configure Instance Details" at bottom-right to proceed.
    step 5
  • Step 6. Use the default settings on the "Configure Instance Details" page, and directly click "Add Storage" at bottom-right to proceed.
    step 6
  • Step 7. Choose your hard drive size. You can select up to 30GB for the free tier; we recommend 100GB or more for keeping data beyond several days, depending on how much information you track. Click "Add Tags" at bottom-right to proceed.
    step 7
  • Step 8. Use the default settings on the "Add Tags" page, and directly click "Configure Security Group" to proceed.
    step 8
  • Step 9. Add two rules to open the ports required by BotSlayer, as shown in the screenshot. Click "Review and Launch" at bottom-right to proceed.
    step 9
  • Step 10. Carefully review the configuration of the machine. If you spot any error, you can go back to fix the error. Otherwise, click "Launch" at bottom-right to proceed.
    step 10
  • Step 11. Create a new key pair, and give it a meaningful name, such as BotSlayer. This key pair is required to access the EC2 machine. Download the key pair and keep it at a secured environment. Then launch the instance.
    step 11
  • Step 12. You have finished setting up BotSlayer, so click the instance link to go to Step 13.
    step 12
  • Step 13. Copy either the domain name or the IP address and paste it into your browser to access the web interface of BotSlayer. Please wait 5 minutes so that the machine has enough time to start BotSlayer. Bookmark the IP address, as this is how you will access the BotSlayer dashboard. Note: if you restart the EC2 instance, the IP address will change. To assign a static IP address you can use an Elastic IP Address (not free).
    step 13

Installation Instructions for Mac Users

  1. Download the stable version of Docker Desktop for Mac.
  2. Launch Docker Desktop.
  3. After you requested the software and agreed to the EULA, you should have received a URL linking to a Mac disk image (.dmg) file and a password. Download the dmg file and double click to mount the disk image, which will appear on your desktop.
  4. Inside the image folder, double-click the file double-click-to-install.command.
  5. Depending on your computer, the installation will take anywhere from a few seconds to a few minutes. When the installation is finished, a website will open at the following URL: localhost:5000/config. If it says that the page cannot be loaded, your system could be starting up; wait a few moments before refreshing the page.

Installation Instructions for Docker Users

BotSlayer can be installed on any server via a set of prebuilt Docker images. However, we recommend installation on Linux servers, since the installation process is more complicated on Windows and we do not provide support to resolve the complications. To install Docker on your Linux server machine, please follow the instructions on the Docker website. To install docker-compose, please follow the instructions on this link.

After you requested the software and agreed to the EULA, you should have received a URL linking to a tar file, and a password. Download the tar file containing multiple Docker images. You will need to enter a username (botslayer) and the given password. Use the following bash commands to download the tar file and install BotSlayer (replace [TAR-URL] with the URL and [VERSION] with the version number): 

wget --user=botslayer --ask-password "[TAR-URL]"
tar xvf "v[VERSION]-beta.tar"

docker load < bsstream.tar.gz
rm bsstream.tar.gz
docker load < bstask.tar.gz
rm bstask.tar.gz
docker load < bsweb.tar.gz
rm bsweb.tar.gz
rm "v[VERSION]-beta.tar"

To verify that BotSlayer is installed properly, you can use the following commmand: docker ps to make sure there are three docker images loaded, namely bsstream, bstask, and bsweb.

To launch BotSlayer after installation, please issue the following command:

nohup docker-compose up task web stream logtab > run.log &

The docker-compose command runs posgresql, redis, rabbitmq, and frontail alongside the downloaded Docker images.

The frontail logging interface is exposed at port 9001, streaming log files inside the container. The postgres database is accessible at port 5432. Do not expose this port to the public; require ssh to access the database internally. The name of the database is botslayer. The username of the database is botslayer. The password is botslayerpsqlpassword.

For easier access to the dashboard on the default HTTP port (80), you could either (1) setup a reverse proxy from port 5000 to port 80, or (2) use sudo to force map port 5000 to port 80 when running your docker container.

If your server restarts for any reason, you will have to restart the BotSlayer Docker containers. An alternative solution is to setup some process manager, such as supervisord.

If you would like to see an example of how we setup our EC2 environment, which includes installing Docker and docker-compose, setting up reverse proxy on nginx, and configuring supervisord, please refer to this gist for our setup steps.