a python application
:for twitter automation with Cloud Foundry on IBM Bluemix
Mini tweet bot is a web application (accessible in the below link to the working app) for automating twitter functions with a designated synced twitter account. The front page has the feature of allowing anybody with access to the app to tweet to the synced account. This is a useful purpose for any situations in which you would want multiple people tweeting to the same account. Twitter offers the service called Tweetdeck (linked below), which has a similar feature allowing multiple users to tweet to the same account; however Tweetdeck requires every user to have a a personal twitter account, and have the private access information (the password) for the shared account. With mini tweet bot, there are no restrictions, no shared passwords, no login process, and no twitter account required to tweet; users simply need access to a web browser either on their own device or be able to use a public computer device at, for example, an event or Tradeshow. The other extended features of mini tweet bot are not available with twitter's tweet deck, but are similar to many of the twitter bot features offered from 3rd party services. Those features, in the order they appear in the app, are:
- tweet image or selfie: this functionality appears on the home page along with the status update tweet functionality.
- translate: mini tweet bot will translate your tweets into L337 or ascii-art text.
- retweet and follow: searches twitter for tweets from the results of a query from the search terms from the given user input, then attempts to retweet on of the returned tweets, then attempts to follows the user of the retweet.
- Follow followers: a minor feature that follows all of the synced user account's followers.
- Follow X more users: searches twitter for tweets from the results of a query from the search terms from the given user input, and follows X number of the users from the tweets in the results of the search. X is taken from user input or default
- Auto retweet: searches twitter for tweets from the results of a query from the search terms from the given user input, then retweets a tweet and follows the user in an automated process and so this cycle is repeated every N seconds for Y intervals. N and Y are taken from user input or default.
- Auto tweet file: accepts a file and retweets a newline from the input file in an automated process and so this cycle is repeated every N seconds.
An example tweet, tweeted from mtb.mybluemix.net:
Listed below are some of the environmental variables, the platforms used, and some of the important links referenced for the mini tweet bot.
- language: Python 3.6.1
- libraries: tweepy, time, multiprocessing, opencv-python, numpy, workzeug
- web framework: flask
- style: PEP 8: https://www.python.org/dev/peps/pep-0008/
- Note: for python-2.7.10, download the .zip file from the Release v0.0.4 - this older release uses web.py:
- infrastructure: IBM Bluemix, https://www.ibm.com/cloud-computing/bluemix/
- platform: Cloud Foundry, https://www.cloudfoundry.org/
- Cloud Foundry command line interface (CLI): https://github.com/cloudfoundry/cli
- CF python template app: https://github.com/IBM-Bluemix/get-started-python
- code on github: https://github.com/johncoleman83/mini-tweet-bot
- working app: https://mtb.mybluemix.net/
- demo app: https://holberton-mtb.mybluemix.net/
- demo sourcecode: https://github.com/johncoleman83/holberton-mtb
- twitter dev tools: https://dev.twitter.com/
- tweet deck: https://tweetdeck.twitter.com/
- twitter (bot) account: https://twitter.com/are_no_one
- api limits: https://support.twitter.com/articles/160385
- best practices: https://dev.twitter.com/basics
While at the Docker Conference (#DockerCon) in Austin, TX 2017, I was introduced to Cloud Foundry applications on IBM Bluemix. This occurred around the time of my spring break from my studies at Holberton School, which was near the end of our C language curriculum, and the week before we began our python curriculum. Since, I had time to rest during spring break, I was excited for the challenge of learning python, interested in navigating IBM's Bluemix, and impressed with twitter's social potential, I made mini tweet bot. This article summarizes some of mini tweet bot's features and how I built it; it is organized by: (1) python, (2) web integration, (3) cloud infrastructure, and (4) bugs & need for improvement, and (5) build your own bot. Noob disclaimer: this mini tweet bot is my first attempt to build anything with python and to integrate with Cloud Foundry apps.
Holberton School is a project-based 2-year full-stack software engineering program. With an income-share agreement, the school has a joint stake in it's student's success, and therefore draws many candidates looking for an alternative to the college and university system. Named after Betty Holberton, one of the pioneer engineers to work on ENIAC, Holberton is also especially considerate of the interests of gender- and racially- diverse populations. To learn more about Holberton School, check out their website or follow their twitter account (below is a recent retweet by @holbertonschool announcing an investment from Ne-Yo).
One of the main engines of the mini tweet bot is the tweepy library for python. This library has many custom integrations with twitter that may be utilized by simply configuring a twitter account with Twitter's development platform, and then calling a function in python. To configure tweepy with your twitter account, first, create a twitter application from the above link, and then second copy your keys, access tokens, and secrets to your python app. Once you have copied your secrets from a twitter app, then there is another set of configuration functions to help tweepy initiate. The below configuration example contains some of code on how I configured tweepy with my twitter app. Note: in order to keep my private information confidential, I stored the data in a separate file that is in my
.gitignore file, and separate from inside the code of my python app. In the below example, I have not provided my personal key, token, nor secret information, but instead only examples. Below, I also included the libraries and variables imported for support functions. To customize your own mini tweet bot, you should use your own key, token and secret information and replace your information with the information in the
mycredentials.py file; all the rest of the python application will function for whatever twitter account is used with the configuration file.
$ cat mycredentials.py consumer_key = '[YOURCOMSUMERKEY]' #(example: '5FmNevRxQriq1') consumer_secret = '[YOURCOMSUMERSECRET]' #(example: 'pMKxr4ocPV451') access_token = '[YOURACCESSTOKEN]' #(example: 'ZuPzOYwz2n6o') access_token_secret = '[YOURACCESSTOKENSECRET]' #(example: 'TCqaOPc8j1qP') $ cat app.py | head -26 import tweepy import multiprocessing import os import cv2 from flask import Flask, render_template, request, jsonify from werkzeug import secure_filename from time import sleep from credentials import (consumer_key, consumer_secret, access_token, access_token_secret) from camera import take_picture # custom imports import censorship from aldict import ascii_dict, leet_dict # custom variable names from imports remove_whitespace = censorship.remove_whitespace censor = censorship.censor # tweepy auth = tweepy.OAuthHandler(consumer_key, consumer_secret) auth.set_access_token(access_token, access_token_secret) api = tweepy.API(auth)
Tweepy comes with many simple functions that can be used to tweet, follow others, and retweet. To read more about the full potential of tweepy, check out their readthedocs.io. For the purposes of this demo, I will explain how to setup tweepy to tweet, retweet, and follow. For more further explanations, you can alway check out the source code in the above link to the mini-tweet-bot github repository!
Since this python application interacts with a web browser, I have set up the mini tweet bot to call this function when the user inputs text to be tweeted to the connected account. The web integration with
flask is explained further below, this snippet is enough to show how simple it is integrate with tweepy. Once mini tweet bot receives the instructions from the user to make a tweet, the
tweet_text() function is called with a variable, which is a string of characters from the user input. In the
tweet_text() function, there is a
try: except: sequence, because this allows to check if the tweet failed to tweet. A tweet would fail most likely due to the tweet being a repeated tweet, or some other problem with the twitter API. For more information on the twitter API, read the above twitter API limits link. To attempt a tweet, a tweepy function is called with the same input string variable and the api authorization from the information from the configurations file. There is no need to check to ensure that the input text fits the twitter requirements for text size because the HTML5 form used to take user input has size restrictions built into it. The profanity check has also already taken place, so at this point in the code, if a tweet is successful, we hope that it is a positive tweet. Below is the
def tweet_text(tweetvar): """ tweets text from input variable """ try: api.update_status(tweetvar) except: return False return True
retweet & follow function
retweet_follow() function is called when the user makes this request. This function uses a
for loop to attempt to retweet (
tweet.retweet()) the loaded tweets from a twitter search query. A tweepy integration runs the twitter search query of tweets, using the search terms from the mini tweet bot user input (
tweepy.Cursor(api.search, q=searchterms)), and returns 10 tweets in the results (
.items(10)). Similar to the above function,
retweet_follow() function, also uses the
try: except:() sequence. If a retweet is not successful, the instructions in the function are to attempt to retweet the next search result. If all 10 attempts are unsuccessful, then the function returns unsuccessful. However, if a retweet is successfully retweeted, the twitter user that made the tweet that mini tweet bot retweeted, is followed (
tweet.user.follow()) by the linked twitter account so long as that user is not already being followed. Then, after the follow attempt, the function returns True, meaning the retweet and/ or follow were successful.
def retweet_follow(searchterms): """searches tweets with searchterms, retweets, then follows""" for tweet in tweepy.Cursor(api.search, q=searchterms).items(10): try: tweet.retweet() if not tweet.user.following: tweet.user.follow() return True except tweepy.TweepError as e: print(e.reason) pass return False
Since this is a public tool, I added my own custom censorship module that stops a user from being able to tweet a phrase that contain at least one of the 700+ strings of profanity words. If a word match is found, the
tweet_text() function is never called, and a failure message is output. Below are some of the functions for the censorship; I'll explain how they work below.
from profanity import profanity def is_clean(inputlist): for i in range(0, len(inputlist)): word = inputlist[i] if word in profanity: return False test1 = remove_whitespace(word, "") if test1 in profanity: return False test2 = leet_detect(word) if test2 in profanity: return False test3 = leet_detect(test1) if test3 in profanity: return False return True
is_clean() function checks if the input list is clean or if it has a word from the profanity set. The profanity set can be viewed from in the mini tweet bot source code (linked above). This is a simple loop that does a string compare function with each word from user input and each word from the profanity set. The file
profanity.py contains a set of vulgar words, and is imported with the first
import line above. I used a set because the way python stores in memory and searches a set is a faster speed of O(log 1) vs. searching through an array or list of strings, which is the speed of O(log n). The list of profanity words is also stored in the
profanity.txt file in the
support directory, and it has no format, just one word per line. The second conditional check in the loop removes all the whitespace characters from each word just in case there is an input of
"bad_word" or "i-am-mean", which would be a user trying to pass in
"badword" or "iammean" to tweet. The last two profanity checks translate ascii-art and L337 to alpha characters, and then perform the same checks. So
"b@dw0rd" or "b/\|)\^/ord" are translated to
"badword" and "badword" and then the profanity check occurs. The ascii and L337 dictionaries can also be viewed in the source code in the file
def censor(tweetvar): test1 = test2 = tweetvar.lower() test2 = remove_whitespace(test2, " ") test1 = test1.split() test2 = test2.split() if is_clean(test1): if is_clean(test2): return True return False
censor() function converts the input string into two test strings, which are then converted into arrays. One string has all the whitespace characters converted to spaces, the other string is left unchanged. The whitespace conversion occurs once here, and then again for each word of the other array that is left unchanged inside the
is_clean() function. The first whitespace conversion changes an input string of:
"1badword-2badword_3badword" to this:
"1badword 2badword 3badword", so that each word is checked to see if has profanity. The second whitespace check is explained above. Finally each array is checked to see if it has profanity with the above
is_clean() function. If there is no profanity, the censor functions return True meaning the string is clean, and the instructions in the
flask @app.route functions respond accordingly.
(2) web integrations
flask provides excellent integrations for python and website browsers. There are other web frameworks that python experts prefer, but I chose this one because it is simple for what I wanted to do, I found the documentation helpful, IBM Bluemix Cloud Foundry python template uses flask, and it will be a part of the Holberton School curriculum when I get to web integrations. The major part of flask that I utilized was the functions to load webpages and take user input from HTML5 forms and input. Once flask is imported, there is no need for any other configuration except to establish the port on the device running the app. The mini tweet bot uses the below example configuration for the port, and I also included some of the variables that I used for the file upload functions.
from flask import Flask, render_template, request, jsonify from werkzeug import secure_filename app = Flask(__name__) port = int(os.getenv('PORT', 8080)) UPLOAD_FOLDER = './uploads' app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER ALLOWED_EXTENSIONS = set(['txt', 'png', 'jpg', 'jpeg', 'gif'])
To help explain how flask interacts with user input, I will use the home page python code, which tweets user input or images directly to the linked twitter account. Below is the snippet of code, which has the route instructions for
POST method requests from the user in the decorator:
@app.route('/', methods=['GET', 'POST']) def index(): if request.method == 'GET': return render_template('index.html', newstring="none") if request.method == 'POST': tweetvar = request.form['tweet'] if not censor(tweetvar): return render_template('confirmtweet.html', tweetvar="profanity") if request.form['translate'] == "ascii": tweetvar = translate(tweetvar, 'a') elif request.form['translate'] == "leet": tweetvar = translate(tweetvar, 'l') if request.form['action'] == 'translate': return render_template('index.html', newstring=tweetvar) else: if request.files['file']: file = request.files['file'] if allowed_file(file.filename): filename = secure_filename(file.filename) filename = os.path.join(UPLOAD_FOLDER, filename) file.save(filename) if tweet_image(filename, tweetvar): return render_template('confirmtweet.html', tweetvar=tweetvar) elif request.form['action'] == 'selfie from webcam': filename = take_picture() if tweet_image(filename, tweetvar): return render_template('confirmtweet.html', tweetvar=tweetvar) elif tweet_text(tweetvar): return render_template('confirmtweet.html', tweetvar=tweetvar) return render_template('confirmtweet.html', tweetvar='failure')
In the above index class example, when a user requests the homepage (i.e. index.html or
'/') the conditional
if request.method == 'GET': deals with that request and returns the index.html template using
return render_template. The instructions from the conditional
if request.method == 'POST': deal with the user request to input data through the index.html form's submit input button. The
request.form call returns the value linked with the input key if the user input data through the web form. The
tweetvar = request.form['tweet'] section stores the user input string in the variable
tweetvar, which needs to be 5 - 140 characters (that check is filtered in the HTML form). Next, is the profanity check, which is necessary because no other automation will occur if there is profanity (this is discussed in the above sections). If profanity is found, then the user receives a failure message with the reason being their use of profanity. That instruction is in the
return statement after the profanity conditional check.
If no profanity is found, next comes the translations. Since, I built a filter to check for users attempting to tweet vulgar words to the public account, I decided to use that feature to allow users to translate their input. If user selects to translate the text, then the same page is rendered with the user input translated into either ascii-art or L337. The translation process is much the same as the censoring functions. However, in the translation function, the instructions are to check each letter from user input, and the character is in the ascii-art or L337 dictionary (depending on what user chose), it is simply replace with its equivalent. Once, the full user input string is parsed, there is a conditional check
if request.form['action'] == 'translate':, which checks if the user clicked the translate button. If this was the user choice, then the translated text is returned to the same page for the user to copy and paste into the tweet, so they have the option to use the translated string. If the translate button is not the submit button choice, this means that the tweet button was clicked, and so the following
else statement deals with the instructions to check if there is also an image input.
If there is an image input, then the file goes through a secure process to save the file and then calls the tweet image and status update function. The secure image process has these steps: (a) takes the file as a variable (b) checks if it is an allowed type, (c) changes the file name to a secure name (i.e. the file
home_vagrant_.bashrc), (d) appends the path to save the file in
./uploads directory, and (c) saves the file. The code for these above steps looks like this:
if request.files['file']: file = request.files['file'] if allowed_file(file.filename): filename = secure_filename(file.filename) filename = os.path.join(UPLOAD_FOLDER, filename) file.save(filename) if tweet_image(filename, tweetvar): return render_template('confirmtweet.html', tweetvar=tweetvar)
The last 3 lines check if the
tweet_image() function successfully tweeted, similar to the
tweet_text() function, which is discussed above. If the tweet is successful, then the confirmation page is returned with success. After this check, there is a check to see if the selfie button was clicked. The selfie button is not yet working in the cloud, but works on my local mac OSX machine, and it uses the camera to tweet a selfie. The last
elif deals with the situation of no image input; so only if the user requests to update the status of the linked account with a text string. If the call to the function
tweet_text() (described above) is successful, then the confirmation tweet occurs.
After the user tweet functions are processed, there is either a successful or unsuccessful page rendered in the return statement. Upon successful completion, that webpage will have a confirmation message displayed with the tweeted string. If the tweet was unsuccessful, for example, if there was profanity, or the tweet failed for another reason, then a failure message is displayed.
Without a server to host the mini tweet bot application, it becomes difficult to allow anybody else to access the application. Additionally, for long term running processes such as automating tweets from a file every N seconds, a Cloud becomes an absolute necessity, unless you have a computer in your home that you always keep running and always keep connected to the internet. Since I do not have my own server, and want to keep mini tweet bot running, the one of the most important elements of this application is the integrated with Cloud Foundry Apps on IBM Bluemix. IBM Bluemix offers a 30-day free trial of their cloud infrastructure; to begin that trial period, sign-up through the above link in the intro. After the 30-day free trial, there is a minimal pay per use plan that charges reasonable rates, which are based upon how much GB hours you use. This app is in the price range of $0.07 cents per GB hour. Since the app is less than 128 MB with one instance, it therefore costs less than $1.00 per month to operate, which is very reasonable. Once you have a Bluemix account, setting up your environment to host an application such as mini tweet bot, is as simple as clicking a button. In your IBM Bluemix Menu, navigate to Apps > Dashboard > Create App. In the Apps section, IBM Bluemix offers approximately 180 options in areas of: Boilerplates, Mobile, DevOps, Security, Storage, IOT, Watson, Data & Analytics, Containers, and Cloud Foundry Apps; Scroll to find the Cloud Foundry Apps section.
Since mini tweet bot is built in python, I used the python CF integration (image displayed above), which I installed after selecting the python app, choosing a name, host, and clicking the "create" button. Once your CF App is installed, IBM Bluemix has a great tutorial on how to get your python app running in the cloud. One simply needs to install the Cloud Foundry command line interface (from the github repo linked above in the intro), edit the Cloud Foundry application configuration files such as the:
manifest.yml, Procfile, and requirements.txt, and store those files in the same directory that contains your python app. These file templates can be found in the Cloud Foundry python template github repo (linked above in the intro), and IBM Bluemix has helpful documentation on how to use those configuration files and many other topics. Some of the other useful features of IBM Bluemix are that: Cloud Foundry CLI, provides the ability to connect to the directory of your project via SSH. There is a user interface through the website that allows you to see all the output logs of your app. The CF Command Line Interface has many other commands that I have not mentioned that improve the integration with Cloud Foundry apps on a cloud.
If you like this twitter bot, but don’t want to host it on IBM Bluemix, you can run the app in your local command line interface, and it should be visible from a local port on your own machine. Or find an alternative hosting company.
(4) bugs & need for improvement
I have not tested to see how many requests the mini tweet bot can handle. Currently, it does handle multiple requests per minute in the same browser, but I have not tested by increasing the number of requests nor using more than 1 browser. Twitter API has restrictions for the limits on tweets per hour and mini tweet bot has "try and except" processes to check for and deal with failed attempts to tweet or access information from twitter. IBM Bluemix services went down approximately 1-3 times since I began working with IBM Bluemix, and if this problem persists, it may interrupt some of the background processes which deal with automatic tweets in a time interval. I did not see any other issues with my running applications though, and despite not seeing any other instances where my app went down, there was another error message that I received periodically, which seemed to an issue with host server that did not effect my application. I did not know the cause of this message; the output text from my logs was:
LGR/nullproxy: error connecting to [IPADDRESS]: dial tcp [IPADDRESS]: i/o timeout
need for improvement:
These are some of the features that I am planning on adding:
- An if conditional to correctly allow more characters for tweets that include URLs.
- Integrate IBM Watson to help improve voice recognition with tweets (i.e. a user simply speaks in order to automate a tweet.
- Like tweets of users that bot retweets.
- Add twitter rant: tweets lines from text file every 60 seconds with max N lines.
Please contact me for more information on vulnerabilities, to help with upgrading, or with any other questions.
(5) build your own bot
- fork or clone the github repository.
- get your own twitter app from twitter dev tools linked above.
- change the
mycredentials.pyfile name to
- change the strings from the credentials file to contain your personal twitter information.
- change the twitter feed link in the
<aside>HTML tag to instead link to your twitter feed.
- change the link of the twitter handle in the
navHTML tag to link instead to your linked twitter account.
- change the icon/ logos to your preference
- find the cloud to host the app. Mini tweet bot is already setup with Cloud Foundry for IBM Bluemix, but other cloud services will work as well.
demo screen shot
for entire demo website see above link.
NOTE: The mini tweet bot functions most successfully when hosted on a cloud. However, if you would like to run the app on your own machine, you can run it, and it will be loaded on a local host port IP address such as: http://0.0.0.0:8080/. If you do not want the user interface, you should then use only the tweet functions, and run them on an as needed basis. Here is an example of how to run a single function from the
$ cat singletweet.py import tweepy from credentials import * auth = tweepy.OAuthHandler(consumer_key, consumer_secret) auth.set_access_token(access_token, access_token_secret) api = tweepy.API(auth) def tweet_text(tweetvar): """ tweets text from input variable """ try: api.update_status(tweetvar) except: print("error") pass tweet_text("this tweet is an example of running a tweet function in python") $ python singletweet.py