Extensions for TTYtter

Hi, this is Iván Sánchez, and these are my extensions for TTYtter, the amazing text-mode twitter client.

I developed these extensions to supercharge my TTYtter experience, but please do feel free to contact me if you have any bugs or issues with these extensions, or want to send me some beer. Or chocolates or something.

I suggest everyone using ttytter should use multigeo.pl and deshortify.pl. Newcomers to ttytter should download them both and add this line to their .ttytterrc file:

exts=/path/to/multigeo.pl,/path/to/deshortify.pl

However, most ttytter newbies can jump-start their ttytter experience by putting all this in their .ttytterrc file:

ansi=1 timestamp=default vcheck=1 ssl=1 avatar=wget %U -O /tmp/avatar && img2txt -f utf8 /tmp/avatar slowpost=3 hold=1 exts=/path/to/multigeo.pl,/path/to/deshortify.pl dostream=1

Besides TTYtter and its requirements, you might need additional perl libraries:

For this extension	install this package	or manually run
multigeo.pl GPSD functionality	libnet-gpsd3-perl	cpan Net::GPSD3
multigeo.pl GeoIP functionality	libgeo-ip-perl	cpan Geo::IP
deshortify.pl	libwww-perl liburi-find-perl	cpan URI::Find cpan URI::Split cpan LWP::UserAgent
autounfollow.pl hipstter.pl		cpan Text::ParseWords
superpwntter.pl	libdbd-mysql-perl libdate-manip-perl	cpan DBD::mysql cpan Date::Manip
superpwntter_pg.pl	libdbd-pg-perl libdate-manip-perl	cpan DBD::Pg cpan Date::Manip

If you're an avid twitter user, you already know that tweets can have a geolocation attached to them. In the web interface, this means seeing a "Sent from wherever" small message next to each tweet. In TTYtter, it means seeing a plus sign next to the name of the sender (e.g. <+RealIvanSanchez> instead of <RealIvanSanchez>).

So I coded my way into making TTYtter show information about location, and locating tweets in an easier way than issuing /set lat and /set long commands. The result of that work is the unified set of functionality that I called multigeo.pl.

MultiGeo implements the following commands and functionality:

Readable location of tweets

MultiGeo will automatically display the geolocation of tweets in an user-friendly, easy-to-read text string just above the tweet:

u5+ West Main Street, Madisonville, Madison, Texas, 77864, United States of America u5> [2011-10-08 17:47:43] <+SeanGorman> @ajturner has gone full cowboy in honor of #ratdiver http://t.co/KPPCaYCX

This works only for geolocated tweets, and uses the multigeo-wide geocoder (see below). The verbosity and detail of the location will depend on the geocoder being used.

There are two kinds of geolocated tweets: those with a pair of (latitude,longitude) coordinates, and those with a Place ID. Place IDs are managed directly by the twitter API and need no name resolution (see this old announcement for context). This means that only tweets with lat-lon will be passed to the geocoder for resolution. The geolocation method will depend on the user agent software of the tweeter, and shouldn't matter much to the reader - multigeo.pl will display both types of geolocation (altough geocoded lat-long tends to be more verbose than Place IDs). In a nutshell, tweets sent from the web use Place IDs, whereas tweets from TTYtter (and GPS-enabled mobile clients) use coordinates, and multigeo.pl will display everything. If you need anything else, /dump the tweet or bug me.

/place

By typing the command /place [name of the place you're at], your tweets from that moment on will be geolocated there, for example:

TTYtter> /place Rio de Janeiro, Brazil <029 -- Your next tweet will be sent at -22.9112163,-43.2093781, which is near Rio de Janeiro, Brasil, South America (a city) TTYtter> Wish my tweets could have a sunbath (testing yet another @ttytter geolocation extension) <088 -- using lat/long: (-22.9112163, -43.2093781) -- waiting 3 seconds to tweet, ^C cancels: "Wish my tweets could have a sunbath (testing yet another @ttytter geolocation extension)" -- sending to server c2+ Viaduto dos Fuzileiros, Praça da Bandeira, Rio de Janeiro, 20211-445, Brasil c2> [2011-10-14 20:31:16] <+RealIvanSanchez> Wish my tweets could have a sunbath (testing yet another @ttytter geolocation extension)

Then you can use /place off or just /place to turn off geolocation of any tweets after that. You can specify a street, city, placemark or business as a place:

TTYtter> /place Rio de Janeiro, Brazil <029 -- Your next tweet will be sent at -22.9112163,-43.2093781, which is near Rio de Janeiro, Brasil, South America (a city) TTYtter> /place NG8 1, United Kingdom <029 -- Your next tweet will be sent at 52.950996099129,-1.19203191262383, which is near NG8 1, United Kingdom (a postcode) TTYtter> /place Bronx, New York <022 -- Your next tweet will be sent at 40.8501002,-73.8662464, which is near Bronx, New York, United States of America (a hamlet) TTYtter> /place Coors Field, Denver, Colorado <036 -- Your next tweet will be sent at 39.7561146965203,-104.994083810925, which is near Coors Field, Blake Street, Denver, Denver County, Colorado, 80205, United States of America (a stadium) TTYtter> /place Statue of Liberty, Liberty Island <040 -- Your next tweet will be sent at 40.689245737137,-74.0445412130299, which is near Statue of Liberty, Hudson River Waterfront Walkway, Jersey City, Hudson County, New Jersey, 07302, United States of America (a attraction) TTYtter> /place Hospital Clinico San Carlos, Madrid, Spain <049 -- Your next tweet will be sent at 40.4407617610234,-3.71992637213825, which is near Hospital Clínico San Carlos, Calle Profesor Martín Lagos, Chamberí, Moncloa-Aravaca, Madrid, 28008, Spain (a hospital) TTYtter> /place Konig Frankfurter, Girona <032 -- Your next tweet will be sent at 41.9854809,2.8221996, which is near Konig Frankfurter, Gran Via de Jaume I, Girona, Generalitat de Catalunya, Catalunya, 17002, Spain (a restaurant) TTYtter> /place off <010 -- Turning geolocation off

This functionality uses the multigeo-wide geocoder (see below). The accuracy of your place queries will depend on the geocoder used.

Thanks to TTYtter user @xurxosanz for the idea for this functionality.

/teleport

Maybe you want to tweet from the same place another person is tweeting from. Maybe you are at the same place as another user, or even at a event with lots of people and geo-located tweets around. Or maybe you'd wish to be there.

Just type /teleport [tweet code] to be there:

d8+ Rua Miguel Couto, Centro, Rio de Janeiro, 20070-030, Brasil d8> [2011-10-14 16:58:02] <+Nighto> http://t.co/4rqPEKEt OH YEAH TTYtter> /teleport d8 <013 -- Your next tweet will be sent at -22.903032,-43.17926. TTYtter> Beam me down to Brazil, Scotty! (testing another @ttytter geolocation extension) <080 -- using lat/long: (-22.903032, -43.17926) -- waiting 3 seconds to tweet, ^C cancels: "Beam me down to Brazil, Scotty! (testing another @ttytter geolocation extension)" -- sending to server e1+ Rua Miguel Couto, Centro, Rio de Janeiro, 20070-030, Brasil e1> [2011-10-14 19:48:34] <+RealIvanSanchez> Beam me down to Brazil, Scotty! (testing another @ttytter geolocation extension)

Please note you won't be able to teleport to tweets that have Place IDs instead of coordinates. TTYtter cannot (as of now) post tweets with a Place ID, and so multigeo.pl cannot clone that ID. That's what's happenning if you see a sorry, no geoinformation in that tweet error message.

Thanks to TTYtter user @vehrka for the idea for this functionality.

GPSD

GPSD is the way to go if you're tweeting from a machine which has a GPS plugged in to it. Be sure to get GPSD working first.

If everything's working, this functionality will automatically geolocate your tweets on the coordinates of your GPS receiver:

TTYtter> Trying out gpsd support for ttytter <035 -- No GPS fix, not using geolocation. -- waiting 3 seconds to tweet, ^C cancels: "Trying out gpsd support for ttytter" ^C -- not sent, cancelled by user

Awww, dammit. No GPS fix inside my flat. Seems I'll have to stick my hand out of the window one more time for this to work. Now, let's try again:

TTYtter> Trying out ttytter+gpsd geolocation (again) <043 -- using lat/long: (40.7379316893733, -3.44560555824903) -- waiting 3 seconds to tweet, ^C cancels: "Trying out ttytter+gpsd geolocation (again)" -- sending to server

By default, GPSD functionality is not active. You need to set the option extpref_multigeo_gpsd=true in your .ttytterrc file. Once TTYtter is running, you can issue the commands /gpsd on and /gpsd off to enable/disable GPSD functionality. Typing the /gpsd command alone will query the current GPS status (offline / no fix / fix and coordinates).

Please note that /place and /teleport implies /gpsd off. Conversely, /gpsd on will override any previous /place or /teleport command.

Be warned that GPS functionality has not been throughoutly tested. Please report bugs and patches.

GeoIP

GeoIP is the poorman's GPS. Based on your current IP and a IP-to-city database, it is able to set your general location. Don't expect accurate geolocation with this method. You might even end up in the wrong city if ISPs do a lazy job.

TTYtter> Trying out #GeoIP extension for @ttytter <040 -- GeoIP extension will try to locate your public IP address now -- GeoIP: 83.50.223.514 is near Valencia, Comunidad Valenciana, Spain -- using lat/long: (39.4667, -0.3667) -- waiting 3 seconds to tweet, ^C cancels: "Trying out #GeoIP extension for @ttytter" -- sending to server c2+ Císcar - Comte d'Altea, Calle de Císcar, Eixample, Valencia, València/Valencia, Comunitat Valenciana, 46006, Spain c2> [2011-10-26 0:40:54] <+RealIvanSanchez> Trying out #GeoIP extension for @ttytter

GeoIP location will only happen once, just before your first tweet, and only if you've set $extpref_multigeo_geoip=true in your .ttytterrc file. If coordinates are already set (e.g. GPSD got a fix, or you've run /place before your first tweet), GeoIP location will not happen. It is quite fair to assume that if coordinates are already set, they're better than the ones GeoIP can provide.

Be warned that GeoIP functionality is not usable at this moment. If you know how to fetch your public IP from Perl, or how to make STUN work, please do let me know. In the meantime, just use /place [your city]

Shared geocoder

The /place and readable locations now share the geocoder. A geocoder is a external service that translates a placename into latitude-longitude coordinates and viceversa. Right now there are three geocoders available:

The OpenStreetMap Nominatim geocoder queries que OpenStreetMap database, containing detailed information about most streets, and businesses if you're lucky. Please do check out www.openstreetmap.org to check if places near you are in the database (add them if they're not in there! It's a wiki!), and check out wiki.openstreetmap.org/wiki/Nominatim or else I will die in the hands of angry lawyers because you didn't read Nominatim's terms of use.

The GeoNames geocoder queries the much simpler www.geonames.org database, which only contains data at city level (i.e. the most detailed information you can get from GeoNames is the name of a city).

The Twitter API geocoder queries two API 1.1 endpoints (for direct and reverse geocoding). This geocoder allows you to fetch the very same placenames that other pure twitter clients show you. I'm guessing this is the "official" way to do geostuff according to the Twitter guys. But, be warned this geocoder can use up your API rate limit more quickly.

Finally, the null geocoder does absolutely nothing. Use this if you don't care about seeing the location of other tweets, and if you won't use /place.

By default, MultiGeo will use OpenStreetMap's Nominatim. In order to manually specify a geocoder, add one of those to your .ttytterrc file:

• extpref_multigeo_geocoder=nominatim
• extpref_multigeo_geocoder=geonames
• extpref_multigeo_geocoder=twitter
• extpref_multigeo_geocoder=none

And before anyone asks: there will not be a Google geocoder, due to Google's terms of use. Ideas about other geocoders will be appreciated.

Note that, right now, TTYtter only has the ability to post tweets with coordinates, and not place IDs. A Place ID geocoder might be feasible, if there is interest from the TTYtter user community.

Yes, yes, I know. You're fed up with all those t.co and ht.ly and fb.me and bit.ly and g.co URLs all over, and blindly clicking through them. If only TTYtter was able to show you the real URL behind that...

Well, it can, silly. You just need to download and use deshortify.pl, and all of a sudden you'll see full link URLs. They will take more screen state to display, but are quite easy to read as they are underlined (if your terminal supports ANSI). Like this:

v1> [2012-03-13 23:53:35] <RealIvanSanchez> Dear @ttytter users: Behold! The new and improved de-shortifier: http://t.co/Xgqqu9Ra (now with underlining!)

v1> [2012-03-13 23:53:35] <RealIvanSanchez> Dear @ttytter users: Behold! The new and improved de-shortifier: http://ivan.sanchezortega.es/ttytter/deshortify.pl (now with underlining!)

Deshortify also does some tricks to URLs to make them shorter: resolves feedburner and google news URLs, and cuts off some extraneous tracking stuff from the end of URLs (those "utm_source=twitter" bits that don't do anything useful to you and just take up space), and properly follows some <iframe>-based shorteners.

Under the hood, deshortify uses HTTP HEAD requests to resolve most URL shorteners (except the iframe-based ones), and this takes time. It is normal for your tweets to lag a second or two before being displayed. Also, any URLs resolved may count as visited for some SEO tracking platforms, even though the requests are made with a bot-like user-agent - privacy junkies be warned.

Deshortify supports some configuration values in your .ttytterrc file:

extpref_deshortifyproxy=http://proxy.localnetwork:8001/

Overrides system proxy settings. Deshortify tries to use the proxy defined in your environment variables but, if that fails for whatever reason, you can specify it in here.

extpref_deshortifyretries=10

It is possible that resolving a shortener will timeout or fail for whatever reason. When this happens, deshortify will retry up to 3 times unless you modify this setting.

extpref_deshortifyalways=1

By default, deshortify only works on a list of known URL shorteners. This means "shorteners that Iván has seen in his timeline". Setting this will cause deshortify to always follow a link, regardless of if the URL seems to come from a shortener. Be warned: this means one more HTTP request per URL, which means more lagginess.

Update: Starting with version 2.1.0, TTYtter supports resolution of t.co URLs. However, deshortify works with a bunch of other URL shorteners, such as fb.me, is.gd, ow.ly, adf.ly, kcy.me and 175 others. You will always see the final URL no matter how many URL shorteners are in the middle.

For example, if you run ttytter with the -verbose flag, you'll see all the deshortening taking place, like so:

-- calling $handle in ttytter/deshortify.pl -- Deshortened: http://goo.gl/UP7oZ -> http://seocustomer.visibli.com/share/5sxB8O -- Deshortify found an iframe for http://seocustomer.visibli.com/share/5sxB8O, and it points to http://bit.ly/ZcgTc8 -- Deshortened: http://bit.ly/ZcgTc8 -> http://seocustomer.com/how-to-rock-your-website-and-get-traffic-now-step-by-step/ zd6> [2013-01-03 10:03:36] <HenrikSandbergJ> How to Rock your Website and Get Traffic Now - step by step - SeoCustomer - http://seocustomer.com/how-to-rock-your-website-and-get-traffic-now-step-by-step/ via @HenrikSandbergJ

What just happened there? Well, one of them fancy SEO experts is using four chained URL shorteners. In this case, t.co (transparently) points to goo.gl; which points to visibli.com; which points to bit.ly; which points to the real site. Or, for another example, this:

-- calling $handle in ttytter/deshortify.pl -- Deshortened: http://flip.it/GGPAg -> http://t.co/BSby9p4N -- Deshortened: http://t.co/BSby9p4N -> http://vrge.co/Vu31EM -- Deshortened: http://vrge.co/Vu31EM -> http://www.theverge.com/2013/1/6/3842494/nfl-jumbotron-size-comparison-infographic zb5> [2013-01-06 21:03:25] <cageyjames> RT @verge: A visual comparison of every jumbotron in the NFL http://www.theverge.com/2013/1/6/3842494/nfl-jumbotron-size-comparison-infographic

This looks like a Flipboard user did a manual retweet. The oritinal t.co URL points to flip.it, which points to another t.co, which points to vrge.co, which points to a verge.com article. It's not uncommon (at least, in my timeline) to find people (re)sharing via facebook (fb.me) or using clients that automatically shorten URLs, copy-paste links of RSS feeds, and so on.

Deshortify will also show user-friendly errors when the destination page is unavailable. For example, if you got a bot checking the wrong RSS feed, you'll get something like:

[2013-01-03 13:41:45] *** Could not deshortify http://feeds.feedburner.com/D178AD80-7A2D-41AC-ADEB-A6E4A2D3C044?utm_medium=twitter further due to 404 Feed not found error: FeedBurner cannot locate this feed URI. za0> [2013-01-02 11:40:02] <ComunidadMatuit> Paros parciales convocados para el 4 de enero. http://feeds.feedburner.com/D178AD80-7A2D-41AC-ADEB-A6E4A2D3C044?utm_medium=twitter vía @EMTmadrid

So, the other day I read this:

zf5> [2012-03-28 20:24:20] <metazool> considering unfollowing anyone who utters the word 'license' over the next week and i mean anyone. Or is it 'licence'? Well, either, both

And I said to myself: "This calls for a ttytter extension!". Behold, autounfollow.pl!:

TTYtter> /follow nyancat <015 -- ok, you have started following user nyancat. TTYtter> /friends +1 <011 nyan cat (nyancat) (f:4/203) (u:1) TTYtter> /killword !!! <013 -- Auto-unfollow armed and ready. -- Auto-unfollow killword active: !!! TTYtter> /a nyancat <010 -- Found tweet from nyancat with killword: !!! -- ok, you have stopped following user nyancat. za0> [2011-12-17 22:35:34] (x12) <nyancat> lets nyan!!! TTYtter> /friends +1 <011 Big Ben (big_ben_clock) (f:0/173677) (u:20302) "Established November 2009. Entirely unofficial & apparently imitated e..."

What is going on in there? Well, I started following @nyancat, but then wanted to unfollow anyone who abuses exclamation signs. As it appears, @nyancat does, so the extension automatically unfollowed it when the tweet was loaded. And it's really unfollowed, because it doesn't appear among my list of followed users anymore.

In case you want to stop the unfollowing massacre, just specify no killwords, like:

TTYtter> /killword <009 -- Auto-unfollow is now disarmed.

Want more than one term? Eeeasy peasy:

TTYtter> /killword "Please RT" viagra "buy rolex" <040 -- Auto-unfollow armed and ready. -- Auto-unfollow killword active: Please RT -- Auto-unfollow killword active: viagra -- Auto-unfollow killword active: buy rolex

This extension is a bit buggy still. Supposedly, it should save you from viewing the tweet that caused you to enter a mad rage and unfollow the user, but right now it doesn't. I'm returning 0 and not calling &defaulthandle, but still... Well, this shouldn't be a problem, 'cause you won't see that user in your timeline ever again!

q5> [2012-03-29 21:31:25] <@xurxosanz> @RealIvanSanchez auto-autounfollow based on trending topics #concepto #heytío //@metazool s2> [2012-03-29 21:35:38] <RealIvanSanchez> Thinking about #hipstter, a @ttytter extension to unfollow anyone tweeting about any trending topic, thanks to @xurxosanz s5> [2012-03-29 21:37:42] <@tjh> @RealIvanSanchez Please, please do it. PLEASE. t3> [2012-03-29 21:42:33] <@pr4wn> @xurxosanz @RealIvanSanchez @ttytter Best keep quiet about my #bieber fixation, then. #hipstter

You asked for it, folks. Go get hipstter.pl before I fix the bugs and is outfashioned.

Right now, hipstter mostly works, but is not able to process trends with non-ascii-7 characters.

On a non-georelated note, consider using the libcaca utilities (apt-get install caca-utils on Debian-like systems) for displaying the avatar image (when doing /whois username), by adding this line to .ttytterrc:

avatar=wget -q %U -O /tmp/avatar && img2txt -f utf8 /tmp/avatar

I know that @pr4wn has been using some of my ttytter extensions for a while, so I decided to give pwntter a try. And it's nice, but I felt it lacked some functionality, specifically logging some metadata introduced with the Twitter API 1.1.

For a few months now, I've been developing SuperPwntter. It's much like pwntter, but super. Logs to several tables, logs user data, tweet data, location data, mentions, and what not. Unfortunately, it's still under development, so caution is advised!

If you want to try it out, you'll need several files:

File	what does it do
superpwntter.pl	The main extension. This is what you add to your .ttytterrc file. By now, this file only works for MySQL databases.
superpwntter_pg.pl	Pretty much like superpwntter.pl, but modified for use with PostgreSQL databases.
superpwntter_tables.sql	Database creation script. Works best with MySQL tables.
superpwntter_tables_pg.sql	Pretty much like superpwntter_tables.sql, but tailored for PostgreSQL databases.
chart.php	Report script. Needs a web server with PHP and MySQL support for PHP.
chart_pg.php	Pretty much like chart.php, but tailored for PostgreSQL. Needs PostgreSQL support for PHP.

Installation notes

You should be able to create your database according to your favourite DB engine, and run the commands inside the SQL file for creating your tables and populating the initial data.

After that, make sure that both the extension .pl file and the reports .php file are set with the correct database name, username and password. Remember to add the .pl file to the .ttytterrc extensions and put the .php file somewhere where your web server will process it.

.php files are renamed to .phps so my webserver shows you the source code instead of running it. Rename them as needed.

Known issues

Lots. This is still under development, so use with care. I still have to refactor code to be more database-agnostic and implement a few missing things. Feel free to add issues to the GitHub issue tracker (and I'll feel free to ignore them since I work on this in my spare time). Pull requests are welcome.

Acknowledments to @pr4wn for the original pwntter, to @xurxosanz for bug-testing this and @vehrka for fixing up PostgreSQL support.