Building a Twitter Reader

Using TTYtter, Perl, and TWiki

DISCLAIMER - Before you read this...
If you want to implement what I have here, you will need a computer capable of running Perl and the Unix cron command.
Capable systems include Linux, BSD, Mac OS X, and Windows (if properly configured, probably with cygwin installed). For any other system, your mileage will vary widely. I have no idea if Windows has something approximating cron.
I use Mac OS X, which is based on BSD Unix. I cannot help you with any other system!

Background

One of the features of Twitter is that it runs 24/7/365. (Un)fortunately, I don't. So, I miss things. I didn't want to miss things, so I looked for a solution.

Being a programmer myself, I wanted a solution I could control and tweak if necessary. However, I didn't want to write something from scratch if I didn't have to!

Twitter has a popular, published API, so I figured someone would have written what I wanted. Someone did.

The Front End

I initially found pyTwerp (written in Python). I'm not a Python programmer, but pyTwerp looked like it does pretty much everything I needed, out-of-the-box. And it did... up till September, 2010 when Twitter went to OAuth only and pyTwerp stopped working.

TTYtter

I looked around a little more and this time I found TTYtter. TTYtter is written in Perl (a plus, given that I know Perl (and I don't know Python)). More important, TTYtter understands OAuth.

TWiki

I could easily read the TTYtter output with any text editor, or as plain .txt files in my web browser. But I wanted something that looked a little nicer. Enter TWiki.

TWiki is a structured enterprise wiki with a lot of programmable features. By using TWiki, I don't need to convert the output to html in order to read it on the web and I get the following useful features:

Blank lines are retained as "paragraph" breaks.
Anything that begins with http:// is rendered as a clickable link.
*word* is converted to bold; _word_ is converted to italic.
I can "script" the results, using TWiki to hide everything I'm not currently reading, or only show me this month's output.
I can easily adjust the look and feel with CSS.

How I Built My Twitter Reader

Steps

Install and Configure TTYtter
Get the Tweets
Filter the Tweets
Format For TWiki
Wrap it up (Packaging)

Install and Configure TTYtter

Start by downloading TTYtter from Floodgap Software.

You will also need Perl 5.005 or better (5.6 and up are recommended) and cURL. Nothing else is necessary

Installation and COnfiguration

To install TTYtter, download it to the location you want to run it from (such as your home directory or bin/), make it executable with chmod +x, and then start it from your shell prompt with something like ./ttytter. If your Perl is not in /usr/bin, change the first line.
When you start TTYtter without a keyfile, which will be the case the first time you run it, it will automatically start an assistant to help you create the keyfile. You only need to create the keyfile once for each account. It will never expire.

Full instructions with examples can be found at the TTYtter home page. .

Get the Tweets

First I used

   ttytter  -runcommand="/r"

but I didn't like the output format. So I asked for assistance and created an "extension" that provides the output format I wanted:

$handle = sub {
    my $ref = shift; # this is a hashref to the tweet structure
    my $sn = &descape($ref->{'user'}->{'screen_name'});
    my $string = ${sn} . '  ' .
            &descape($ref->{'text'}) . 
            ' -- ' .  $ref->{'user'}->{'protected'} . 
            ' ' .  "[[Twitter:${sn}/statuses/$ref->{'id'}][view]] " .
            "DATE=" . &descape($ref->{'created_at'}) . 
            "\n_EOTWEET_\n";
    print $streamout $string;
    return 1; # one logical tweet accepted
};

Now I can run

ttytter -exts=/home/vlb/ttytter/vlbext.pl -runcommand="/r"

Filter the Tweets

Next, I wrote a small shell/Perl script I named gottytter. This does some housekeeping tasks and runs ttytter. It also time-stamps the output every 30 minutes. (Full gottytter code can be seen at the end of this article.)

Prettification With CSS

I made a few tweaks to the look of the output, increasing the font size slightly, highlighting italic (<em>) in green, and setting color classes so that different senders are highlighted in different colors. (The latter idea came from using Adium's "Minimal_mod" message style with the "Sender Color" variant.)

CSS Code

Wrap It Up (Packaging)

Next I wrote a TWiki application that lets me search through output pages and read the latest tweets.

Then I set gottytter to run every 5 minutes, sending its results to text files in my TWiki directory.

    0,10,20,30,40,50 * * * *  $HOME/bin/gottytter

And here we are: TwitterReader.

The Code

CSS

twitterstyles.css

Show CSS Code in place

#patternOuter {
   margin-left:0;
}
#patternLeftBar {
    display:none;
}
/* for the content left margin use a bit smaller margin */
#patternMainContents {
    padding-left:2em; /*S6*/
}
.preElement
{
 white-space: pre; /* Camino */
 white-space: -moz-pre-wrap;
 white-space: pre-wrap;  /* Firefox 3 */
 width:800px;
 font-size: large;
 line-height: 120%;
}
.preElement em {
  color: #106b71;
}
.tweet {
  font-size:larger;
  line-height:110%;
}
.tweet em {
  color: #666;
}
.tweet strong em {
  color: #000000;
}
.tweet h3 {
  font-size:large;
b.p0 {color: #000000;}
b.p1 {color: #990033;}
b.p2 {color: #3333FF;}
b.p3 {color: #009966;}
b.c0 { color: #000000; }
b.c1 { color: #000033; }
b.c2 { color: #000066; }
b.c3 { color: #000099; }

gottytter

gottytter.pl

Show Code for gottytter in place

Remember to change vlb and Vicki to your own Twitter handle/name!


#!/bin/sh 

# ttytter -exts=/home/vlb/ttytter/vlbext.pl -runcommand="/r"
#
# produces something like this:
#
#  <i> <b class="p0">lane </b> </i> Not packing a rain coat or umbrella was a dick move on my part DATE=23 Oct
# _EOTWEET_
# <i> <b class="p0">yarnharlot </b> </i>   I think this rain is nature's way of saying "Don't go for a run." DATE=23 Oct
# _EOTWEET_

# NOTE: Tweets can contain embedded newlines!
# So we use \n_EOTWEET_\n as record separator

dir=$HOME/web/Twitter
DATE=`date "+%Y-%b-%d"`
TIME=`date "+%H"`
twitterlog=TwitterLog${DATE}.txt
twtime=$HOME/.twtime

cd $dir

if [ ! -f $twitterlog ]; then
    touch $twitterlog
    chmod g+w $twitterlog
fi

ttytter -exts=/home/vlb/ttytter/vlbext.pl -runcommand="/r" |
perl -e '
    use Date::Parse;

    open(TIMEDATA, "'$twtime'");
    $lasttime = <TIMEDATA>;
    close TIMEDATA;

    open(TWITTERLOG, ">>'$twitterlog'");
    $now = `date "+%H%M"`;

# Hash to match color-code to sender handle
%colors = (
  "name" => "p1",
  "name2" => "p2",
);
    $/ = "\n_EOTWEET_\n";
    while ($line = <>) {
        $star = 0;

        chomp $line;

        # Tweets can have embedded newlines
        # Convert to TWiki-style line breaks instead
        $line =~ s/\n\n/%BR%/g;
        $line =~ s/\n/%BR%/g;

        ($date = $line) =~ s/^.*DATE=(.*)$/\1/;
        $time = str2time($date);
        next if ($time <= $lasttime);

        ($sender = $line) =~  s/^(\w+) .*/\1/;
        $line =~ s/^(\w+) (.*)/\2/;
        $sender = lc($sender);

        ($tweet = $line) =~ s/^(.*)DATE=.*/\1/;

        # Filter: I do not want to see these
        next if ($tweet =~ /pattern I do not care for/i);

       # me
       # Change vlb and Vicki to your own name/handle!
       if (($tweet =~ /vlb/) && ($sender !~ /vlb/)) {
           $star = 1;
       }
       $sender = "Vicki" if ($sender =~ /vlb/);

       # Twitter links
       $tweet =~ s/@(\w+)/[[http:\/\/twitter.com\/\1][@\1]]/g;

       # Escape < in output
       $tweet =~ s/</\&lt;/g; 

       # Locked profile?
       $tweet =~ s/ -- false / -- /;
       $tweet =~ s/ -- true / -- _Locked_ /;

       # Datestamp
       $date = localtime($time);
       $date =~ s/:\d\d 20\d\d//;

       if ($star) {
         print TWITTERLOG "%S% ";
       }

       if (exists ($colors{$sender})) {
#         print "<b class=\"$colors{$sender}\">$sender</b>:";   #DEBUG
          print TWITTERLOG "<b class=\"$colors{$sender}\">$sender</b>: ";
       } else {
          print TWITTERLOG "<i><b class=\"p0\">$sender</b></i>: ";
       }
#      print $tweet, " _", $date, "_\n";  # DEBUG
       print TWITTERLOG $tweet, " _", $date, "_";
       print TWITTERLOG "\n--------------\n"
    }
    if ($now =~ /\d\d[03][0-4]/) {
        print TWITTERLOG "\n---+++ $now\n"; 
    }
    close TWITTERLOG;
    if ($time > $lasttime) {
        open(TIMEDATA, ">'$twtime'");
        print TIMEDATA $time;
        close TIMEDATA;
    }
'

Twitter Reader

View raw TWiki code

Show Code for Twitter Reader in place

Remember to change vlb to your own Twitter handle!

<noautolink>
<!---
   * Set TODAY = %SERVERTIME{"$year-$month-$day"}%
   * Set WHEN =  %URLPARAM{"when" default="%TODAY%"}%
   * Set PAGETITLE = %TOPIC% - %WHEN%
-->
---+!!  Twitter Reader (@vlb)

[[TwitterReaderWhat][What is this?]]

<table cellspacing=20><tr><td valign="top" width="60%">
<form action="%SCRIPTURL{"view"}%/Vicki/Twitter/TwitterReader" >
  <input type="text" name="when"  size="20" value="%WHEN%" id="datecell" \
  class="twikiEditFormTextField" /><input type="image" name="calendar"  \
  src="%PUBURL%/TWiki/JSCalendarContrib/img.gif" align=MIDDLE alt="Calendar" \
  onclick="return showCalendar('datecell', '%Y-%b-%d')" /> 
<br />
  <input type="submit" value="Show Me">
</form>

*Twitter Logs for [[http://twitter.com/vlb][@vlb]] from [[TwitterLog%WHEN%][%WHEN%]]*

%TWISTY{showlink="Contents" start="hide"}%
%TOC%
%ENDTWISTY%

</td><td valign="top">
%CALENDAR{daynames="Mo|Tu|We|Th|Fr|Sa|Su" showweekdayheaders="1" width="200" gmtoffset="-8"}%
</td></tr></table>
<!--
<!input type="button" value="Embiggen" onclick='Embiggen={"runNow":true}; var s=document.createElement("script"); s.type="text/javascript"; s.src="http://ghill.customer.netspace.net.au/embiggen/embiggen.js"; document.getElementsByTagName("head")[0].appendChild(s);'><!/input>
-->

#AnchorTop

<table width="900">
<tr><td>
<div class="tweet">

%INCLUDE{"TwitterLog%WHEN%"}%

</div>

</td></tr>
</table>

[[#AnchorTop][Top]]

</noautolink>

<!-- JSCALENDAR -->
%INCLUDE{"%TWIKIWEB%/JSCalendarContribInline"}%

<link rel="stylesheet" href="%PUBURL%/%WEB%/WebHome/twitterstyles.css" type="text/css" media="all" />

<!--
   * #Set COVER = print
-->

Books, Cats, Tech

"Everything that moves serves to interest and amuse a cat." — F.A. Paradis Moncrif

Vicki Brown