During 2009 Twitter launched their Streaming API, which allows developers to get an (almost) real-time notification of tweets. The “filter.json” resource allows some filtering of the content. With CURL you could listen to the Twitter stream like:
curl http://stream.twitter.com/1/statuses/filter.json?track=myKeyWord -umyUserAccount:password
Note that the track parameter could be also send to the stream.twitter.com server (which is a Jetty-powered service) as a POST (-d option).
To have a bit more comfort, one wants to use a (Java) REST API. Apache Wink is pretty nice and listening to the stream is simple:
ClientConfig cc = new ClientConfig();
String logon = "myAccount:myPasswd";
String encodedLogon = new BASE64Encoder().encode(logon.getBytes());
RestClient client = new RestClient(cc);
Resource twitter = client.resource("http://stream.twitter.com/1/statuses/filter.json?track=Berlin");
twitter.header("Authorization", "Basic " + encodedLogon);
twitter.contentType(MediaType.APPLICATION_FORM_URLENCODED_TYPE);
twitter.accept(MediaType.APPLICATION_JSON_TYPE);
// do the HTTP GET
ClientResponse cr = twitter.get();
// we want the input stream
InputStream is = cr.getEntity(InputStream.class);
try
{
BufferedReader r = new BufferedReader(new InputStreamReader( is, "UTF8"));
while(someCondition)
{
String rawJsonTweetData = r.readLine();
// parse the tweet and submit to your "system"
...
Once you submitted the HTTP_GET you need to transform the returned value to be InputStream. Now the stream is in your hand and you can parse the returned JSON value as you like. From there you can use the streamed data in your system.
I used the stream value to visualize the data with ADF Faces’ comet facility (aka ADS). I created a simple monitoring processor, which receives the tweets and submits them to the ADS subsystem. From there I “pushed” them to the browser, so my ADF Faces table got updated, with the latest tweets.
Note: “Following” a Twitter trend key-word can generate quite a huge amount of data, which could flood your UI (or other processing (sub)system)
There are some libraries available that give similar access to the streaming API. I found the TweetStream Ruby GEM pretty interesting, as you can run it as daemon.
