Recursive Twitter search with PHP

I’ve been playing with the Twitter API recently and more specifically the Twitter search API. Sometimes the results from a Twitter search span across multiple pages and because of this, the result from a call to the Twitter search API may include a next_page variable. Using this variable we can recursively search Twitter until there are no more results. In this blogpost I’ll show you how.

I didn’t write my own Twitter search library. Instead, I extended the excellent PHPTwitterSearch library by Ryan Faerman. Don’t forget to download that and include or require it before using this class. Here is the code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
<?php
require 'PHPTwitterSearch.php';
 
class RecursiveTwitterSearch extends TwitterSearch {
    var $request_count = 0;
    var $max_request_count = 5;
    var $max_id;
 
    function recursive_results() {
        $request  = 'http://search.twitter.com/search.'.$this->type;
        $request .= '?q='.urlencode($this->query);
 
        if(isset($this->rpp)) {
            $request .= '&rpp='.$this->rpp;
        }
 
        if(isset($this->page)) {
            $request .= '&page='.$this->page;
        }
 
        if(isset($this->lang)) {
            $request .= '&lang='.$this->lang;
        }
 
        if(isset($this->since)) {
            $request .= '&since_id='.$this->since;
        }
 
        if($this->show_user) {
            $request .= '&show_user=true';
        }
 
        if(isset($this->geocode)) {
            $request .= '&geocode='.$this->geocode;
        }
 
        if(isset($this->max_id)) {
            $request .= '&max_id='.$this->max_id;
        }
 
        $response = $this->objectify($this->process($request));
        $this->request_count++;
        if ($response) {
            $results = $response->results;
            if(!empty($response->next_page)) {
                preg_match('|?page=([0-9]*)&max_id=([0-9]*)&|', $response->next_page, $matches);
                $this->page = $matches[1];
                $this->max_id = $matches[2];
                if ($this->request_count < $this->max_request_count) {
                    $results = array_merge($results, $this->recursive_results());
                }
            }
            return $results;
        }
        else return false;
    }
 
    function max_request_count($n) {
        $this->max_request_count = $n;
        return $this;
    }
}
?>

Basically what I did is duplicate the results() function from the parent class and add some code to it. Now, recursive_results() calls the API and when the variable next_page is present it will call itself again and merge the result sets. This process keeps on going either until we’re at the last page of search results or until a certain number of requests have been made. You can set this number yourself with the function max_request_count().

Here is some example code:

1
2
3
4
5
<?php
$rts = new RecursiveTwitterSearch('starbucks');
print_r($rts->recursive_results());
echo 'It took us ' . $rts->request_count . ' request(s) to get this result.';
?>

Got questions? Found a bug? Ideas for optimizations? General shout-outs? I’d love to hear them! :-)

Working quicker in Ubuntu by using bash scripts

I’m sure that there are a couple of commands you use on a regular basis when working in Ubuntu. I for instance often open the file browser by pressing ALT+F2 and entering ‘gksu nautilus’ which opens the file browser as root. The same goes for the ‘gksu gnome-terminal’ command. Entering commands with ALT+F2 is cool and all but when the commands start to leave a lot of room for typo’s it can be annoying.

I knew that I could make bash scripts in Linux to execute certain commands so I figured I’d simply make some bash scripts for these commands. The way to do that is like so:

Press ALT+F2 and enter ‘gksu gedit /usr/local/bin/sunau’. This opens the text editor with new file called ‘sunau’ in the location /usr/local/bin. We’ll make a script in this file that opens the file browser as root. Enter the following into the empty file:

#!/bin/bash
gksu nautilus

Now save the script and close the editor. The only thing left now is to make the script executable. Press ALT+F2 again and enter ‘sudo chmod +x /usr/local/bin/sunau’. After this you can call your script by pressing ALT+F2 and entering ‘sunau’. This will start the file browser as root. By changing the contents of the file you can execute all kinds of commands with bash scripts.

How to transfer ALL your e-mail to Gmail (even if it’s in folders)

I recently helped my mother with switching over to Gmail. She has always used the webmail service from our webhost with an e-mail adress at amazoneadvies.nl (she runs a dutch PR consultancy) but the webmail service had been buggy lately. Setting up Gmail to receive e-mail from another account with POP3 access is easy but I was getting worried when I noticed the mail that she had put in folders wasn’t coming through. When I went googling for a clue I found the Gmail answer to this problem:

You’re only receiving mail from the inbox of your other account and not from any folders. Each webmail provider determines which messages to make available for POP download and sends a list of the messages to Gmail. Since Gmail can’t retrieve any messages that aren’t on the list, we suggest contacting your other provider to learn about alternatives.

I didn’t really feel like doing it this way, it meant waiting for our hosting provider to respond and everything. Plus I wasn’t convinced that Gmail would automatically label the messages with the folder name. So I had to come up with another solution.

Gmail allows you to access your e-mail through IMAP which means that you can access your mail with a standard desktop e-mail client like Mozilla Thunderbird. Every change you make in Thunderbird (such as deleting a message) will reflect in your Gmail as well (in contrast to making changes with POP3 access). Our hosting company provides IMAP access as well. What I did was the following:

I turned on IMAP access in Gmail and configured Thunderbird so I could access my mother Gmail (read all about it). Then I created another account in Thunderbird to access her mailbox at our hosting provider (the one the still held some mail in folders). After doing this I could easily create folders for Gmail in Thunderbird (Gmail converts them to labels automatically) that match the folders at our hosting provider. After that I could simply drag and drop all the mail from the hosting provider to Gmail (hold [CTRL] while dragging and dropping to copy instead of move). Voila!

This way all the e-mail that was archived in folders at our hosting provider now appears under matching labels in Gmail.