Benchmarking acts_as_ferret and acts_as_solr

The Problem

I've been a fan of the acts_as_ferret plugin for a while and have had great success in using it while developing Rails applications using WEBrick. It's a cinch to use and can be highly configurable. However, once I deployed applications that used acts_as_ferret to production things started to come apart. acts_as_ferret simply didn't work in a multi-process (read fastcgi) or multi-server environment.

The problem is due to concurrency issues between processes. Each fastcgi process assumes it is the only one using the index files and would not respect other processes' actions. Under heavy loads, and during concurrent read/write accesses, file locking breaks down and the application begins to throw errors (including some nasty segfaults).

The problem gets worse when not only are there multiple processes on a server but there are multiple servers as well. In this configuration, keeping the ferret index files separated on each server isn't an option as that would lead to out of sync indices. The most obvious solution is to use a centralized location for these files and link each server to it. But that is the same situation as above, only this time there are more processes!

So, is acts_as_ferret a 'development-only' plugin that's not ready for production? Fortunately not!


The Solution

There is now a DRb Server implementation for acts_as_ferret. From the authors:

"In production environments most often multiple processes are responsible for serving client requests. Sometimes these processes are even spread across several physical machines.

Just like the database, the Ferret index of an application is a unique resource that has to be shared among all servers. To achieve this, acts_as_ferret comes with a built in DRb server that acts as the central hub for all indexing and searching in your application."

Perfect! Now acts_as_ferret is ready for production environments. The only question is, "How does it scale?"

I recently ran across another 'acts_as_' plugin for Rails, called acts_as_solr, that has many of the same features, and many that acts_as_ferret doesn't, but its index server is implemented in Java using the apache-solr engine. I began to wonder which one was faster and thus would scale better. Time for a benchmark!


The Benchmark

System

• CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (using non-smp kernel)
• RAM: 2GB
• OS: Kubuntu Edgy (latest updates as of 3/14/07)
• Ruby: ruby 1.8.4 (2005-12-24) [i686-linux]
• Rails: 1.1.6
• MySQL: mysql Ver 14.7 Distrib 4.1.15, for pc-linux-gnu (i486) using readline 5.1
• ferret gem: 0.11.3
• acts_as_ferret: svn://projects.jkraemer.net/acts_as_ferret/trunk/plugin/acts_as_ferret (as of 3/14/07)
• acts_as_solr: http://opensvn.csie.org/acts_as_solr/trunk (as of 3/13/07)
• apache-solr: apache-solr-1.1.0-incubating

Hard Drive information ('hdparm -I /dev/hda')

ATA device, with non-removable media
        Model Number:       WDC WD800BB-22HEA1
        Serial Number:      WD-WMAJ51461221
        Firmware Revision:  14.03G14
Standards:
        Supported: 6 5 4
        Likely used: 6
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:   16514064
        LBA    user addressable sectors:  156301488
        device size with M = 1024*1024:       76319 MBytes
        device size with M = 1000*1000:       80026 MBytes (80 GB)
Capabilities:
        LBA, IORDY(can be disabled)
        Standby timer values: spec'd by Standard, with device specific minimum
        R/W multiple sector transfer: Max = 16  Current = 0
        Recommended acoustic management value: 128, current value: 254
        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5
             Cycle time: min=120ns recommended=120ns
        PIO: pio0 pio1 pio2 pio3 pio4
             Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
        Enabled Supported:
           *    SMART feature set
           *    Power Management feature set
           *    Write cache
           *    Look-ahead
           *    Host Protected Area feature set
           *    WRITE_BUFFER command
           *    READ_BUFFER command
           *    DOWNLOAD_MICROCODE
                SET_MAX security extension
                Automatic Acoustic Management feature set
           *    Device Configuration Overlay feature set
           *    Mandatory FLUSH_CACHE
           *    SMART error logging
           *    SMART self-test
HW reset results:
        CBLID- above Vih
        Device num = 0 determined by CSEL

Network
Everything was done on a single machine (see assumptions as to why).

Benchmark Routine
This routine simply loops a specified number of times, evaluates a given routine, and outputs basic statistical information. It catches all exceptions thrown (caused by passing in invalid characters as part of the query) and treats that timing instance as noise (not included in success totals).

def benchmark_routine(times, routine)
  success_total_time = 0.0
  error_total_time = 0.0
  successes = 0
  errors = 0
  high = nil
  low = nil
  times.times do
    start_time = Time.now
    begin
      eval routine
      time = Time.now - start_time
      high = time if high == nil or time > high
      low = time if low == nil or time < low
      success_total_time += time
      successes += 1
    rescue Exception => e
      time = Time.now - start_time
      error_total_time += time
      errors += 1
    end
  end
  return {:successes => successes,
          :success_total_time => success_total_time,
          :success_high => high,
          :success_low => low,
          :success_average => success_total_time / successes
          :errors => errors,
          :error_total_time => error_total_time  }
end

ActiveRecord Class
Note: only one of the 'acts_as' declarations was uncommented at a time.

class Article < ActiveRecord::Base
  # acts_as_solr :fields => [ :url, :title, :description ]
  # acts_as_ferret ({ :fields => [ :url, :title, :description ], :remote => true })
end

Database
The table I used contained 414 rows of news articles pulled from various news feeds (Yahoo, MSNBC, etc.). The columns indexed were the url of the full article, the title of the article, and a synopsis of the article content. The article table is as follows:

CREATE TABLE `articles` (
  `id` int(11) NOT NULL auto_increment,
  `url` text NOT NULL,
  `title` varchar(255) NOT NULL default '',
  `description` text NOT NULL,
  `article_date` datetime default NULL,
  `created_at` datetime default NULL,
  PRIMARY KEY  (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Assumptions
All routines were performed on the same local machine. Thus these tests do not account for any latency that may occur when in an environment distributed across different machines on a network. Any latency differences between a multi-server environment versus a localhost-only environment would be constant, and thus not affect the relative performance, as both plugins are opening sockets to connect to their respective servers.

I am assuming that 414 rows of aggregated news article summaries from several different sites provides a large enough pool of information that searches terms pulled from these articles can be considered random.

Routines
There were four different routines ran for each plugin: random search with no background updates, cached search with no background updates, random search with continuous background updates, cached search with no background updates. Each routine/plugin combination ran a benchmark of 10, 100, 1000, and 10000 queries.

Random search with No Background Updates:
The routines I used for random searching is as as follows:

acts_as_solr

Article.find_by_solr(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])

or

acts_as_ferret

Article.find_by_contents(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])

Article.find(:first, :order => 'rand()').description.split(' ')[rand()] is used as the search term to provide random queries by pulling a random word from a random article description. The idea here is to see what non-cached query benchmarks are. I am not accounting for time taken to perform the query to get a random word from the database.

Cached search with No Background Updates:
As many of the feeds used to generate the articles table were of a technical nature I chose the word 'computer' to maximize the number of matches. The routine I used for cached searching is as follows:

acts_as_solr

Article.find_by_solr('computer')

or

acts_as_ferret

Article.find_by_contents('computer')

Random Search with Continuous Background Updates:
Using two different 'script/console's, one would continuously select a random article and save it, thus updating the indices, while the other continuously ran the random query as described above.

Cached search with Continuous Background Updates:
Using two different 'script/console's, one would continuously select a random article and save it, thus updating the indicies, while the other continuously ran the cached query as described above.

Results
Here is the output from two script/console sessions (one for acts_as_solr and the other for acts_as_ferret):

acts_as_solr Random Query Test (with no background updates):

>> Article.rebuild_solr_index
=> true
>> benchmark_routine(10, "Article.find_by_solr(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_high=>0.082648, :error_total_time=>0.013225, :success_low=>0.009316, :successes=>9, :success_average=>0.0225318888888889, :errors=>1, :success_total_time=>0.202787}
>> benchmark_routine(100, "Article.find_by_solr(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_high=>0.061882, :error_total_time=>0.0, :success_low=>0.005442, :successes=>100, :success_average=>0.01048266, :errors=>0, :success_total_time=>1.048266}
>> benchmark_routine(1000, "Article.find_by_solr(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_high=>0.061038, :error_total_time=>0.129032, :success_low=>0.00526, :successes=>990, :success_average=>0.0103782050505051, :errors=>10, :success_total_time=>10.274423}
>> benchmark_routine(10000, "Article.find_by_solr(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_high=>0.10479, :error_total_time=>1.821459, :success_low=>0.005218, :successes=>9865, :success_average=>0.0105283624936645, :errors=>135, :success_total_time=>103.862296}

acts_as_ferret Random Query Test (with no background updates):

>> Article.rebuild_index
=> {}
>> benchmark_routine(10, "Article.find_by_contents(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_high=>0.07014, :error_total_time=>0.0, :success_low=>0.007163, :successes=>10, :success_average=>0.014553, :errors=>0, :success_total_time=>0.14553}
>> benchmark_routine(100, "Article.find_by_contents(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_high=>0.053758, :error_total_time=>0.0076, :success_low=>0.006882, :successes=>99, :success_average=>0.00910739393939394, :errors=>1, :success_total_time=>0.901632}
>> benchmark_routine(1000, "Article.find_by_contents(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_high=>1.15512, :error_total_time=>0.193326, :success_low=>0.006969, :successes=>975, :success_average=>0.0103111794871795, :errors=>25, :success_total_time=>10.0534}
>> benchmark_routine(10000, "Article.find_by_contents(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_high=>0.07639, :error_total_time=>1.869886, :success_low=>0.006948, :successes=>9770, :success_average=>0.00901357379733882, :errors=>230, :success_total_time=>88.0626160000002}

acts_as_solr Cached Query Test (with no background updates):

>> Article.rebuild_solr_index
=> true
>> benchmark_routine(10, "Article.find_by_solr('computer')")
=> {:success_high=>0.043805, :error_total_time=>0.0, :success_low=>0.003642, :successes=>10, :success_average=>0.0081675, :errors=>0, :success_total_time=>0.081675}
>> benchmark_routine(100, "Article.find_by_solr('computer')")
=> {:success_high=>0.050668, :error_total_time=>0.0, :success_low=>0.003121, :successes=>100, :success_average=>0.00472935, :errors=>0, :success_total_time=>0.472935}
>> benchmark_routine(1000, "Article.find_by_solr('computer')")
=> {:success_high=>0.074675, :error_total_time=>0.0, :success_low=>0.003167, :successes=>1000, :success_average=>0.00468432899999999, :errors=>0, :success_total_time=>4.68432899999999}
>> benchmark_routine(10000, "Article.find_by_solr('computer')")
=> {:success_high=>0.145258, :error_total_time=>0.0, :success_low=>0.003079, :successes=>10000, :success_average=>0.0047008854, :errors=>0, :success_total_time=>47.008854}

acts_as_ferret Cached Query Test (with no background updates):

>> Article.rebuild_index
=> {}
>> benchmark_routine(10, "Article.find_by_contents('computer')")
=> {:success_high=>0.055753, :error_total_time=>0.0, :success_low=>0.003231, :successes=>10, :success_average=>0.0095941, :errors=>0, :success_total_time=>0.095941}
>> benchmark_routine(100, "Article.find_by_contents('computer')")
=> {:success_high=>0.048506, :error_total_time=>0.0, :success_low=>0.003188, :successes=>100, :success_average=>0.00418007, :errors=>0, :success_total_time=>0.418007}
>> benchmark_routine(1000, "Article.find_by_contents('computer')")
=> {:success_high=>0.05537, :error_total_time=>0.0, :success_low=>0.003131, :successes=>1000, :success_average=>0.00417725, :errors=>0, :success_total_time=>4.17725}
>> benchmark_routine(10000, "Article.find_by_contents('computer')")
=> {:success_high=>0.13851, :error_total_time=>0.0, :success_low=>0.003122, :successes=>10000, :success_average=>0.0042286392, :errors=>0, :success_total_time=>42.286392}

acts_as_solr Random Query Test (with background updates):
Console 1:

>> while true
>>   Article.find(:first, :order => "rand()").save
>> end

Console 2:

>> Article.rebuild_solr_index
=> true
>> benchmark_routine(10, "Article.find_by_solr(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_high=>0.163176, :error_total_time=>0.0, :success_low=>0.005903, :successes=>10, :success_average=>0.035533, :errors=>0, :success_total_time=>0.35533}
>> benchmark_routine(100, "Article.find_by_solr(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_high=>0.328657, :error_total_time=>0.03657, :success_low=>0.005748, :successes=>99, :success_average=>0.0402173939393939, :errors=>1, :success_total_time=>3.981522}
>> benchmark_routine(1000, "Article.find_by_solr(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_high=>0.395172, :error_total_time=>0.528478, :success_low=>0.005612, :successes=>989, :success_average=>0.0381039544994945, :errors=>11, :success_total_time=>37.6848110000001}
>> benchmark_routine(10000, "Article.find_by_solr(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_high=>0.428329, :error_total_time=>5.812703, :success_low=>0.004934, :successes=>9852, :success_average=>0.0349491181485993, :errors=>148, :success_total_time=>344.318712}

acts_as_solr Cached Query Test (with background updates):
Console 1:

>> while true
>>   Article.find(:first, :order => "rand()").save
>> end

Console 2:

>> Article.rebuild_solr_index
=> true
>> benchmark_routine(10, "Article.find_by_solr('computer')")
=> {:success_high=>0.109399, :error_total_time=>0.0, :success_low=>0.01582, :successes=>10, :success_average=>0.0307117, :errors=>0, :success_total_time=>0.307117}
>> benchmark_routine(100, "Article.find_by_solr('computer')")
=> {:success_high=>0.311115, :error_total_time=>0.0, :success_low=>0.004612, :successes=>100, :success_average=>0.02918882, :errors=>0, :success_total_time=>2.918882}
>> benchmark_routine(1000, "Article.find_by_solr('computer')")
=> {:success_high=>0.347518, :error_total_time=>0.0, :success_low=>0.003539, :successes=>1000, :success_average=>0.030654657, :errors=>0, :success_total_time=>30.654657}
>> benchmark_routine(10000, "Article.find_by_solr('computer')")
=> {:success_high=>0.919183, :error_total_time=>0.0, :success_low=>0.003729, :successes=>10000, :success_average=>0.0301908975999999, :errors=>0, :success_total_time=>301.908975999999}

acts_as_ferret Random Query Test (with background updates):
Console 1:

>> while true
>>   Article.find(:first, :order => "rand()").save
>> end

Console 2:

>> Article.rebuild_index
=> {}
>> benchmark_routine(10, "Article.find_by_contents(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_total_time=>0.368735, :success_high=>0.194123, :error_total_time=>0.0, :success_low=>0.007293, :successes=>10, :success_average=>0.0368735, :errors=>0}
>> benchmark_routine(100, "Article.find_by_contents(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_total_time=>1.922619, :success_high=>0.164076, :error_total_time=>0.00814, :success_low=>0.006691, :successes=>99, :success_average=>0.0194203939393939, :errors=>1}
>> benchmark_routine(1000, "Article.find_by_contents(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_total_time=>20.138275, :success_high=>0.159272, :error_total_time=>0.396986, :success_low=>0.006772, :successes=>977, :success_average=>0.0206123592630502, :errors=>23}
>> benchmark_routine(10000, "Article.find_by_contents(Article.find(:first, :order => 'rand()').description.split(' ')[rand()])")
=> {:success_total_time=>202.389328, :success_high=>1.209112, :error_total_time=>3.954261, :success_low=>0.006643, :successes=>9755, :success_average=>0.0207472401845207, :errors=>245}

acts_as_ferret Cached Query Test (with background updates):
Console 1:

>> while true
>>   Article.find(:first, :order => "rand()").save
>> end

Console 2:

>> Article.rebuild_index
=> {}
>> benchmark_routine(10, "Article.find_by_contents('computer')")
=> {:success_high=>0.118141, :error_total_time=>0.0, :success_low=>0.007559, :successes=>10, :success_average=>0.0348693, :errors=>0, :success_total_time=>0.348693}
>> benchmark_routine(100, "Article.find_by_contents('computer')")
=> {:success_high=>0.081537, :error_total_time=>0.0, :success_low=>0.004364, :successes=>100, :success_average=>0.01803433, :errors=>0, :success_total_time=>1.803433}
>> benchmark_routine(1000, "Article.find_by_contents('computer')")
=> {:success_high=>0.152265, :error_total_time=>0.0, :success_low=>0.003591, :successes=>1000, :success_average=>0.0346570519999999, :errors=>0, :success_total_time=>34.6570519999999}
>> benchmark_routine(10000, "Article.find_by_contents('computer')")
=> {:success_high=>0.257654, :error_total_time=>0.0, :success_low=>0.006658, :successes=>10000, :success_average=>0.0427653468, :errors=>0, :success_total_time=>427.653468}

Conclusion
The results were surprisingly close, with the largest margin of difference being approximately 0.01 seconds and acts_as_ferret performing faster on most test cases. I honestly would have figured a Java implementation would have been faster with all the negative press out there about Ruby benchmarking.

Now to be fair to the solr server, it does appear to have many features that the acts_as_ferret DRb server does not, and thus could be doing a lot more than just building the index files. I didn't look into this further, though that would make a good follow up to see how much (if at all) these extra features affect these results and if they can be configured to improve performance.

Here a breakdown of the results for different tests:

Test                   Winner              Avg. Margin
random (no updates)    acts_as_ferret      0.002734
random (with updates)  acts_as_ferret      0.012787
cached (no updates)    acts_as_ferret      0.000026
cached (with updates)  acts_as_solr        0.002395