Categories
Uncategorized

autocomplete sh: <( compgen -d -- '' ): No such file or directory

Not much of a title, I know, but something for fellow stumped google-wanderers..

I’ve been playing around with deprec for automatically provisioning new servers for my Rails applications. However, I hit a strange problem where certain command-line auto-completions did not work; I’d get an error message like this:

$ ls
temp

$ cd [hit tab to get autocomplete]
-sh: <( compgen -d -- '' ): No such file or directory
-sh: <( compgen -f -X  -- '' ): No such file or directory

The confusing thing was that only the unpriviliged user exhibited this behaviour, not the root user.

I tried reinstalling the bash-completion package to no avail (even an aptitude purge then aptitude install did not work). Eventually I came across this rather long post on bugs.debian.org where someone else has the same problem. Essentially it comes down to the unpriviliged user using /bin/sh as the default shell; under these circumstances autocompletion is supposed to not function (although presumably without error). This is despite /bin/sh being a symlink to /bin/bash

The fix, then, is actually quite simple. Change the default shell for all your unprivileged users to bash:

# usermod -s /bin/bash dansketcher

and then when you make new users, make sure you select bash explicitly:

# useradd newuser -m -s /bin/bash
Categories
hosting

Free year-long SSL Certificate

Just discovered StartSSL’s Free SSL Certificate – 1 year (chained) SSL certificate for free.

I’m not affiliated in any way – I just like free stuff.

Categories
gems Rails rant software development

Thinking-Sphinx :with => @geodist returns no results

Having just recently started using Thinking Sphinx, I’ve been toying with the geograhical search that it provides to simply do distance-based searches while querying other information. The Geo-searching part of the manual shows how to do searches limiting distance:

# Searching for places within 10km
Place.search "pancakes", :geo => [@lat, @lng],
  :with => {"@geodist" => 0..10_000}
# Searching for places sorted by closest first
Place.search "pancakes", :geo => [@lat, @lng],
  :order => "@geodist ASC, @relevance DESC"

Unfortunately, the first style of query where @geodist is limited in range – when used as-is – does not work I found in my testing that I could order by @geodist as per the second query but when I added the :with range, I had no results.

The solution was actually simple, but really annoying. You have to use Floats instead of Integers for the range numbers! So changing the first query to:

Place.search "pancakes", :geo => [@lat, @lng],
  :with => {"@geodist" => 0.0..10_000.0}

# or more neatly

Place.search "pancakes", :geo => [@lat, @lng],
  :with => {"@geodist" => Range.new(0.0, 10_000.0)}

makes it all work. Now to get someone to update those docs!
Update: Updated!

Categories
Uncategorized

Rails.cache with Memcache and a File cache for fragments

One of the things that we do at TouchLocal is provide Search Engine Optimised pages that drive traffic to the site through the UK National Directory pages. As you can imagine, there’s quite a lot of data intensive processing that is required to generate these pages, and there are an awful lot of them. As it changes relatively rarely, for a long time we had been caching these pages in a Rails Action cache so that the headers and footers are not cached but the content is.

As we enhanced the caching strategy through the site, and the Rails support for Memcache improved, we changed the Rails cache to use Memcache. This by implication meant that the default cache we had been using for actions (the filestore cache) was no longer used and the infrastructure team could not flush the cache in the same way it previously had.

So that we could use the Rails.cache helper and still have the benefit of the Action cache, we changed our production environment configuration to look like this:

# Namespace for memcache to automatically flush on app version updates.
# APP_REVISION is updated automatically via asking the Version Control System
ns = "dir_#{RAILS_ENV}_#{$persona}_#{APP_REVISION}"

# MC_ADDRESSES constant is put into the root namespace - we use it for other things
::MC_ADDRESSES = ['10.0.0.1:11211', '10.0.0.2:11211', '10.0.0.3:11211']

# Rails.cache is memcache
config.cache_store = :mem_cache_store, MC_ADDRESSES,{:namespace => ns}

# Force the ActionController cache store to be the file one as per the previous default.
config.action_controller.cache_store = :file_store, "#{RAILS_ROOT}/tmp/cache"

Happy days.

Categories
Uncategorized

Integrating Zeus ZXTM Load Balancer and Capistrano for Ruby on Rails websites

At TouchLocal, one of the difficulties in automating deployment was the tight coupling between the app servers that were currently running and what the Load Balancer said was running. The implication of this is that at deployment time a human had to go into the ZXTM interface and update the list of running nodes before and afterwards. To ease this, and allow fully automatic deployment of Ruby on Rails apps running under Phusion Passenger, I built the ZXTMManager class.

This class implements an interface to the Zeus ZXTM API, allowing (in its current state) the ability to programatically change active nodes and manage other details of the commonly changed data in the Load Balancer. On top of that, there is an included deploy.rb file that shows the deployment recipes that can be simply and easily added to your own Capistrano configuration to use this class quickly and easily.

Enjoy.

Categories
Uncategorized

Using DataFabric to replace Masochism

When we upgraded from Rails 1.x to 2.x, we also had to migrate from Masochism to DataFabric. Unfortunately, the DataFabric way of doing things requires adding a declaration to each of the model classes. For a site like TouchLocal, where there are literally hundreds of model classes, this was a daunting task.

I’ve just gone through extracting the magic that makes it possible to do this programatically, released as DataFabric::Initializer. I hope it helps someone other than me!

Categories
Uncategorized

Howto recover lost commits from a git rebase

Yesterday, I was working on a branch of a branch in a git repository, and I wanted to merge the last branch back to master. Following chapter 3.6 of ProGit, I ran

git rebase --onto master branch1 branch2

Unfortunately for me, I seem to have either done it from the wrong place or done the wrong thing for my situation, because git rewound branch2 to the revision that it was branched from master and all the later commits were inaccessible!

However, I was not going to give up quite so easily, and I figured that once a revision was committed, it had to be somewhere.. Fortunately for me, I manged to find this post on how to recover lost commits in a git repository – I subsequently ran:

git reflog show
# find the commit with the last set of changes in your branch - 
# good commit comments are so useful!
git co b6654bd
git co -b branch2_2

With my branch recovered, I did a simple rebase of the new (old) branch onto master and merged it, and all was well.

Categories
gems Rails software development

Workling support for Synchronous AMQP RabbitMQ Clients and Amazon SQS Queues

As a part of contracting work I have been doing for TouchLocal, I have just opensourced some code I wrote to support new Workling clients. As you may know, Workling is a Rails-oriented system for performing asynchronous processing and optionally returning data from these background workers. However, because of the implementation of the original AMQP client, you could not use RabbitMQ queues from non-evented Mongrel or Phusion Passenger servers (only evented Mongrel or Thin).

Building on the work of celdee-bunny and famoseagle-carrot, I implemented a RabbitMQ workling client that could be used from within Phusion Passenger and Mongrel. The Synchronous AMQP Workling client allows RabbitMQ to be used from Workling without requiring complicated changes to deployment scenarios. Also, I implemented the Return Store functionality, so that RabbitMQ users can get data back from the workers, just like when using Starling.

Additionally, it was useful at the time to add support for an Amazon SQS Workling client, more as an exercise in testing its performance than anything else. As with the SyncAmqpClient, support for the Return Store is present. One of the discoveries in working with Amazon SQS (via the RightAws gem) was the discovery that the default key structure for Workling (which uses colon characters for segment delimiters) is not supported by Amazon. As a result, if you define the keys used for AWS configuration, even if you don’t use them, they will change the Workling key structure. This is not a problem for new implementations, but for existing deployments adding this will mean that Workling cannot see the old queues and you may not be able to access them without removing the AWS configuration… not a deal breaker, but something to be aware of.

So, the TouchLocal github account holds the version of Workling that has these two implementation for now, at least until my pull request to the main branch is accepted :)

Categories
gems hosting Rails software development

touchlocal-openx gem released

After the last post, I spent a bit of time integrating my work (along with other upgrades) into a fork of the openx code on github and have released a gem version of it on GemCuttter.org

As you may or may not know, GemCutter.org will be the new default gem source by becoming rubygems.org. This is pretty exciting, because previously you had to have you project registered on rubygems.org in order to publish to it, or use the non-standard github gem host. Good stuff.

In any case, the new OpenX gem can be installed now by executing

sudo gem install touchlocal-openx --source "http://gemcutter.org"

# Load it using
require 'rubygems'
gem 'touchlocal-openx' 
require 'openx'

In Rails, include it like this in your Rails::Initializer block:

  config.gem "touchlocal-openx", :lib => "openx", :source => "http://gemcutter.org"

I’m a proud parent!

Categories
hosting Rails software development

Minimum OpenX XMLRPC Ruby Client

2009-11-04 Update: Gem version released

For TouchLocal I am currently reworking some of the internal advertising systems to use OpenX. That way we get better reporting and better reliability, but we can use things like OpenX Direct Selection to make the best use of our existing infrastructure.

While there are 2 Ruby projects (1 a more recent fork of the other), both are oriented around the API for administering OpenX rather than serving ads via the API. So, here is the minimum you need to do from Ruby to get a banner served:

(Based on http://www.openx.org/en/docs/tutorials/Advanced+XML-RPC)

require 'xmlrpc/client'
# The settings are the  HTTP Headers - the PHP client sets many, but this is the minimum requirement
settings = {:cookies => [], :remote_addr => 'localhost'}

# 'what' in our case is a Direct Selection
params = [
  what = 'Plumber',
  campaignId = 0,
  target = '',
  source = '',
  withText = false,
  xmlContext = [{'!=' => 'campaignid:2'}]
]

b = nil
begin
  server = XMLRPC::Client.new2("http://www.example.com/www/delivery/axmlrpc.php")
  b = server.call('openads.view', settings, *params)
  b['html']
rescue Exception => e
  puts e.message
end

In particular, note the content of the xmlContext parameter – that took me quite a few hours of digging through the PHP to work out. It’s an array of hashes, that use the keys “==” and “!=” to include or exclude banners based on the values. The values must be of the form “type:id”, with type being one of “campaignid”, “clientid”, “companionid”, and “bannerdid” (although technically, anything other than the first 3 is treated as a banner ID as of version 2.8.1 of OpenX)

I hope that saves someone the pain that I just went through to discover it!