Capistrano Multistage file encoding gotcha

We use Capistrano exclusively for deployment to live and pre-live environments, so imagine my surprise when, copying a configuration into files on a new app, that I started seeing this:

Flexo:app dansketcher$ cap production misc:show_path
    triggering load callbacks
  * 2013-01-31 13:17:53 executing `production'
    triggering start callbacks for `misc:show_path'
  * 2013-01-31 13:17:53 executing `multistage:ensure'
  * 2013-01-31 13:17:53 executing `misc:show_path'
  * executing "echo $PATH"
`misc:show_path' is only run for servers matching {}, but no servers matched

The ‘production’ stage was configured, but even adding things like raise '' in the config/deploy/production.rb file failed to have an effect. It was like the file wasn’t even being loaded, except that trying to execute against a non-existent stage (‘work_damn_you’ stage, anyone?) did raise an error.

In the end, out of desperation, I ran dos2unix on the files, which weirdly complained about a binary file. Hrm. I re-saved the files as UTF-8 and all started working.

So be warned.


Unfuddle Git deploy gotcha

Having provisioned a new server, I was updating my Capistrano deploy.rb to handle the new box, and on testing a new deploy, I was getting this:

** [server :: out] == Unfgit: =============================================================
** [server :: out] Permission denied public key error. account: xxxxx; repository: xxxxx; person: none; message: no person found on this account using this public key
** [server :: out] ========================================================================

Annoying, seeing as it worked just find on the old box.

I went onto the machine and confirmed using the ssh key there I could check out the repo properly. Unfuddle support tried to help but could not. I fortunately came across this github resource which explains some of the settings. I think on the third or so re-reading I noticed this real impact of this

Agent Forwarding

If you’re using your own private keys for git you might want to tell Capistrano to use agent forwarding with this command. Agent forwarding can make key management much simpler as it uses your local keys instead of keys installed on the server.

ssh_options[:forward_agent] = true

So, I tried removing that line from my cap deploy.rb, and the checkout worked fine.

Hope this helps someone else!


Configuring Rails and Splunk (Part 3)

Success! I have managed to get Splunk to group Rails logs into logical groups.

First, I changed my Rails web server to be a light forwarder, as I wanted the indexer to do the grouping (as per the behaviour breakdown from the previous part of this series). Then:


host =

disabled = false
followTail = 0
sourcetype = rails_log

disabled = false
followTail = 0
sourcetype = access_combined

I’m not sure if I need to have the [rails_log] declaration on the LightForwarder’s etc/system/local/props.conf file, but it’s there as an empty declaration just in case.

Before we go further, note that my Rails logging uses log4r and a custom formatter, so my logs look like this:

[ApplicationName][INFO] 100819 06:37:07 :: 

Processing SessionsController#new (for at 2010-08-19 06:37:07) [GET]
[ApplicationName][INFO] 100819 06:37:07 ::   Parameters: {"unauthenticated"=>"true", "action"=>"new", "controller"=>"sessions"}
[ApplicationName][INFO] 100819 06:37:07 :: Rendering template within layouts/agent
[ApplicationName][INFO] 100819 06:37:07 :: Rendering sessions/new
[ApplicationName][INFO] 100819 06:37:07 :: Completed in 20ms (View: 12, DB: 5) | 200 OK []
[ApplicationName][INFO] 100820 01:01:25 :: 

Processing Index#index (for at 2010-08-20 01:01:25) [GET]
[ApplicationName][INFO] 100820 01:01:25 ::   Parameters: {"action"=>"index", "controller"=>"indes"}
[ApplicationName][INFO] 100820 01:01:25 :: Filter chain halted as [:authenticate] rendered_or_redirected.
[ApplicationName][INFO] 100820 01:01:25 :: Completed in 3ms (View: 1, DB: 4) | 401 Unauthorized []

The important thing to note here is that Rails inserts 2 linebreaks before putting “Processing blah blah” so we’ll use that to split. In the props.conf documentation, it recommends using the LINE_BREAKER item to handle multiline breaks, and it seemed easier to me as it uses just one regexp to break the input stream. The important part for this kind of splitting is to know:

* Wherever the regex matches, the start of the first matching group is considered the end of the 
  previous event, and the end of the first matching group is considered the start of the next event.
* The contents of the first matching group is ignored as event text.

What this means is that the LINE_BREAKER regexp (using PCRE) must have a match group that can be discarded for matching. If you grep over the default provided files (`grep -Rn “LINE_BREAKER” /opt/splunk/`), you’ll see that most of the options use a linebreak or multiples thereof as the first match group. We’ll do the same. Other than that, we just want to match on the “Processing” text as the start delimiter… which comes out to the below…

On the indexer’s $SPLUNK_HOME/etc/system/local/props.conf

LINE_BREAKER = ([\r\n]).* \:\: [\r\n]+Processing

Now, on the index, the rails_log items are multiline events!


Getting Phusion Passenger to run under SELinux on Centos 5.4

Edit 2010-03-17: Harder than it first seemed! That permission only allows Apache to load Passenger. See here for the full requirements

I’ve just started playing with SELinux on Centos, and while the idea is great, it’s not exactly what I’m used to. Take for example adding Phusion Passenger to Apache. When I first restarted, I got this

[root@localhost modules]# /etc/init.d/httpd restart
Stopping httpd:                                            [FAILED]
Starting httpd: httpd: Syntax error on line 210 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.d/passenger.conf: Cannot load /opt/ruby-enterprise-1.8.7-2009.10/lib/ruby/gems/1.8/gems/passenger-2.2.9/ext/apache2/ into server: /opt/ruby-enterprise-1.8.7-2009.10/lib/ruby/gems/1.8/gems/passenger-2.2.9/ext/apache2/ failed to map segment from shared object: Permission denied


So it turns out after reading through some google results and then subsequently man httpd_selinux that the following will fix it:

chcon -R -h -t httpd_sys_script_exec_t /opt/ruby-enterprise-1.8.7-2009.10/lib/ruby/gems/1.8/gems/passenger-2.2.9/ext/apache2/

And so it does. “httpd_sys_script_exec_t” allows Apache to execute the SO, and we’re all good to restart.


autocomplete sh: <( compgen -d -- '' ): No such file or directory

Not much of a title, I know, but something for fellow stumped google-wanderers..

I’ve been playing around with deprec for automatically provisioning new servers for my Rails applications. However, I hit a strange problem where certain command-line auto-completions did not work; I’d get an error message like this:

$ ls

$ cd [hit tab to get autocomplete]
-sh: <( compgen -d -- '' ): No such file or directory
-sh: <( compgen -f -X  -- '' ): No such file or directory

The confusing thing was that only the unpriviliged user exhibited this behaviour, not the root user.

I tried reinstalling the bash-completion package to no avail (even an aptitude purge then aptitude install did not work). Eventually I came across this rather long post on where someone else has the same problem. Essentially it comes down to the unpriviliged user using /bin/sh as the default shell; under these circumstances autocompletion is supposed to not function (although presumably without error). This is despite /bin/sh being a symlink to /bin/bash

The fix, then, is actually quite simple. Change the default shell for all your unprivileged users to bash:

# usermod -s /bin/bash dansketcher

and then when you make new users, make sure you select bash explicitly:

# useradd newuser -m -s /bin/bash

Rails.cache with Memcache and a File cache for fragments

One of the things that we do at TouchLocal is provide Search Engine Optimised pages that drive traffic to the site through the UK National Directory pages. As you can imagine, there’s quite a lot of data intensive processing that is required to generate these pages, and there are an awful lot of them. As it changes relatively rarely, for a long time we had been caching these pages in a Rails Action cache so that the headers and footers are not cached but the content is.

As we enhanced the caching strategy through the site, and the Rails support for Memcache improved, we changed the Rails cache to use Memcache. This by implication meant that the default cache we had been using for actions (the filestore cache) was no longer used and the infrastructure team could not flush the cache in the same way it previously had.

So that we could use the Rails.cache helper and still have the benefit of the Action cache, we changed our production environment configuration to look like this:

# Namespace for memcache to automatically flush on app version updates.
# APP_REVISION is updated automatically via asking the Version Control System
ns = "dir_#{RAILS_ENV}_#{$persona}_#{APP_REVISION}"

# MC_ADDRESSES constant is put into the root namespace - we use it for other things
::MC_ADDRESSES = ['', '', '']

# Rails.cache is memcache
config.cache_store = :mem_cache_store, MC_ADDRESSES,{:namespace => ns}

# Force the ActionController cache store to be the file one as per the previous default.
config.action_controller.cache_store = :file_store, "#{RAILS_ROOT}/tmp/cache"

Happy days.


Integrating Zeus ZXTM Load Balancer and Capistrano for Ruby on Rails websites

At TouchLocal, one of the difficulties in automating deployment was the tight coupling between the app servers that were currently running and what the Load Balancer said was running. The implication of this is that at deployment time a human had to go into the ZXTM interface and update the list of running nodes before and afterwards. To ease this, and allow fully automatic deployment of Ruby on Rails apps running under Phusion Passenger, I built the ZXTMManager class.

This class implements an interface to the Zeus ZXTM API, allowing (in its current state) the ability to programatically change active nodes and manage other details of the commonly changed data in the Load Balancer. On top of that, there is an included deploy.rb file that shows the deployment recipes that can be simply and easily added to your own Capistrano configuration to use this class quickly and easily.



Using DataFabric to replace Masochism

When we upgraded from Rails 1.x to 2.x, we also had to migrate from Masochism to DataFabric. Unfortunately, the DataFabric way of doing things requires adding a declaration to each of the model classes. For a site like TouchLocal, where there are literally hundreds of model classes, this was a daunting task.

I’ve just gone through extracting the magic that makes it possible to do this programatically, released as DataFabric::Initializer. I hope it helps someone other than me!


Howto recover lost commits from a git rebase

Yesterday, I was working on a branch of a branch in a git repository, and I wanted to merge the last branch back to master. Following chapter 3.6 of ProGit, I ran

git rebase --onto master branch1 branch2

Unfortunately for me, I seem to have either done it from the wrong place or done the wrong thing for my situation, because git rewound branch2 to the revision that it was branched from master and all the later commits were inaccessible!

However, I was not going to give up quite so easily, and I figured that once a revision was committed, it had to be somewhere.. Fortunately for me, I manged to find this post on how to recover lost commits in a git repository – I subsequently ran:

git reflog show
# find the commit with the last set of changes in your branch - 
# good commit comments are so useful!
git co b6654bd
git co -b branch2_2

With my branch recovered, I did a simple rebase of the new (old) branch onto master and merged it, and all was well.


can’t dup NilClass

I’m in the middle of upgrading an old Rails 1.2.6 app to Rails 2.3, and all of a sudden when logging in as a user, I get this:

TypeError in BusinessController#view

can't dup NilClass

C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:2189:in `dup'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:2189:in `scoped_methods'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:2193:in `current_scoped_methods'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:2183:in `scope'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:1548:in `find_every'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:1588:in `find_one'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:1574:in `find_from_ids'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:616:in `find'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/associations/belongs_to_association.rb:44:in `find_target'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/associations/association_proxy.rb:240:in `load_target'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/associations/association_proxy.rb:112:in `reload'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/associations.rb:1231:in `user'
C:/development/community_s0024_upgrade_upgrade/webapps/touchlocal/app/controllers/application_controller.rb:178:in `check_ticket_and_session'

Of course, that error means nothing as it is. What the underlying cause is though, is that the class reloader is trying to unload a class but it can’t because of included modules that can’t be unloaded. The fix is to use the keyword unloadable in the model to tell the reloader to force a reload on each request:

class User < ActiveRecord::Base


All fixed!