Category: Rails

Categories
big sur osx Rails

Homebrew on M1 is apparently all working – if you’re up to date

While checking how to install Ruby 3 on my new M1, I saw (also from brandur) that the full homebrew is now ready to roll!

However…..

This is really only true if you’re on the latest and greatest for all your projects. If, for example, you need a ruby <2.4 for something, then OpenSSL needs to be <1.1, and that’s not supported on arm, per https://github.com/rbenv/ruby-build/issues/1353#issuecomment-573414540. So I still set up the ibrew/abrew stuff, and then had to install old OpenSSL in the ibrew variant per above (or was it this? or this?). Btw I used the iTerm Rosetta.app variant to get things installed, and then tried to keep using it as I installed/built things, just to minimise variables to get to functional fast

ibrew install rbenv/tap/[email protected]

This only works on x86_64, so In order to get an old ruby installed, I needed double installs. In fact I needed to use x86_64 readline to get 2.4 & 2.5 installed.

# 2.3, but also works for 1.9.3-p551
RUBY_CONFIGURE_OPTS="--with-openssl-dir=$(/usr/local/bin/brew --prefix [email protected]) --with-readline-dir=/usr/local/opt/readline" arch -x86_64 rbenv install 2.3.3

# 2.4.10, 2.5.8
RUBY_CONFIGURE_OPTS="--with-readline-dir=/usr/local/opt/readline" arch -x86_64 rbenv install 2.4.10

Make sure you install rbenv yourself, not from brew, as you may need to pass some arch details

capybara-webkit still needs an old version of Qt, so you need to install the old 5.5 just like before. Make sure to follow all the steps! Including, adding it to your .profile, and make sure you run source ~/.profile before trying to compile again

pg will also probably freak out about not being able to get a clear line on which homebrew to use (especially as I installed the arm one as well), so you may need to specify

env ARCHFLAGS="-arch x86_64" gem install pg -v '0.21.0' -- --with-pg-config="$(ibrew --prefix postgresql-common)/bin/pg_config"

or

bundle config build.pg --with-pg-config="$(ibrew --prefix postgresql-common)/bin/pg_config"

libv8 and therubyracer continue to be pains in the arse:

brew install [email protected]
bundle config build.libv8 --with-system-v8
bundle config build.therubyracer --with-v8-dir=$(ibrew --prefix [email protected])
bundle install
Categories
Rails splunk

Configuring Rails and Splunk (Part 2)

As my wrestle with configuring Splunk continues, I discovered the following helpful article explaining what happens where when forwarding data to a Splunk server.

Basically, it depends. Which is not surprising, given how complicated Splunk is, but it’s good to have on hand!

Combined with details on configuring linebreaking for multi-line events, perhaps I can get my rails logs handled as one entity in the near future…

Categories
log4r Rails splunk

Configuring Rails and Splunk (Part 1)

To increase our application intelligence, I’ve started installing Splunk. However, what was not clear to me was the best way of getting my Web/App Rails logs into Splunk. The Web server is on a different box (as is the DB server) so I wanted to know the best way of getting the log data to the Splunk host.

Posts like this imply that Syslog-ng is a good way of getting data to Splunk, but the hidden detail is that if the Splunk server is down, the data is lost. A better way (thanks to the helpful folk on IRC) is to set up a Splunk forwarder on the Web/App/DB hosts and go from there. I went initially with setting them up as (free licenced) Standard Forwarders, so that the web interface is still running for configuration. Remember that any firewall (iptables) will need to be opened for the management ports (8000, 8089 ) and the forward port (9997).

Something else I discovered (but can no longer find in the docs) is that if you configure data sources using the web interface on a Forwarder, but subsequently change the instance to a LightForwarder, you also have to move the configuration files from $SPLUNK_HOME/etc/apps/search/local/ to $SPLUNK_HOME/etc/system/local/

Categories
gems Rails rant software development

Thinking-Sphinx :with => @geodist returns no results

Having just recently started using Thinking Sphinx, I’ve been toying with the geograhical search that it provides to simply do distance-based searches while querying other information. The Geo-searching part of the manual shows how to do searches limiting distance:

# Searching for places within 10km
Place.search "pancakes", :geo => [@lat, @lng],
  :with => {"@geodist" => 0..10_000}
# Searching for places sorted by closest first
Place.search "pancakes", :geo => [@lat, @lng],
  :order => "@geodist ASC, @relevance DESC"

Unfortunately, the first style of query where @geodist is limited in range – when used as-is – does not work I found in my testing that I could order by @geodist as per the second query but when I added the :with range, I had no results.

The solution was actually simple, but really annoying. You have to use Floats instead of Integers for the range numbers! So changing the first query to:

Place.search "pancakes", :geo => [@lat, @lng],
  :with => {"@geodist" => 0.0..10_000.0}

# or more neatly

Place.search "pancakes", :geo => [@lat, @lng],
  :with => {"@geodist" => Range.new(0.0, 10_000.0)}

makes it all work. Now to get someone to update those docs!
Update: Updated!

Categories
gems Rails software development

Workling support for Synchronous AMQP RabbitMQ Clients and Amazon SQS Queues

As a part of contracting work I have been doing for TouchLocal, I have just opensourced some code I wrote to support new Workling clients. As you may know, Workling is a Rails-oriented system for performing asynchronous processing and optionally returning data from these background workers. However, because of the implementation of the original AMQP client, you could not use RabbitMQ queues from non-evented Mongrel or Phusion Passenger servers (only evented Mongrel or Thin).

Building on the work of celdee-bunny and famoseagle-carrot, I implemented a RabbitMQ workling client that could be used from within Phusion Passenger and Mongrel. The Synchronous AMQP Workling client allows RabbitMQ to be used from Workling without requiring complicated changes to deployment scenarios. Also, I implemented the Return Store functionality, so that RabbitMQ users can get data back from the workers, just like when using Starling.

Additionally, it was useful at the time to add support for an Amazon SQS Workling client, more as an exercise in testing its performance than anything else. As with the SyncAmqpClient, support for the Return Store is present. One of the discoveries in working with Amazon SQS (via the RightAws gem) was the discovery that the default key structure for Workling (which uses colon characters for segment delimiters) is not supported by Amazon. As a result, if you define the keys used for AWS configuration, even if you don’t use them, they will change the Workling key structure. This is not a problem for new implementations, but for existing deployments adding this will mean that Workling cannot see the old queues and you may not be able to access them without removing the AWS configuration… not a deal breaker, but something to be aware of.

So, the TouchLocal github account holds the version of Workling that has these two implementation for now, at least until my pull request to the main branch is accepted :)

Categories
gems hosting Rails software development

touchlocal-openx gem released

After the last post, I spent a bit of time integrating my work (along with other upgrades) into a fork of the openx code on github and have released a gem version of it on GemCuttter.org

As you may or may not know, GemCutter.org will be the new default gem source by becoming rubygems.org. This is pretty exciting, because previously you had to have you project registered on rubygems.org in order to publish to it, or use the non-standard github gem host. Good stuff.

In any case, the new OpenX gem can be installed now by executing

sudo gem install touchlocal-openx --source "http://gemcutter.org"

# Load it using
require 'rubygems'
gem 'touchlocal-openx' 
require 'openx'

In Rails, include it like this in your Rails::Initializer block:

  config.gem "touchlocal-openx", :lib => "openx", :source => "http://gemcutter.org"

I’m a proud parent!

Categories
hosting Rails software development

Minimum OpenX XMLRPC Ruby Client

2009-11-04 Update: Gem version released

For TouchLocal I am currently reworking some of the internal advertising systems to use OpenX. That way we get better reporting and better reliability, but we can use things like OpenX Direct Selection to make the best use of our existing infrastructure.

While there are 2 Ruby projects (1 a more recent fork of the other), both are oriented around the API for administering OpenX rather than serving ads via the API. So, here is the minimum you need to do from Ruby to get a banner served:

(Based on http://www.openx.org/en/docs/tutorials/Advanced+XML-RPC)

require 'xmlrpc/client'
# The settings are the  HTTP Headers - the PHP client sets many, but this is the minimum requirement
settings = {:cookies => [], :remote_addr => 'localhost'}

# 'what' in our case is a Direct Selection
params = [
  what = 'Plumber',
  campaignId = 0,
  target = '',
  source = '',
  withText = false,
  xmlContext = [{'!=' => 'campaignid:2'}]
]

b = nil
begin
  server = XMLRPC::Client.new2("http://www.example.com/www/delivery/axmlrpc.php")
  b = server.call('openads.view', settings, *params)
  b['html']
rescue Exception => e
  puts e.message
end

In particular, note the content of the xmlContext parameter – that took me quite a few hours of digging through the PHP to work out. It’s an array of hashes, that use the keys “==” and “!=” to include or exclude banners based on the values. The values must be of the form “type:id”, with type being one of “campaignid”, “clientid”, “companionid”, and “bannerdid” (although technically, anything other than the first 3 is treated as a banner ID as of version 2.8.1 of OpenX)

I hope that saves someone the pain that I just went through to discover it!

Categories
Rails software development

Boolean.parse

So often in Ruby on Rails I find myself needing to convert a variable to a Boolean, and there is no built in way of doing it. So, how about this:

class Boolean
  def self.parse(obj)
    %w(true t 1 y).include?(obj.to_s.downcase.strip)
  end
end

We use this at work and it’s a nice neat way of ensuring non-repetitive code reuse :)

Categories
Rails

Howto get capistrano-ext to run migrations as the correct environment

If like me, you use the Capistrano Multistage extensions in the capistrano-ext plugin, you’d have noticed to your horror at some point that if you deploy to an environment other than production, your migrations will still run against the production database.

As the extension has no native support for this change in behaviour, the following method override will do the trick:

namespace :deploy do
  desc "Invoke the db migration using the correct stage"
  task :migrate, :roles => :app do
    send(run_method, "cd #{release_path} && rake db:migrate RAILS_ENV=#{stage} ")
  end
end

While I’m here, here’s a simple trick I came up with: at work we use SVN branches heavily, some for each Sprint and then for Releases. As each release has its own branch, we need to change the Capistrano repository URL for each release, which is tedious and easy to forget.

Instead of the manual change, why not just ask SVN to tell you the current URL for the current path, which is in our case the root of the app? If, during the cap deploy, we run some ruby code to call ‘svn info’, like so (the first line is not necessary – it’s just there for context):

set :scm, :subversion
lines = %x[svn info]

The ‘svn info’ command obviously relies on the svn executable being in your path, but it is required for the rest of deployment so I’ll assume that’s already taken care of. The output of the command is something like:

Path: .
URL: https://svn.example.com/svn/project/branches/release_12345
Repository Root: https://svn.example.com/svn
Repository UUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Revision: 1359
Node Kind: directory
Schedule: normal
Last Changed Author: dan
Last Changed Rev: 13582
Last Changed Date: 2009-03-17 22:34:44 +0000 (Thu, 19 Feb 2009)

the %x[] command takes this output and puts it in a long string with line breaks. Then, because this output is well formed, we can use a little regex-fu to get out the parh of the repository, like so (full commands):

lines = %x[svn info]
set :repository, /URL\: (.*)\n/.match(lines)[1]

The regular expression we use here is looking for a string that is between “URL: ” and “\n”, which as you would see from the output above, is:

https://svn.example.com/svn/project/branches/release_12345

Just what we were after!

Categories
Rails

Running one migration by hand

Sometimes, you want to run a single migration by hand (to test it maybe…)

You can do this from the command line:


ruby script/runner 'require "db/migrate/005_create_blogs"; CreateBlogs.migrate(:down)'
ruby script/runner 'require "db/migrate/005_create_blogs"; CreateBlogs.migrate(:up)'