Notes on SOA @ Square

My notes from Service Oriented Architecture at Square talk:

monolithic app -- becomes
bigger & bigger
- generally try to make a new service
unless it is really, really related to what's
already there

since they are independent, they can be put
in different data centers

json
(others use a messaging system: a giant queue,
where you can throw all you)

to quickly start a service,
setup a "template" to "rails new":
 - documentation, testing, code quality,
   deployment, monitoring, security
and you can reusing that template

these are apis:
so documentation driven development
docs have to "work" - makes the build green

documentation is the main thing; there is 
even a empty scaffolding generator for the
the documentation

description:
reqParams:
  properties:
    user_id:
      description:
      require:
      type:
      example:
resParams:
resCodes:

https://github.com/square/fdoc

growing oos, guided by tests

mock the services
http://ddollar.github.com/foreman/
services.yml


code quality:
https://github.com/square/cane

small, fast and can embed into your
rails app:
http://jetty.codehaus.org/jetty/
use jetpack to package it
to deploy:
jetpack . && cap deploy
https://github.com/square/jetpack

deployed and now it needs monitoring:
airbrake
hoptoad
collect exception and comment on them

monitoring tool:
cubism.js
cool dashboards
https://github.com/square/cubism

use splunk to figure out what's breaking:
search for all your logs from one place

security:
ssl certificates with mutual authentication
ssl certificate for each new service
on ssl certificate there is a key called
organizational unit -> name of service

have a acl for each service -- (25:29) including
all clients allowed to talk to this resource

how to pull out services from the monolithic rails
app: inttially take out the code then do a redirect
and then chop it off.

Distance between 2 addresses in CoffeeScript

A small script to find the distance of two addresses using the Google Maps API:

Two useful scripts

Here are two Ruby scripts that I find useful. Both of them have been tested under Ruby 1.9.2p0. The first is mv_ldown: it moves the most recently downloaded file (from ~/Downloads) into the current directory.

#! /usr/bin/env ruby
src = Dir[File.expand_path('~/Downloads/*')].
  sort_by { |x| File.mtime(x) }.
  last
abort("nothing in ~/Downloads dir") unless src
dest = File.join(File.absolute_path('.'), File.basename(src))
abort("#{dest} already exists") if File.exists?(dest)
File.rename(src, dest)
puts "#{File.basename src} to ."

The second is sl: it just like the command "ln -s SRC DEST", but you don't have to remember the order of the parameters. The file that exists is the source and the other, the destination.

#! /usr/bin/env ruby
require 'fileutils'
abort("Usage: sl SRC DEST") unless ARGV.length == 2
a, b = ARGV
if File.exists?(a) && File.exists?(b)
  abort("Both #{a} & #{b} exists")
elsif File.exists?(a)
  src, dest = a, b
elsif File.exists?(b)
  src, dest = b, a
else
  abort("Neither #{a} nor #{b} exists")
end
src = File.absolute_path src
abort("#{src} not a file") unless File.file?(src)
dest = File.absolute_path dest
system("ln -s #{src} #{dest}")

Extracting phone numbers from Outlook Msg Files

Recently, I had to extract phone numbers from a bunch of Outlook .MSG files (see here and here).

Looking for an open source solution, I came across MsgParser. It almost did what I needed. (I just had to modify a "protected" field and make it "public" -- I needed to access it from a driver program; there might be a easier way to do this -- maybe make the driver program "run" from the same package as MsgParser and it'll have access to the protected field...) But downloading its dependencies was a bit of a pain.

So here is a Ruby script that does all that for you. You specify a directory where the Outlook .MSG files are. It'll extract all the phone numbers from them and print it. Tested under OS X Leopard; needs curl & java to be installed. Run: ruby msg_extractor.rb MSGS_DIR.

# Released under the GNU GPL.
# http://www.gnu.org/copyleft/gpl.html

require 'fileutils'

DOWNLOADS = %w{
  http://auxilii.com/msgparser/download/msgparser-1.6.zip
  http://www.freeutils.net/source/jtnef/jtnef-1_6_0.zip
  http://apache.ziply.com/poi/release/bin/poi-bin-3.6-20091214.tar.gz
  http://www.trieuvan.com/apache/ant/binaries/apache-ant-1.8.0-bin.zip }

JCLASS = 'MyMsgParser'
JAVA_INPUT = 'My.msg'
JAVA = <<END
  import java.util.*;
  import com.auxilii.msgparser.*;
  import com.auxilii.msgparser.attachment.*;
  public class #{JCLASS} {
    public static void main(String[] args) throws Exception{
      MsgParser msgp = new MsgParser();
      Message msg = msgp.parseMsg("#{JAVA_INPUT}");
      // UNUSED
      String fromEmail = msg.getFromEmail();
      String fromName = msg.getFromName();
      String subject = msg.getSubject();
      String body = msg.getBodyText();
      String longString = msg.toLongString();
      // We modify source of msgparser to make
      // properties public
      Iterator it = msg.properties.keySet().iterator();
      while (it.hasNext()) {
       String key = (String) it.next();
       Object val = msg.properties.get(key);
        System.out.println(key + ": " + val);
      }
    }
  }
END

class MsgExtractor
  def initialize(msgs_dir)
    @msgs_dir = msgs_dir
    @cp = '.:msgparser-1.6/dist/msgparser-1.6.jar:' + 
          'poi-3.6/poi-3.6-20091214.jar:lib/tnef.jar'
    download
    javac
  end
  
  def download
    DOWNLOADS.each do |u|
      b = File.basename(u)
      if !File.exists?(b)
        puts "downloading #{u}"
        `curl --silent -O #{u}` unless File.exists?(b)
      end
      next if File.exists?("#{b}.zipped")
      puts "unpacking #{b}"
      case b
      when /zip$/; `unzip -o #{b}`
      when /gz$/;  `gzip -dc #{b} | tar -xpvf -`
      end
      FileUtils.touch("#{b}.zipped")
    end
  end
  
  def javac
    modify_msgparser
    @cp.split(':').each do |f|
      abort("#{f} doesn't exist") unless File.exists?(f) 
    end
    open("#{JCLASS}.java", "w") { |f| f.write(JAVA) }
    `javac -classpath #{@cp} #{JCLASS}.java`
  end
  
  def modify_msgparser
    jp = 'msgparser-1.6/src/main/java/com/auxilii/msgparser/Message.java'
    old1 = 'protected Map<String,Object> properties ' + 
           '= new HashMap<String,Object>();'
    new1 = old1.dup
    new1['protected'] = 'public'
    modified = File.read(jp)
    return if modified[new1]
    modified[old1] = new1
    puts "modifying #{jp}..."
    open(jp, 'w') { |f| f.write(modified) }
    FileUtils.cd('msgparser-1.6') do
      puts "rebuilding msgparser"
      `../apache-ant-1.8.0/bin/ant jar`
      FileUtils.mv 'dist/msgparser.jar', 'dist/msgparser-1.6.jar'
    end
  end
  
  def run(&extract_by)
    result = []
    Dir[File.join(@msgs_dir, '*.msg')].each do |f|
      b = File.basename(f)[0..-5]
      puts "extracting #{b}"
      FileUtils.cp(f, JAVA_INPUT)
      out = `java -classpath #{@cp} #{JCLASS}`
      data = extract_by.call(out)
      next if data.nil?
      break if $INTERRUPTED
      result << [b, data]
    end
    result.sort
  end
end

$INTERRUPTED = false

def main
  unless ARGV[0]
    abort("Usage: ruby msgs_extractor.rb msgs_dir")
  end
  tmp = ARGV[1] || "msgs_extractor"
  Dir.mkdir tmp rescue nil
  msgs_dir = File.expand_path ARGV[0]
  Dir.chdir tmp
  extractor = MsgExtractor.new(msgs_dir)
  trap("INT") { $INTERRUPTED = true; puts; }
  result = extractor.run { |o| o.scan /\d{7,20}/ } # extract phone numbers
  result.each { |n, ph| puts("%20s: %s" % [n, ph.join(", ")]) }
end

if $0 == __FILE__
  main
end

Don't Make Me Think

I loved Don't Make Me Think. It takes you inside the mind of the user (of your website). And it tells you who they are, what they expect and what confuses them. And the golden rule is, of course, don't make them think.