Two useful scripts

Here are two Ruby scripts that I find useful. Both of them have been tested under Ruby 1.9.2p0. The first is mv_ldown: it moves the most recently downloaded file (from ~/Downloads) into the current directory.

#! /usr/bin/env ruby
src = Dir[File.expand_path('~/Downloads/*')].
  sort_by { |x| File.mtime(x) }.
  last
abort("nothing in ~/Downloads dir") unless src
dest = File.join(File.absolute_path('.'), File.basename(src))
abort("#{dest} already exists") if File.exists?(dest)
File.rename(src, dest)
puts "#{File.basename src} to ."

The second is sl: it just like the command "ln -s SRC DEST", but you don't have to remember the order of the parameters. The file that exists is the source and the other, the destination.

#! /usr/bin/env ruby
require 'fileutils'
abort("Usage: sl SRC DEST") unless ARGV.length == 2
a, b = ARGV
if File.exists?(a) && File.exists?(b)
  abort("Both #{a} & #{b} exists")
elsif File.exists?(a)
  src, dest = a, b
elsif File.exists?(b)
  src, dest = b, a
else
  abort("Neither #{a} nor #{b} exists")
end
src = File.absolute_path src
abort("#{src} not a file") unless File.file?(src)
dest = File.absolute_path dest
system("ln -s #{src} #{dest}")

Extracting phone numbers from Outlook Msg Files

Recently, I had to extract phone numbers from a bunch of Outlook .MSG files (see here and here).

Looking for an open source solution, I came across MsgParser. It almost did what I needed. (I just had to modify a "protected" field and make it "public" -- I needed to access it from a driver program; there might be a easier way to do this -- maybe make the driver program "run" from the same package as MsgParser and it'll have access to the protected field...) But downloading its dependencies was a bit of a pain.

So here is a Ruby script that does all that for you. You specify a directory where the Outlook .MSG files are. It'll extract all the phone numbers from them and print it. Tested under OS X Leopard; needs curl & java to be installed. Run: ruby msg_extractor.rb MSGS_DIR.

# Released under the GNU GPL.
# http://www.gnu.org/copyleft/gpl.html

require 'fileutils'

DOWNLOADS = %w{
  http://auxilii.com/msgparser/download/msgparser-1.6.zip
  http://www.freeutils.net/source/jtnef/jtnef-1_6_0.zip
  http://apache.ziply.com/poi/release/bin/poi-bin-3.6-20091214.tar.gz
  http://www.trieuvan.com/apache/ant/binaries/apache-ant-1.8.0-bin.zip }

JCLASS = 'MyMsgParser'
JAVA_INPUT = 'My.msg'
JAVA = <<END
  import java.util.*;
  import com.auxilii.msgparser.*;
  import com.auxilii.msgparser.attachment.*;
  public class #{JCLASS} {
    public static void main(String[] args) throws Exception{
      MsgParser msgp = new MsgParser();
      Message msg = msgp.parseMsg("#{JAVA_INPUT}");
      // UNUSED
      String fromEmail = msg.getFromEmail();
      String fromName = msg.getFromName();
      String subject = msg.getSubject();
      String body = msg.getBodyText();
      String longString = msg.toLongString();
      // We modify source of msgparser to make
      // properties public
      Iterator it = msg.properties.keySet().iterator();
      while (it.hasNext()) {
       String key = (String) it.next();
       Object val = msg.properties.get(key);
        System.out.println(key + ": " + val);
      }
    }
  }
END

class MsgExtractor
  def initialize(msgs_dir)
    @msgs_dir = msgs_dir
    @cp = '.:msgparser-1.6/dist/msgparser-1.6.jar:' + 
          'poi-3.6/poi-3.6-20091214.jar:lib/tnef.jar'
    download
    javac
  end
  
  def download
    DOWNLOADS.each do |u|
      b = File.basename(u)
      if !File.exists?(b)
        puts "downloading #{u}"
        `curl --silent -O #{u}` unless File.exists?(b)
      end
      next if File.exists?("#{b}.zipped")
      puts "unpacking #{b}"
      case b
      when /zip$/; `unzip -o #{b}`
      when /gz$/;  `gzip -dc #{b} | tar -xpvf -`
      end
      FileUtils.touch("#{b}.zipped")
    end
  end
  
  def javac
    modify_msgparser
    @cp.split(':').each do |f|
      abort("#{f} doesn't exist") unless File.exists?(f) 
    end
    open("#{JCLASS}.java", "w") { |f| f.write(JAVA) }
    `javac -classpath #{@cp} #{JCLASS}.java`
  end
  
  def modify_msgparser
    jp = 'msgparser-1.6/src/main/java/com/auxilii/msgparser/Message.java'
    old1 = 'protected Map<String,Object> properties ' + 
           '= new HashMap<String,Object>();'
    new1 = old1.dup
    new1['protected'] = 'public'
    modified = File.read(jp)
    return if modified[new1]
    modified[old1] = new1
    puts "modifying #{jp}..."
    open(jp, 'w') { |f| f.write(modified) }
    FileUtils.cd('msgparser-1.6') do
      puts "rebuilding msgparser"
      `../apache-ant-1.8.0/bin/ant jar`
      FileUtils.mv 'dist/msgparser.jar', 'dist/msgparser-1.6.jar'
    end
  end
  
  def run(&extract_by)
    result = []
    Dir[File.join(@msgs_dir, '*.msg')].each do |f|
      b = File.basename(f)[0..-5]
      puts "extracting #{b}"
      FileUtils.cp(f, JAVA_INPUT)
      out = `java -classpath #{@cp} #{JCLASS}`
      data = extract_by.call(out)
      next if data.nil?
      break if $INTERRUPTED
      result << [b, data]
    end
    result.sort
  end
end

$INTERRUPTED = false

def main
  unless ARGV[0]
    abort("Usage: ruby msgs_extractor.rb msgs_dir")
  end
  tmp = ARGV[1] || "msgs_extractor"
  Dir.mkdir tmp rescue nil
  msgs_dir = File.expand_path ARGV[0]
  Dir.chdir tmp
  extractor = MsgExtractor.new(msgs_dir)
  trap("INT") { $INTERRUPTED = true; puts; }
  result = extractor.run { |o| o.scan /\d{7,20}/ } # extract phone numbers
  result.each { |n, ph| puts("%20s: %s" % [n, ph.join(", ")]) }
end

if $0 == __FILE__
  main
end

Ruby Object Model

These are my notes from Dave Thomas's Ruby Object Model.

# Ruby Object Model
# http://www.engineyard.com/blog/community/scotland-on-rails/page-2/

# everything is an object
# object is a class
# class is an object

# self:
#  - current object
#  - where instance vars are found
#  - default receiver for methods calls

# changes in self changes ruby's view of the universe
# by: methods calls & class/module definitions

animal = "cat"
puts animal.upcase

# find animal object -> find table of methods -> call method
# class => object that has a table of methods
# every single method call works exactly the same way

puts animal.object_id
# if can't methods in 1st table => goes to parent 
# (superclass of the object)

# Object toplevel in Ruby 1.8, BasicObject in Ruby 1.9
# If no matching methods found in ruby, you are back
# to original object and look for method_missing method.

def animal.speak
  puts "meaow"
end

animal.speak

# --- SELF changes on method call

# Method lookup -> one right and one up
# So the above object instance method sits 
# one to the right of that particular object.
# So a newly created class just for the animal
# object is anonymous. But when asked what class
# the above object belongs to, it says string,
# and not the newly created anonymous class.
# AKA, singleton class or metaclasss or eigenclass
# How is the animal.speak method defined?
# (1) Ruby sets "self" to the cat string object -- self
#     set to the receiver
# (2) Looks for the method in the method table associated
#     with that object (look in self's class for the object)
# (3) If not found, scan for method_missing
# (4) Eventually find some method & invoke it & when done,
#     restores original value of self.
#
# If a method call in Ruby doesn't have an explicit
# receiver -- then Ruby doesn't reset self -- self
# stays the same & no need to pop back at the end.

# So the method "puts" is a private method in Object (actually
# in Kernel), so you can only call it with an implicit receiver,
# not an explicit one.

class String
  puts "in string"
end

begin
  "hi".puts
rescue NoMethodError
  puts "puts is private"
end

# In Ruby class definitions are executable code.
puts "===class defn"
puts "toplevel #{self}"
class << self
  puts "class << self's #{self}"
end
class A
  puts "class A's #{self}"
end
module B
  puts "module B's #{self}"
end

# When you define a class like "class A; end",
# Ruby creates a brand new class and assigns that
# to the constant A.

# When excuting a class or module definition, self
# is set to the class or module object. Why? You can
# do "def self.a" for class methods (or old school
# ruby programmers using "def A.a"); But ruby doesn't
# have class methods (or static methods), all methods
# are the same. These methods are just defined on the class.
# It inserts a singleton class on that particular class object.

# To do metaprogramming in Ruby:
# - Instance variables are looked up in self
# - Methods are looked up in self's class

# these 2 are the same:
def animal.do
  puts "doing"
end
# "class << animal" means open up the singleton class of this object
# self is set to that singleton class
class << animal
  puts "class << animal: #{self}"
  def do
    puts "doing"
  end
end

class N
  # opening up the meta/singleton class and dumping all the
  # methods defined there
  class << self
    # class methods
  end
end

# This doesn't work
class F
  @a = 10 # self is class object
  def f
    @a # self is the instance 
  end
  def F.f # this work
    @a
  end
  
  class << self # this also works
    attr_accessor :f
  end
  
end

class Y; end
class Z < Y; end
# the Y in "Z < Y" is an expression
wxyz = Y
class Z < wxyz; end # same as above

a = Struct.new(:a, :b, :c) # creates a class object on the fly
puts a.new(1,2,3) 
# Struct is just a data holder

class T < Struct.new(:a, :b)
end

# include Module in a class, then invoke the instance methods
# of the module with class as the receiver
# It doesn't add the methods; any instance methods define
# in class overrides the module methods, even if it is included
# after the method definition.

module M
  def speak; puts :no; end
end

class AM
  def speak; puts :ok;  end
  include M
end

am = AM.new.speak

# include => singleton as immediate parent of my class
# and its methods are methods of my module

# hierarchy => singleton, class, module

visualize.rb

Here is a small script (visualize.rb) that helps me visualize the Structure of Ruby code. Before I jump into the source, I want to see the "big" picture of things. This script helps me with that. [There is a small bug with the script -- see the source for more details.]

Here it is being run on itself:

module Visualize
  class Construct
    def initialize
    def self.fromtree
    def to_a
    def same
    def merge
    def pp
  class SourceFile
    def initialize
    def collect_requires
    def collect_tokens
    class StatementModifier
      def initialize
    def mark_statement_modifiers
    def getname
    def gather
    def pp
    def merge
  class SourcePackage
    def initialize
    def pp
def opt?
def print_tokens
def main

And here is the code structure of Capistrano:

module Capistrano
  class ServerDefinition
    def initialize
    def <=>
    def eql?
    def hash
    def to_s
  class Command
    def self.process
    def initialize
    def process!
    def stop!
    def logger
    def open_channels
    def replace_placeholders
    def extract_environment
...
...
   class RemoteDependency
      def initialize
      def directory
      def writable
      def command
      def gem
      def or
      def pass?
      def message
      def try
def depend

My first Ruby program

I decided to learn Ruby and Rails — now I need to get books on both (damn that DHH and his contagious enthusiasm)

Below are some snippets from my first Ruby program: convert.rb. It is a CGI program, that converts PostScript, TeX, CWEB and a couple of other formats to PDF. It also converts .tar.gz, .tar.bz2 and other compressed files to .zip.

[Note: I wrote this program flipping back and forth between the PixAxe1 and a bunch of other online resources — so forgive me if it isn't in the spirit of Ruby.]

Ruby blocks are cool!!!

def ignore_errors &block
  begin
    block.call
  rescue Exception => e
    e
  end
end

def with_tmpdir &block
  result = nil
  original = Dir.pwd
  tmp = tmpname # creates a temporary name (not safe)
  begin
    Dir.mkdir tmp
    Dir.chdir tmp
    result = block.call original
  ensure
    Dir.chdir original
  end
  FileUtils.rm_rf tmp
  result
end

and it is quiet easy to create DSLs:

# converters operate like so:
# 3 args: first, a symbol naming the converter;
# second, a regexp that if matched the third arg
# is executed: which is a block that takes one
# argument -- a local filename
# the converter should return true or nil:
# true to continue and try other converters;
# nil to return -- as this is the final one.

defconverter(:ps, /\.ps$/) { | x |
  sh "ps2pdf #{x}"
  nil
}

defconverter(:cweb, /\.w$/) { | x |
  sh "cweave #{x} - #{x}.tex"
  sh "tex #{x}.tex"
  sh "dvips #{x}.dvi"
  sh "ps2pdf #{x}.ps"
  nil
}

Finally, CGI.rb's HTML output facility make the code elegant (I wish that, optionally, you could output to a stream...)

  cgi.out {
    cgi.html("PRETTY" => " ") {
      cgi.head { cgi.title { 'convert.rb' } } +
      cgi.body {
        cgi.h3 { 'convert.rb — convert file to PDF or ZIP' } +
        cgi.form('get', $0) {
        cgi.b { 'url:' } +
        cgi.text_field('url', '', 150) + ' ' +
        cgi.submit + cgi.br +
        cgi.checkbox('zipitup') + ' zip it up? ' +
        cgi.small { '[with this turned on, you can convert
                     unix archive formats (like .tar.gz, .tar.bz2,
                     ...) to .zip format]' }

      ...
      ...
      ...
      [snipped]

Here are some initial thoughts:

• I was surprised at how easily and quickly I was able to develop the program
• calling methods (those that don't take args and those that take a single arg) without parenthesis is quiet refreshing:
  (Before this, looking at Ruby from the "outside", this convention was disturbing — "How do I pass a function to another function" — but I didn't encounter that problem with this program because it seems to me that you pass in closures (blocks) to other functions...)
  
  compare method.to_a.to_s to method.to_a().to_s()
  sh "rpm2cpio #{x} | cpio -dimv --no-absolute-filenames"
• love the short, nicely named methods: FileUtils.rm_rf tmp
• ability to specify integer as 100_000_000 is nice