Installing Ruby 1.9.3 in Windows with Redcarpet and Github Markdown

This is the third of three posts describing how to install Ruby 1.9.3 native MingW environment. It's much simpler to use Ruby Installer to do this but if you want all the details of how to do it natively, read on…

If you're wondering why I did this, I wanted to use redcarpet/github-linguist for markdown in Octopress; there's a dependency on the charlock_holmes Gem that requires native code libraries (file, ICU), some of which it needs to build. I couldn't figure out how to make the Ruby Installer do this and I wanted to understand the process, so I did the whole thing from scratch. The first part is here and the second part is here.

Setup ruby. In the build location (/build32 if following the instructions)…

First, install Python 2.7, if you havn't already - this is required for pygments.rb. Make sure Python is in your path. Check in a Mingw window:

python --version

We need some additional libraries for the Ruby Gem charlock_holmes. I've built them both with static linking here so that we don't need to worry about where the DLLs might have gone. <!– more –>

Download and build ICU (this post on compiling for qt and this post for compiling for OpenTTD both help):

cd /build32
wget http://download.icu-project.org/files/icu4c/4.6/icu4c-4_6-src.zip
unzip icu4c-4_6-src.zip
wget http://devs.openttd.org/~terkhen/libicu/libicu_4_6_mingw32.diff
wget http://devs.openttd.org/~terkhen/libicu/libicu_reduce_icudata_size.diff
cd icu
patch -p1 -i ../libicu_4_6_mingw32.diff
patch -p1 -i ../libicu_reduce_icudata_size.diff
cd source
./configure --prefix=/mingw --enable-static --disable-shared --disable-strict --disable-threads
#./configure --prefix=$PWD/../dist --enable-static --disable-shared --disable-strict --disable-threads
make && make install
cd ../..

Download and install libgnurx (required for file-5.08, also a dependency of charlock_holmes):

cd /build32
wget http://sourceforge.net/projects/mingw/files/Other/UserContributed/regex/mingw-regex-2.5.1/mingw-libgnurx-2.5.1-src.tar.gz
tar zxvf mingw-libgnurx-2.5.1-src.tar.gz
cd mingw-libgnurx-2.5.1

Apply the following patch for static build: (diff -up Makefile.in Makefile.in.new > ../mingw-libgnurx-static.patch)

--- Makefile.in 2007-05-07 20:28:28 +0100
+++ Makefile.in.new     2013-09-23 12:44:15 +0100
@@ -56,7 +56,7 @@ SRCDIST_FILES = ${srcdir}/configure ${sr
 ZIPCMD = @ZIPCMD@
 ZIPEXT = @ZIPEXT@

-all: libgnurx-$(DLLVERSION).dll libgnurx.dll.a libregex.a @GNURX_LIB@
+all: libgnurx.a libgnurx-$(DLLVERSION).dll libgnurx.dll.a libregex.a @GNURX_LIB@

 Makefile: config.status Makefile.in
        ./config.status
@@ -69,6 +69,10 @@ $(OBJECTS): Makefile
 libgnurx-$(DLLVERSION).dll libgnurx.dll.a: $(OBJECTS)
        $(CC) $(CFLAGS) -shared -o libgnurx-$(DLLVERSION).dll $(LDFLAGS) $(OBJECTS)

+libgnurx.a: $(OBJECTS)
+       ar rcu $@ $(OBJECTS)
+       ranlib $@
+
 libregex.a: libgnurx.dll.a
        cp -p libgnurx.dll.a $@

@@ -81,6 +85,11 @@ install-dll:
        mkdir -p ${bindir}
        cp -p $(BINDIST_FILES) ${bindir}

+install-static: libgnurx.a
+       mkdir -p ${includedir} ${libdir}
+       cp -p ${srcdir}/regex.h ${includedir}
+       cp -p ${srcdir}/libgnurx.a ${libdir}
+
 install-dev:
        mkdir -p ${includedir} ${libdir}
        cp -p ${srcdir}/regex.h ${includedir}

Build and install:

patch < ../mingw-libgnurx-static.patch
./configure -prefix=/mingw
make && make install-static

Optional: check that file-5.08 builds (note: no make install) without errors:

cd /build32
wget ftp://ftp.astron.com/pub/file/file-5.08.tar.gz
tar zxvf file-5.08.tar.gz
cd file-5.08
./configure --prefix=/mingw  --disable-shared --enable-static --with-pic
make

Download Octopress (see also Octopress setup instructions). The easiest way to do this is to first install Git for Windows. Start Git Bash, and then (note: this example uses /build32 but you can put octopress where you like…):

cd /c/mingw/build32
git clone git://github.com/imathis/octopress.git octopress

At this stage (back in MingW build environment):

cd /build32/octopress
$ gem list

*** LOCAL GEMS ***

bigdecimal (1.1.0)
io-console (0.3)
json (1.5.5)
minitest (2.5.1)
rake (0.9.2.2)
rdoc (3.9.5)

Install bundler and the default bundle:

gem install bundler

$ gem list

*** LOCAL GEMS ***

bigdecimal (1.1.0)
bundler (1.3.5)
io-console (0.3)
json (1.5.5)
minitest (2.5.1)
rake (0.9.2.2)
rdoc (3.9.5)

Then install the default bundle:

bundle install

Update the Gemfile to install pygments 0.3.7 - this is required so that we have the updated lexer Augeas:

gem 'pygments.rb', '~> 0.3.7'

and update the bundle:

bundle update pygments.rb

Now it should look like this:

$ bundle list
Gems included by the bundle:
  * RedCloth (4.2.9)
  * bundler (1.3.5)
  * chunky_png (1.2.5)
  * classifier (1.3.3)
  * compass (0.12.2)
  * directory_watcher (1.4.1)
  * fast-stemmer (1.0.1)
  * fssm (0.2.9)
  * haml (3.1.7)
  * jekyll (0.12.0)
  * kramdown (0.13.8)
  * liquid (2.3.0)
  * maruku (0.6.1)
  * posix-spawn (0.3.6)
  * pygments.rb (0.3.7)
  * rack (1.5.2)
  * rack-protection (1.5.0)
  * rake (0.9.2.2)
  * rb-fsevent (0.9.1)
  * rdiscount (2.0.7.3)
  * rubypants (0.2.0)
  * sass (3.2.9)
  * sass-globbing (1.0.0)
  * sinatra (1.4.2)
  * stringex (1.4.0)
  * syntax (1.0.0)
  * tilt (1.3.7)
  * yajl-ruby (1.1.0)

Create a patch file for pygments.rb:

--- popen.rb 2013-09-24 14:41:22 +0100
+++ popen.rb.new  2013-09-24 15:12:42 +0100
@@ -110,7 +110,7 @@
     def lexers
       begin
         lexerfile = File.expand_path('../../../lexers', __FILE_)
-        raw = File.open(lexer_file, "r").read
+        raw = File.open(lexer_file, "rb").read
         Marshal.load(raw)
       rescue Errno::ENOENT
         raise MentosError, "Error loading lexer file. Was it created and vendored?"

Now apply the patch to popen.rb in pygments.rb:

cd /local32/lib/ruby/gems/1.9.1/gems/pygments.rb-0.3.7/lib/pygments
patch -u -i /build32/pygments.rb.popen.rb.patch
cd /build32/octopress

Create a file named 'rake' in /local32/bin. This looks like the central section of rake.bat:

#!C:/mingw/local32/bin/ruby

#--
# Copyright (c) 2003, 2004, 2005, 2006, 2007  Jim Weirich
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to
# deal in the Software without restriction, including without limitation the
# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
# sell copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
# IN THE SOFTWARE.
#++

begin
  require 'rubygems'
rescue LoadError
end

require 'rake'

Rake.application.run
END

Install the default Octopress theme, create a new test page and generate the site:

rake install
rake new_post["first post"]
rake generate

You should get the site built in octopress/public. Check that index.html is generated properly (size > 0).

At this point we have the default Octopress installed using rdiscount for markdown generation and we've setup the required libraries for redcarpet. Now we need to install charlock_holmes manually:

gem install rake-compiler
mkdir /build32/tmp
cd /build32/tmp
gem fetch charlock_holmes
gem spec charlock_holmes-0.6.9.4.gem --ruby > charlock_holmes.gemspec
gem unpack charlock_holmes-0.6.9.4.gem
mv -v charlock_holmes.gemspec charlock_holmes-0.6.9.4
cd charlock_holmes-0.6.9.4

Now modify ext/charlock_holmes/extconf.rb:

--- extconf.rb   2013-09-24 14:49:14 +0100
+++ extconf.rb.new    2013-09-24 14:51:54 +0100
@@ -9,7 +9,7 @@
   ret
 end
 
-if `which make`.strip.empty?
+if `sh which make`.strip.empty?
   STDERR.puts "\n\n"
   STDERR.puts "***************************************************************************************"
   STDERR.puts "*************** make required (apt-get install make build-essential) =( ***************"
@@ -37,7 +37,8 @@
   end
 end
 
-unless have_library 'icui18n' and have_header 'unicode/ucnv.h'
+#unless have_library 'icui18n' and have_library 'icuuc' and have_library 'regex' and have_header 'unicode/ucnv.h'
+unless have_library 'icudata' and have_library 'icuuc' and have_library 'icui18n' and have_library 'gnurx' and have_header 'unicode/ucnv.h'
   STDERR.puts "\n\n"
   STDERR.puts "***************************************************************************************"
   STDERR.puts "*********** icu required (brew install icu4c or apt-get install libicu-dev) ***********"
@@ -57,7 +58,7 @@
 
   sys("tar zxvf #{src}")
   Dir.chdir(dir) do
-    sys("./configure --prefix=#{CWD}/dst/ --disable-shared --enable-static --with-pic")
+    sys("sh ./configure --prefix=#{CWD}/dst/ --disable-shared --enable-static --with-pic")
     sys("patch -p0 < ../file-soft-check.patch")
     sys("make -C src install")
     sys("make -C magic install")

cd ext/charlock_holmes
patch -u -i /build32/charlock_holmes_extconf.rb.patch
cd ../..
gem build charlock_holmes.gemspec
gem install charlock_holmes-0.6.9.4.gem --platform=ruby

We should now have charlock_holmes installed:

$ gem list

*** LOCAL GEMS ***

bigdecimal (1.1.0)
bundler (1.3.5)
charlock_holmes (0.6.9.4)
chunky_png (1.2.5)
classifier (1.3.3)
compass (0.12.2)
directory_watcher (1.4.1)
fast-stemmer (1.0.1)
fssm (0.2.9)
haml (3.1.7)
io-console (0.3)
jekyll (0.12.0)
json (1.5.5)
kramdown (0.13.8)
liquid (2.3.0)
maruku (0.6.1)
minitest (2.5.1)
posix-spawn (0.3.6)
pygments.rb (0.3.7, 0.3.4)
rack (1.5.2)
rack-protection (1.5.0)
rake (0.9.2.2)
rake-compiler (0.9.1)
rb-fsevent (0.9.1)
rdiscount (2.0.7.3)
rdoc (3.9.5)
RedCloth (4.2.9 x86-mingw32)
rubypants (0.2.0)
sass (3.2.9)
sass-globbing (1.0.0)
sinatra (1.4.2)
stringex (1.4.0)
syntax (1.0.0)
tilt (1.3.7)
yajl-ruby (1.1.0 x86-mingw32)
cd /build32/octopress

Now we can modify the Gemfile to include redcarpet:

  gem 'rdiscount', '~> 2.0.7'
  gem "charlock_holmes", "~> 0.6.9.4"
  gem "github-linguist", "~> 2.4"
  gem 'redcarpet', '~> 2.1.1'
  gem 'pygments.rb', '~> 0.3.7'

And update the Gems:

cd /build32/octopress
bundle install

Test. Save this code into a file in (for example) /tmp/test.rb:

require 'rdiscount'
require 'charlockholmes'
require 'escape_utils'
require 'pygments'
require 'yaml'
require 'linguist/classifier'
require 'linguist/samples'
require 'linguist/file_blob'
require 'redcarpet'

puts "Ruby test"

#puts Pygments.highlight(File.read(FILE_), :lexer => 'ruby')

@rdiscount_extensions = "smart"

content = "Some text"
rd = RDiscount.new(content, @rdiscount_extensions)
html = rd.to_html

puts html

detection = CharlockHolmes::EncodingDetector.detect(html)

puts detection

fb = Linguist::FileBlob.new("c:\Ruby193\devkit\dk.rb")
puts fb.language.name

markdown = Redcarpet::Markdown.new(Redcarpet::Render::HTML, :autolink => true, :space_after_headers => true)
puts markdown.render("This is *bongos c++ int main() { return 0 }, indeed.")

And run the test:

$ ruby test.rb
...
Ruby test
<p>Some <em>text</em></p>
{:type=>:text, :encoding=>"ISO-8859-2", :confidence=>40, :language=>"cs"}
Ruby
<p>This is <em>bongos</em> <code>c++ int main() { return 0 }</code>, indeed.</p>

Now modify _config.yml for octopress to generate redcarpet markdown:

markdown: redcarpet
redcarpet:
  extensions: ["no_intra_emphasis", "fenced_code_blocks", "autolink", "tables", "with_toc_data"]

Modify the post created above to include some github markdown, e.g. (note: replace last [backtick]…):

Some text

c++ int main() { return 0; } “[backtick]

Some more text.

And regenerate:

rake generate “` You should now have syntax highlighted code blocks!

Graoil Limited

Installing Ruby 1.9.3 in Windows With Redcarpet and Github Markdown