Fastruby v0.0.6 (PoC) released

Fastruby is a gem which allows to execute ruby code faster than normal (about 20X of the MRI1.8), see this post for more info

Improved ruby support

In the first gem released version of fastruby (v0.0.1), the ruby support was very limited since that version was designed as a spike (e.g. it can't handle block calls, exceptions, break, etc...).
These ruby support improvements are based on frame structures used at runtime and some other tricks such non-local goto's to implement return, next, break; this make the generated code a bit more expensive but still 20X faster (instead of 100X of the first version)

Now (version 0.0.6) fastruby support a wide range of ruby constructions, such:

• Exceptions (begin, rescue, ensure, end)
• Classes and modules
• Constants, globals
• Flow control sentences if, unless, and case..when..end
• Literals
• Singleton methods (almost as slow as normal ruby)
• Ruby built-in goto's (break, return, next)
  

But for now, leaves out the following:

• Class variables (*)
  
• Methods with multiple number of arguments (*)
• Callcc (*)

* Issue created for task

Native object cache to reduce bootstrap overhead

Usually the execution of ruby code on fastruby implies parsing the code, translate to C, and build by compiling it using gcc .
Even small code snippets may take a few seconds, a lot of time comparing it with the execution of same code using MRI, which takes hundredths of a second and imagining what would happen with larger code...
The response to this issue is the implementation of the cache, transparent to the invoker. In the image, a script is executed by first time to show it takes 0,642 seconds, and when the same script is executed again, the same script takes 0,057 seconds. The script test.rb is a very simple test:

require "fastruby"

fastruby '

print "hello world\n"

'

The cache is located on $HOME/.fastruby and the cache feature can be deactivated by setting the environment variable FASTRUBY_NO_CACHE to 1 when execute a ruby script that uses fastruby.

Implementation details of cache

As the diagram shows (click to view the full resolution version)

Each code snippet has a SHA1 sum associated and each SHA1 has a collection of native libraries including both the main object and multiple built of the methods defined in that snippet.
The diagram is only a conceptual model and does not represent any entity in fastruby source code, the sha1 association is implemented using the filesystem by saving each object collection in a directory named as the hexadecimal sha1 of the corresponding code snippet (inspired by git internals :D )




Links

Fastruby github page: https://github.com/tario/fastruby
Previous post on fastruby: http://tario-project.blogspot.com/2011/07/fastruby-v001-poc-released.html

Fastruby v0.0.1 (PoC) released

Fastruby is a gem which allows to execute ruby code faster than normal (about 100X of the MRI1.8)

Fastruby IS NOT a separated ruby interpreter. Then, the design is simple

Fastruby IS NOT a DSL to generate C code using ruby or a Ruby to C translator, the goal of fastruby is to execute RUBY code

Core Concepts

Native build

All code processed by fastruby ends with a native representation, in the current version, this is acomplished using RubyParser to parse rubycode.
The ruby code is translated to C and then processed with RubyInline

Transparent multimethods (and multiblocks)

The methods processed by fastruby has multiple internal implementations depending on the type of the arguments.
Each possible signature has a version of the method and this are built in runtime when the method is called with a new signature.
The same concept will be applied to blocks (anonymous methods) in future releases

Type inference

Each version of a method is built for a specific signature, so, the builder can assume a type for the arguments and build method calls using that assumption.
Whereever the translator can asume a type for a expression involved in a method call (used as argument or as receiver), this information can be used to encode
direct calls instead of normal and expensive ruby calls.

The currently implementation only can infer types for method and block arguments, and for literals

Customization through build directives and API

To compensate for the described limitations, fastruby suport a few build directives to allow the programmer help the inference.
The syntaxis of these directives are the same as normal ruby call (see examples)
Also, fastruby will define a API to customize aspects of fastruby internals. E.g the build method to invoke the build of methods with a specific signature (see examples)

The Image

The image of this post shows a display with the execution of the current benchmarks of the test

All of them follow the pattern of executing the code with normal ruby and with fastruby. The fourth benchmark adds a few additional measures (see the benchmark source for more info)

You can locate the benchmark sources installed in the gem directory of fastruby or in the github repository and the full resolution version of the image here

Installation

The install is as simple as execute the well-known gem install:

sudo gem install fastruby

Examples

The syntax is simple since one of the main of goals of fastruby is to be transparent. The current API serves for testing and customization of this poc version of fastruby

Prebuild of methods:

 require "fastruby"

class X
fastruby '
def foo(a,b)
  a+b
end
'
end

X.build([X,String,String] , :foo)

p X.new.foo("fast", "ruby") # will use the prebuilded method
p X.new.foo(["fast"], ["ruby"]) # will build foo for X,Array,Array signature and then execute it

NOTE: this is not necessary to do this, the methods will built automatically when there are called for first time

Variable "types"

Like static languages, you can define a type for a variable to help the inference and gain some performance. This can be done by using lvar_type directive

 class X
  fastruby '
  def foo
    lvar_type(i, Fixnum)
    i = 100
    while (i > 0)
      i = i - 1
    end
    nil
  end
  '
end

With no lvar_type, the calls to Fixnum#> and Fixnum#- will be dynamic and more expensive

Links

• Github page: https://github.com/tario/fastruby
  
  

Shikashi v0.5.0 released

Shikashi is a sandbox for ruby that handles all ruby method calls executed in the interpreter to allow or deny these calls depending on the receiver object, the method name, the source file from where the call was originated

For more info about the project, visit the project page

You can install the gem by doing

gem install shikashi

New Enhancements

The primary focus of this latest release was improve the performance of execution of ruby code in the sandbox but maintaining the purity of ruby (i.e. without implementing anything with C or similar)

Code packets

In the previous version of shikashi (0.4.0), to execute code in the sandbox, you do:

Shikashi::Sandbox.run("1+1", Shikashi::Privileges.allow_method(:+))

This, internally, implies:

• Parse the code
• Process the syntax tree (evalhook)
  
• Emulate the transformed tree (partialruby)
  
• Execute the emulation code using eval...
  

This was a great performance problem in cases where you need to execute the same piece of ruby code many times in the sandbox (e.g. sandboxed web code executed for each incoming request)
To solve this, the concept of "code packet" was implemented; a "code packet" is the ultimate product of the above mentioned processing chain ready to be executed as many times as necessary without reprocess the code and tree. Example:

packet = Shikashi::Sandbox.packet(code)
# after that, you can run the "packet" as many times as you want without doing all
# parsing, tree processing and emulation stuff internally
packet.run(privileges)
packet.run(privileges)
packet.run(privileges)
packet.run(privileges)

The performance difference is of the order of 20X, you can check by doing the following benchmark:

require "rubygems"
require "shikashi"
require "benchmark"

code = "class X
 def foo(n)
 end
end
X.new.foo(1000)
"

s = Shikashi::Sandbox.new

Benchmark.bm(7) do |x|

x.report("normal") {
1000.times do
s.run(code, Shikashi::Privileges.allow_method(:new))
end
}

x.report("packet") {
packet = s.packet(code, Shikashi::Privileges.allow_method(:new))
1000.times do
packet.run
end
}

end

In my laptop, the result of this benchmark is:

            user     system      total        real
normal   5.870000   0.540000   6.410000 (  6.436331)
packet   0.290000   0.020000   0.310000 (  0.315351)

The repeated execution of code using packets is 20 times faster than normal

Refactor of evalhook internals

Evalhook is the support gem that provides the ability to hook events on the execution of ruby code. Evalhook implemenations initially was not desgined for performance, then it's possible to refactor the implementation to increase the performance by supressing unnecessary calls and other optimizations.

require "rubygems"
require "shikashi"
require "benchmark"

s = Shikashi::Sandbox.new

class NilClass
def foo
end
end

Benchmark.bm(7) do |x|

x.report {

code = "
 500000.times {
 nil.foo
 }
 "
s.run code, Shikashi::Privileges.allow_method(:times).allow_method(:foo)
}

end

The result without the optimizations:

            user     system      total        real
      36.140000   1.420000  37.560000 ( 37.672708)

The result with the optimizations:

            user     system      total        real
      27.460000   1.130000  28.590000 ( 28.658031)

The difference is not so significant because most of the time is used by privileges checking in both cases. Similar bechmarks executed against evalhook directly throws greater differences

Links

• Shikashi github page
• Shikashi RDoc
  
• Evalhook github page

Released ImageRuby-Devil - a bridge between ImageRuby and Devil image library

ImageRuby-devil is the bridge between ImageRuby and Devil image library, it adds a group of new features to ImageRuby which are based on devil features:

• Load and save image formats supported by devil library including jpg, tga, png, etc...
• New image operations including "alienify", "blur" and "contrast"

The development, release and usage of this gem was predicted in the first ImageRuby release announcement

Install the gem

In the command line, execute:

gem install imageruby-devil

Example

# is not necessary to make a explicit require of imageruby-devil
require "imageruby"

image = ImageRuby::Image.from_file("input.jpg") # now, jpg files can be loaded

image.color_replace!(Color.black, Color.purple) # use a method of ImageRuby
image.blur(1) # use a method of added by imageruby-devil

image.save("output.png", :png) # save (and load) in png format is now supported

Links:

ImageRuby-devil at github: https://github.com/tario/imageruby-devil
ImageRuby at github: https://github.com/tario/imageruby
ImageRuby-bmp-c: https://github.com/tario/imageruby-bmp-c
Devil image library official site: http://openil.sourceforge.net/

Released Shikashi v0.4.0 (and dependencies)

Shikashi is an sandbox for ruby that handles all ruby method calls executed in the interpreter to allow or deny these calls depending on the receiver object, the method name, the source file from where the call was originated

For more info about the project, visit the project page

You can install the gem by doing

gem install shikashi 

New Enhancements

removed evalmimic dependency

evalmimic is a gem to emulate the behavior of the binding argument of the eval method (the default binding), to allow the API to do this:

a = 5
Sandbox.run("a+1", Privileges.allow_method(:+)) # return 6

The implementation of evalmimic is somewhat complex because it relies in a C extension and even implements some low level hacks using unsupported features of MRI (e.g. evalmimic will not compile and install for ruby 1.9)

So it was decided to remove the evalmimic dependency from shikashi and evalhook, and remove the feature shown in the above example. The only difference now is that you must add the binding as parameter if you decide to execute the sandbox in that way.

a = 5
Sandbox.run("a+1", Privileges.allow_method(:+), binding) # return 6

And if you do not specify the binding, the default behavior is use the global binding nested in the sandbox namespace


Sugar Syntax

As seen in previous code examples, it's no longer necessary to instanciate lot of objects in order to execute code in the sandbox, only Sandbox.run and Privileges now use method chaining syntax. Example:

require "shikashi"

Sandbox.run('print "hello world\n"', Privileges.allow_method(:print))

$a = 1
Sandbox.run('print $a, "\n"',
Privileges.allow_method(:print).allow_global_read(:$a)
)

Control over read access of constants and global variables

Now, you must grant read privileges over global variables and constants in order to allow the read access to them. By default, trying to access to global variables and constants will result on SecurityError exceptions. Constants defined inside the base namespace of the sandbox are allowed by default (e.g. classes defined in the same code)

# this will work
include Shikashi
Sandbox.run("
class X
  def foo
  end
end
X.new.foo
", Privileges.allow_method(:new))

$a = 4
Sandbox.run("$a", Privileges.allow_global_read(:$a)) # 4

A = 4
Sandbox.run("A", Privileges.allow_const_read("A") # 4

Sandbox.run("$a") # raise SecurityError

Sandbox.run("A") # raise SecurityError

Interception of method calls using super on evalhook


Now, call to super methods are intercepted by evalhook and rejected by shikashi when appropiate

include Shikashi
#=begin
Sandbox.run("
class X
  def system(*args)
    super # raise SecurityError
  end
 end
 X.new.system('ls -l')
", Privileges.allow_method(:new))

Refactor to use Ruby2Ruby on partialruby

Partialruby is the gem that emulates ruby using ruby to allow the changes to AST needed by evalhook for interceptions. Up to version 0.1.0, partialruby implements the emulation with abstract tree processing from scratch. Now, at released version 0.2.0, partialruby relies on more mature and stable gem Ruby2Ruby which converts AST to executable ruby source code

Links

• Shikashi github page
• Shikashi RDoc
  
• Evalhook github page
• Partialruby github page
• Ruby2Ruby github page
  

Released ImageRuby - a flexible ruby gem for image processing

ImageRuby is a flexible gem for image processing, it's designed to be easy to install and use. The core of ImageRuby is written in pure ruby and the installation is as simple as execute "gem install imageruby". The API of the library take advantage of sugar syntax constructions specific of ruby language such as method chaining and blocks. e.g, the next code loads an image from "input.bmp", crop the rectangle from 0,0 to 49,49 (50x50 pixels), replace the orange color to red color, replace the black color to orange color and finally saves the image as "output.bmp"

require "imageruby"

ImageRuby::Image.from_file("input.bmp")[0..49,0..49].
color_replace(Color.orange,Color.red).
color_replace(Color.black,Color.orange).
save("output.bmp", :bmp)

Current Status

The ImageRuby had his first release (version 0.1.0) the last week with imageruby-c (extension to override a few methods with C methods) and imageruby-bmp (support for save and load images on BMP format),

You can install the gem by doing

gem install imageruby

The current version o imageruby has no dependencies, and it's installation should be as simple as that, optionally, you can install imageruby-c to improve the performance of draw and color_replace methods and imageruby-bmp to have support for bmp images.
The extensions take effect automagically by installing the gem, there is no need to do something specific in the ruby script (e.g. when install imageruby-c the methods will be optimized)

Goals of the project

• Become synonymous of image processing in the ruby world, displacing other options in the "market" but re-using the existent stuff (giving credit, obviously)
• Highlight deficiencies in the ruby language when used for heavy processing (e.g. draw method implemented in ruby is very slow)
• Spike workarounds for the Ruby language issues
• Spike "soft" dependency model or plugin gems to avoid the problems of "static" dependencies
  

Competitors

There is a range of gems for image processing on ruby, most of these are based on C libraries and implies the need for a compiler in order to install them. Some have "low level" bugs such memory leaks but most of them have many possibilities through their API

• RMagick: Ruby wrapper of the archi-famous ImageMagick library. Has reported problems with low level memory handling and is not recommended for services environment (such as a Rails application) but is good for scripting
  
• ImageScience: More stable alternative to RMagick, written on top of FreeImage and RubyInline (for more optimal code in C), an issue when using this library is the RubyInline dependency which implies the need for GCC installed on the system, something that is simple on Linux but is so tricky in other environments such Windows and Heroku
  
• Camellia: A C/C++ library with a Ruby interface apparently directed to photo processing, there is no gem and the installation should be using the classic command sequence ./configure; make; make install.
• Devil: Wrapper of C library of the same name. It has a wide range of operations over an image including blur and equalize. (see documentation)
  
• Ruby-Processing: Well documented library mainly oriented to interactive media including video features. Has nice pencil functions
  

Future

New image formats

One of the next big steps will be develop decoder and encoders for common image formats, the encoders and decoders can be released as separated gems (like imageruby-bmp) without need for change imageruby core. In fact, anyone can make their own encoder or decoder.

Competitors become allies

Re-use the existing development around image processing in the ruby world, divided in two big groups: Interfaces and Implementations

Reusing interfaces implies the creation of "mocks" or "wrappers" of ImageRuby providing the same API of other library (e.g. RMagick or ImageScience) to reduce the learning curve of ImageRuby API and provide a method to replace that library with ImageScience transparently for code developed on top of that (e.g. a rails application using ImageMagick and then the ImageMagick gem is replaced by ImageRuby, app will not need to be modified)

Reusing implementation is about build features on top of other image libraries, but without setting a "hard" o "static" dependency with the library, but by creating a optional extension for ImageRuby (something like a port between libraries). Example: imageruby-devil.

Could be installed so:

gem install imageruby-devil

And used like

# is not necessary to make a explicit require of imageruby-devil
require "imageruby"

image = ImageRuby::Image.from_file("input.bmp")

image.color_replace!(Color.black, Color.purple) # use a method of ImageRuby
image.devil.blur(1) # use a method of Devil library on the image

image.save("output.bmp", :bmp)

Feedback for Ruby improvements

One of the goals of ImageRuby is to Highlight deficiencies in the ruby language , methods such draw or color_replace process hundred of thousands of pixels per image which in the ruby language implies still much more unnecessary rubymorphic calls just to access each bit of pixel data of the processed images. This produces a very very slow result, the current workaround for this issue is override the "heavy" methods with the imageruby-c extension. This was achieved as a optional extension to avoid the unnecessary hard dependency of have a compiler in the system, ImageRuby without imageruby-c installed HAS the draw and color_replace method, but with the imageruby-c gem installed, is much much more fast (near to 100x more faster)

Links

ImageRuby rdoc reference: http://tario.github.com/imageruby/doc/
ImageRuby GitHub site: https://github.com/tario/imageruby

Evalhook v0.3.0 released and shikashi works on Heroku

Evalhook is a vital dependency of shikashi gem, provide the feature of execute managed ruby code in the way of allow "hooks" of common events like method calls, system calls, assignment of global variables and constants, etc...

Fixed Issues

With this release, problems of compatibility of actual version of shikashi (v0.3.1 at the moment of writing this post) were solved. The most important change was refactor of the hooking to make it more portable, less prone to error and implemented in pure ruby (the C extension was removed from the gem). this implies that now shikashi works on heroku and possibly in other environments

Update/Install

To update current installation of evalhook gem, run in terminal:

gem install evalhook

NOTE: To update the gem on heroku you must change Gemfile and Gemfile.lock files, then push this files to your application git repository, setting the content of Gemfile to something like that:

source :rubygems

gem 'shikashi'
gem 'evalhook', '0.3.0'
gem 'partialruby', '0.1.0'

Fundamentals

The most important concept behind evalhook is the AST hook technique which consist in change the Abstract Syntax Tree before it is interpreted and executed (view this other post for more info)

In previous versions of evalhook, the method to implement the AST hook was change the internal structures (representations of AST) on the version 1.8 of Matz Ruby Interpreter. This implementation breaks the encapsulation of MRI and depends on the specific implementation of that ruby interpreter making a very poor portability and being prone to error (e.g. the ruby interpreter crashes in heroku)

The reimplementation of the evalhook core is based on partialruby, a "para-interpreter" which executes ruby code by emulating it using ruby. Partialruby also exports the hooking services needed by evalhook thus avoiding the need for "hacks" to implement that service in a interpreter that does not have that. This concept is much more portable and theoretically should work with any interpreter of ruby 1.8 code.

Links

Shikashi Project description: http://tario-project.blogspot.com/2011/03/shikashi-flexible-sandbox-for-ruby.html
AST Hook fundamentals: http://tario-project.blogspot.com/2011/03/how-tree-hooks-works-and-what-are.html
Evalhook on Github https://github.com/tario/evalhook
Shikashi on Github: https://github.com/tario/shikashi
Heroku; http://heroku.com