viernes, 29 de abril de 2011

Released Shikashi v0.4.0 (and dependencies)

Shikashi is an sandbox for ruby that handles all ruby method calls executed in the interpreter to allow or deny these calls depending on the receiver object, the method name, the source file from where the call was originated

For more info about the project, visit the project page

You can install the gem by doing


gem install shikashi


New Enhancements

removed evalmimic dependency

evalmimic is a gem to emulate the behavior of the binding argument of the eval method (the default binding), to allow the API to do this:

a = 5
Sandbox.run("a+1", Privileges.allow_method(:+)) # return 6


The implementation of evalmimic is somewhat complex because it relies in a C extension and even implements some low level hacks using unsupported features of MRI (e.g. evalmimic will not compile and install for ruby 1.9)

So it was decided to remove the evalmimic dependency from shikashi and evalhook, and remove the feature shown in the above example. The only difference now is that you must add the binding as parameter if you decide to execute the sandbox in that way.

a = 5
Sandbox.run("a+1", Privileges.allow_method(:+), binding) # return 6


And if you do not specify the binding, the default behavior is use the global binding nested in the sandbox namespace


Sugar Syntax

As seen in previous code examples, it's no longer necessary to instanciate lot of objects in order to execute code in the sandbox, only Sandbox.run and Privileges now use method chaining syntax. Example:

require "shikashi"

Sandbox.run('print "hello world\n"', Privileges.allow_method(:print))

$a = 1
Sandbox.run('print $a, "\n"',
Privileges.allow_method(:print).allow_global_read(:$a)
)



Control over read access of constants and global variables

Now, you must grant read privileges over global variables and constants in order to allow the read access to them. By default, trying to access to global variables and constants will result on SecurityError exceptions. Constants defined inside the base namespace of the sandbox are allowed by default (e.g. classes defined in the same code)

# this will work
include Shikashi
Sandbox.run("
class X
def foo
end
end
X.new.foo
", Privileges.allow_method(:new))

$a = 4
Sandbox.run("$a", Privileges.allow_global_read(:$a)) # 4

A = 4
Sandbox.run("A", Privileges.allow_const_read("A") # 4

Sandbox.run("$a") # raise SecurityError

Sandbox.run("A") # raise SecurityError




Interception of method calls using super on evalhook


Now, call to super methods are intercepted by evalhook and rejected by shikashi when appropiate

include Shikashi
#=begin
Sandbox.run("
class X
def system(*args)
super # raise SecurityError
end
end
X.new.system('ls -l')
", Privileges.allow_method(:new))


Refactor to use Ruby2Ruby on partialruby

Partialruby is the gem that emulates ruby using ruby to allow the changes to AST needed by evalhook for interceptions. Up to version 0.1.0, partialruby implements the emulation with abstract tree processing from scratch. Now, at released version 0.2.0, partialruby relies on more mature and stable gem Ruby2Ruby which converts AST to executable ruby source code

Links


martes, 19 de abril de 2011

Released ImageRuby - a flexible ruby gem for image processing

ImageRuby is a flexible gem for image processing, it's designed to be easy to install and use. The core of ImageRuby is written in pure ruby and the installation is as simple as execute "gem install imageruby". The API of the library take advantage of sugar syntax constructions specific of ruby language such as method chaining and blocks. e.g, the next code loads an image from "input.bmp", crop the rectangle from 0,0 to 49,49 (50x50 pixels), replace the orange color to red color, replace the black color to orange color and finally saves the image as "output.bmp"

require "imageruby"

ImageRuby::Image.from_file("input.bmp")[0..49,0..49].
color_replace(Color.orange,Color.red).
color_replace(Color.black,Color.orange).
save("output.bmp", :bmp)

Current Status

The ImageRuby had his first release (version 0.1.0) the last week with imageruby-c (extension to override a few methods with C methods) and imageruby-bmp (support for save and load images on BMP format),

You can install the gem by doing

gem install imageruby


The current version o imageruby has no dependencies, and it's installation should be as simple as that, optionally, you can install imageruby-c to improve the performance of draw and color_replace methods and imageruby-bmp to have support for bmp images.
The extensions take effect automagically by installing the gem, there is no need to do something specific in the ruby script (e.g. when install imageruby-c the methods will be optimized)

Goals of the project

  • Become synonymous of image processing in the ruby world, displacing other options in the "market" but re-using the existent stuff (giving credit, obviously)
  • Highlight deficiencies in the ruby language when used for heavy processing (e.g. draw method implemented in ruby is very slow)
  • Spike workarounds for the Ruby language issues
  • Spike "soft" dependency model or plugin gems to avoid the problems of "static" dependencies

Competitors

There is a range of gems for image processing on ruby, most of these are based on C libraries and implies the need for a compiler in order to install them. Some have "low level" bugs such memory leaks but most of them have many possibilities through their API

  • RMagick: Ruby wrapper of the archi-famous ImageMagick library. Has reported problems with low level memory handling and is not recommended for services environment (such as a Rails application) but is good for scripting
  • ImageScience: More stable alternative to RMagick, written on top of FreeImage and RubyInline (for more optimal code in C), an issue when using this library is the RubyInline dependency which implies the need for GCC installed on the system, something that is simple on Linux but is so tricky in other environments such Windows and Heroku
  • Camellia: A C/C++ library with a Ruby interface apparently directed to photo processing, there is no gem and the installation should be using the classic command sequence ./configure; make; make install.
  • Devil: Wrapper of C library of the same name. It has a wide range of operations over an image including blur and equalize. (see documentation)
  • Ruby-Processing: Well documented library mainly oriented to interactive media including video features. Has nice pencil functions
Future

New image formats

One of the next big steps will be develop decoder and encoders for common image formats, the encoders and decoders can be released as separated gems (like imageruby-bmp) without need for change imageruby core. In fact, anyone can make their own encoder or decoder.

Competitors become allies

Re-use the existing development around image processing in the ruby world, divided in two big groups: Interfaces and Implementations

Reusing interfaces implies the creation of "mocks" or "wrappers" of ImageRuby providing the same API of other library (e.g. RMagick or ImageScience) to reduce the learning curve of ImageRuby API and provide a method to replace that library with ImageScience transparently for code developed on top of that (e.g. a rails application using ImageMagick and then the ImageMagick gem is replaced by ImageRuby, app will not need to be modified)

Reusing implementation is about build features on top of other image libraries, but without setting a "hard" o "static" dependency with the library, but by creating a optional extension for ImageRuby (something like a port between libraries). Example: imageruby-devil.

Could be installed so:
gem install imageruby-devil

And used like
# is not necessary to make a explicit require of imageruby-devil
require "imageruby"

image = ImageRuby::Image.from_file("input.bmp")

image.color_replace!(Color.black, Color.purple) # use a method of ImageRuby
image.devil.blur(1) # use a method of Devil library on the image

image.save("output.bmp", :bmp)

Feedback for Ruby improvements

One of the goals of ImageRuby is to Highlight deficiencies in the ruby language , methods such draw or color_replace process hundred of thousands of pixels per image which in the ruby language implies still much more unnecessary rubymorphic calls just to access each bit of pixel data of the processed images. This produces a very very slow result, the current workaround for this issue is override the "heavy" methods with the imageruby-c extension. This was achieved as a optional extension to avoid the unnecessary hard dependency of have a compiler in the system, ImageRuby without imageruby-c installed HAS the draw and color_replace method, but with the imageruby-c gem installed, is much much more fast (near to 100x more faster)

Links

ImageRuby rdoc reference: http://tario.github.com/imageruby/doc/
ImageRuby GitHub site: https://github.com/tario/imageruby

lunes, 4 de abril de 2011

Evalhook v0.3.0 released and shikashi works on Heroku

Evalhook is a vital dependency of shikashi gem, provide the feature of execute managed ruby code in the way of allow "hooks" of common events like method calls, system calls, assignment of global variables and constants, etc...

Fixed Issues

With this release, problems of compatibility of actual version of shikashi (v0.3.1 at the moment of writing this post) were solved. The most important change was refactor of the hooking to make it more portable, less prone to error and implemented in pure ruby (the C extension was removed from the gem). this implies that now shikashi works on heroku and possibly in other environments

Update/Install

To update current installation of evalhook gem, run in terminal:

gem install evalhook

NOTE: To update the gem on heroku you must change Gemfile and Gemfile.lock files, then push this files to your application git repository, setting the content of Gemfile to something like that:

source :rubygems

gem 'shikashi'
gem 'evalhook', '0.3.0'
gem 'partialruby', '0.1.0'


Fundamentals

The most important concept behind evalhook is the AST hook technique which consist in change the Abstract Syntax Tree before it is interpreted and executed (view this other post for more info)

In previous versions of evalhook, the method to implement the AST hook was change the internal structures (representations of AST) on the version 1.8 of Matz Ruby Interpreter. This implementation breaks the encapsulation of MRI and depends on the specific implementation of that ruby interpreter making a very poor portability and being prone to error (e.g. the ruby interpreter crashes in heroku)

The reimplementation of the evalhook core is based on partialruby, a "para-interpreter" which executes ruby code by emulating it using ruby. Partialruby also exports the hooking services needed by evalhook thus avoiding the need for "hacks" to implement that service in a interpreter that does not have that. This concept is much more portable and theoretically should work with any interpreter of ruby 1.8 code.

Links

Shikashi Project description: http://tario-project.blogspot.com/2011/03/shikashi-flexible-sandbox-for-ruby.html
AST Hook fundamentals: http://tario-project.blogspot.com/2011/03/how-tree-hooks-works-and-what-are.html
Evalhook on Github https://github.com/tario/evalhook
Shikashi on Github: https://github.com/tario/shikashi
Heroku; http://heroku.com

sábado, 19 de marzo de 2011

How AST hook works and what are the current implementations

AST (Abstract Syntax Tree) hook is a technique to control the behavior of certain ruby node elements like method calls, global variables, etc...

First of all, certain ruby interpreters (including MRI) has an internal representation of AST which is called "node tree", each piece of code that is read by the interpreter for execution is parsed and is represented using a node tree structure. Then. this structure is read at runtime and "executed" by the interpreter core/vm.

So, the tree hook technique implies the modification of that tree before it is read and executed by the interpreter in order to perform certain "hooks". This is done by patching the tree and inserting new node elements.

Roughly, In the normal execution flow of the interpreter, the code is parsed and translated into the AST and finally executed by the vm which is part of the interpreter core:




When AST hook operate, it change the node tree after this is built and before it is executed



For example, the call node can be intercepted by changing_


To



The last node layout is a valid node tree structure too, which emulates the call after notify the event to a handler which decides to do

Current Implementations

MRI (Matz Ruby Interpreter) Hack

The first implementation of tree patching was a hack of the MRI which implements the patching directly to the node tree structure located in the memory of the ruby interpreter process, certain nodes pointer can be obtained in a C extension using a trick like that:



VALUE hook_block(VALUE self, VALUE handler) {
process_node(ruby_frame->node->nd_recv, handler);
}

And then, the node tree can be walked to make the patching. For example, patching the call node:



void patch_call_node(NODE* node, VALUE handler) {
NODE* args1 = NEW_LIST(NEW_LIT(ID2SYM(node->nd_mid)));
NODE* args2 = NEW_LIST(NEW_LIT(handler));

node->nd_recv = NEW_CALL(node->nd_recv, method_hooked_method, args1);
node->nd_recv = NEW_CALL(node->nd_recv, method_set_hook_handler, args2);

node->nd_mid = method_call;
}

Advantages
  • Easy to implement in C using interpreter code
Disadvantages
  • MUST be implemented in C to access the interpreter internal structures
  • Poor compatibility, the implementation works using internal structures of particular ruby interpreter version (e.g. wont work in ruby 1.9)

https://github.com/tario/evalhook /blob/v0.2.0/ext/evalhook_base/evalhook_base.c


Partial Ruby

In the previous detailed implementation, the main problem of the hack was the compatibility, because the tree patching is performed by changing internal structures of the interpreter which may or not may exists in the interpreter. This was done in that way because the interpreter does not expose any services in their API which serves to modify the tree (in fact, could not be any node tree in many interpreters).
The solution, is to create another interpreter which the needed services are exposed, using resources that exists in the environment: parser, api and VM.
Basically, PartialRuby parse the input ruby source file to an AST represented with ruby structures, after that, executes the ruby AST by emulating it using ruby and finally, pass the emulation ruby code to the real interpreter.






In this scenario, partial ruby expose in their API the services needed to perform the node tree patching

Advantages
  • Can be implemented in pure ruby
  • No access to ruby interpreter internals needed: compatibility granted
Disadvantages
  • Must re-implement a part of the Ruby VM

https://github.com/tario/partialruby

jueves, 17 de marzo de 2011

Shikashi - A flexible sandbox for ruby

Shikashi is an sandbox for ruby that handles all ruby method calls executed in the interpreter to allow or deny these calls depending on the receiver object, the method name, the source file from where the call was originated

Goals of the project

Provide a sandbox API to build up scripting services with granular control of privileges even at method and global variable levels

The API

The API of Shikashi expose two main services: a "eval" method and a privilege representation. The "eval" just run code in the sandbox and apply the restrictions specified in the privileges passed as parameter. Example


require "shikashi"

sandbox = Shikashi::Sandbox.new
privileges = Shikashi::Privileges.new

privileges.allow_method :"+" # mandatory to prevent SecurityError exception

result = sandbox.run "2+2", privileges
print result,"\n" # 4



Also, the API allows more effective control of the privileges by objects/method names and is not limited to methods, but also allows to control the access to global variables and constants


# ...
privileges.allow_global:$a # mandatory to prevent SecurityError exception

sandbox.run("$a = 4", privileges)

print $a, "\n"
# ...



Current Status

Currently, there are a stable version of the gem (0.3.1), but the compatibility only is guaranteed for very few environments, exluding many of the interesting like Heroku which I have verified that the gem did not work

Future

The next improvement for shikashi (for version 0.4.0) will be of course the compatibility, defining two specific objetives:
  • Making it work in Ruby 1.9
  • Making it work in Heroku (without UI or any web programming)
In addition, secondary objectives about API usability were defined for the next release, these include sugar syntax to make it easier to use like so


Sandbox.run "2+2", Privileges.allow_method("+")


Links:
https://github.com/tario/shikashi
http://tario.github.com/shikashi/doc/

domingo, 13 de marzo de 2011

Hello, and Welcome to the blog

Hello, welcome to Tario's Project blog. A site dedicated to announcements and informations for projects mainly developed in ruby gems. A site where you will find the last information and all the future plans that are coming.

Blog Content

This blog will not follow a specific pattern, but most post will be of one (or combination) of the following type(s):

  • Releases of the projects detailing new features, bugs fixed and other improvements
  • Announcements about future release dates with their respective details (features, bug fixes, etc...)
  • Creation of new projects with their details including: goals, dependencies, connection with other projects, repository, ideas, etc...
  • Re-factor or re-engineering and how this modifications will impact in new improvements in the near future (e.g. compatibility), and it's explanation
  • Technical information that affects everyone in many or all the projects (dependency maps, tooling, technical knowledge, development methodologies, etc...)
  • Announcements about project management: registration of new commiters, merging of external contributions, merging of forks of the projects, contributions to other open source projects
  • Retrospectives about the development practice and the blog itself as a communication tool part of the development process

Current Projects

About the current projects, these are mainly ruby gems and are hosted in github (http://github.com/tario). Most of the projects use ruby as principal language, but include C when applicable and necessary (e.g. evalhook). There are at least twenty projects, some with more activity than other, some dead waiting to resurrect or passing to a better live in the project heaven, and some very live. The mainly live projects are two:

  • Shikashi. a ruby sandbox designed for "unprivileged" code execution by restrict the allowed ruby opeations (method calls, global variable access, constant access, etc...)
  • ImageRuby. A flexible pure ruby library for image handling that potentially accepts plugins to add new image processing operations, new image formats and even C implementations of the processing operations to improve performance
The rest of the live projects, are dependencies of these main projects, PoC projects that depends on these or PoC of ideas for future projects (mainly about language engineering)

For example, rallhook (and dependencies), evalhook, evalmimic, getsource and partialruby are related with the sandbox and nogaku is a PoC for ruby VM in pure ruby

In the next, there will be a detailed post for each project, starting of course with the main projects and following others

Links:
https://github.com/tario