Mostrando entradas con la etiqueta release. Mostrar todas las entradas
Mostrando entradas con la etiqueta release. Mostrar todas las entradas

lunes, 23 de abril de 2012

Fastruby 0.0.21: Inline of corelib fixnum methods, restored ruby 1.8 compat

Fastruby is a gem which allows to execute ruby code much faster than normal, currently in a state of transition between a spike and a usable gem, it is released when possible with incremental improvements.

One of the main improvements on fastruby v0.0.20 v0.0.21 was the implementation of a few corelib methods (e.g. Fixnum#+) in ruby to allow inline them. The other is the restored compatibility with ruby 1.8

Install

You can clone the repository at github:
git clone git://github.com/tario/fastruby.git
git checkout v0.0.21

Or install it using gem install:
gem install fastruby


Speedup

Before (v0.0.19 and prior)







Now:








NOTE: Since issues with 1.8 compatibility, inline of corelib methods won't works on ruby 1.8, this will be fixed for v0.0.22 in future releases

Next

martes, 14 de febrero de 2012

Fastruby 0.0.19 released: improve of inline, performance bugs fixed

Fastruby is a gem which allows to execute ruby code much faster than normal, currently in a state of transition between a spike and a usable gem, it is released when possible with incremental improvements.

The main improvements on fastruby v0.0.19 are refinements on the inliner algorithm:
  • Inline of block calls (was not possible before)
  • Inline of methods with block calls (was not possible before)
And the fix of two important bugs:
  • Duplicated entries on $LOAD_PATH creates an incremental latency on require as new methods are built. Fixing this bug allows obtaining a reduction of total time of tests of %80 (150 seconds to 30 seconds on my machine). Moral: don't touch $LOAD_PATH if possible
  • Replacement of CFUNC calls using the previously implemented observer mechanism for inlining
  • Fixed case statements reduction to use deterministic names for temporal variable and allow caching of methods using case statements
No new big features...

Install

You can clone the repository at github:
git clone git://github.com/tario/fastruby.git
git checkout v0.0.19

Or install it using gem install:
gem install fastruby


domingo, 5 de febrero de 2012

Fastruby 0.0.18 released: more performance improvements

Fastruby is a gem which allows to execute ruby code much faster than normal, currently in a state of transition between a spike and a usable gem, it is released when possible with incremental improvements.

The main improvemens on fastruby v0.0.18 are the optimizations:
  • Method inlining, with the following limitations: no inline of iter calls (blocks), no inline of methods with iter calls and only one level of inlining for now
  • Refactor of cache to do not make use of the code snippet as key, instead caching is applied at other level using the syntax tree and other data as key
And bug fixes:
  • Fixed replacements of methods by redesigning the cache

Install

You can clone the repository at github:
git clone git://github.com/tario/fastruby.git
git checkout v0.0.18

Or install it using gem install:
gem install fastruby

Inlining in Ruby

Inlining is a technique very very common and popular on static compiled languages like C, C++ and even on managed static languages as Java or .NET. It basically consists on replacing a method call with their implementation to save the overhead cost of calling to a separated method.
On C programs this means avoid having to execute call instructions, allocating a de-allocating function frames, etc... In the ruby world, while the concept of inlining turns a little more complex, inlining implies saving creations of ruby frames (surely much more expensive than C frames) and lookup on method tables which have the same cost as accessing a associative hash.

Many can realize that making use of inlining technique on dynamic languages such as Javascript, Python or Ruby on this case is very complicated. If you do not realize this, look at the following example:

Suppose that you have this ruby code:
class X
def foo(a,b)
a+b
end
end

class Y
def bar(x)
i = 100000
ret = 0
while (i > 0)
ret = x.foo(ret,i)
i = i - 1
end
ret
end
end

p Y.new.bar(X.new) # 50005000

For now, avoid paying attention to the loop code on bar, so, if we apply the inline logic to X#foo, the code will be converted to:
class Y
def bar(x)
i = 100000
ret = 0
while (i > 0)
ret = (
a = ret
b = ret
a+b
) # x.foo(ret,i)
i = i - 1
end
ret
end
end

p Y.new.bar(X.new) # 50005000

But there is a problem on this picture, what happens if we call the same bar method passing a object of other type than X?:
class X2
def foo(a,b)
a*b
end
end
p Y.new.bar(X2.new) # ?
The result will not be correct, the implementation of bar method with inlined foo is only valid while the implementation of the method being called is the same used at the moment of inlining it

Transparent multimethod and inlining

From earliest released versions of fastruby the base concept of multimethods was applied, this means that for a given method will be as many implementations as possible signatures (signatures given by arguments types including type of receiver object) , by example, the method Y#bar is implemented using two functions, one for X class and the other for X2 class

When the method is implemented in that way, we can know the type of arguments in a given function and use this information to inline the correct method being called.

Will be a version of Y#bar for X:
class Y
def bar(x) # we know the type of x is X
i = 100000
ret = 0
while (i > 0)
ret = (
a = ret
b = ret
a+b
) # x.foo(ret,i) # we know the implementation of X#foo
i = i - 1
end
ret
end
end
Will be a version of Y#bar for X2:
class Y
def bar(x) # we know the type of x is X2
i = 100000
ret = 0
while (i > 0)
ret = (
a = ret
b = ret
a*b
) # x.foo(ret,i) # we know the implementation of X2#foo
i = i - 1
end
ret
end
end
And this is correct, because the dispatcher calls the version of X or X2 depending on the type of the argument. The only problem with this is that the change of implementations of inlined methods (such as X#foo), when somebody do something like that:
class X
def foo(a,b)
a + b % 5
end
end

Open Classes: Observers

The solution to the problem described in the previous paragraph is to rebuild methods with inlined calls each time the implementation of inlined methods changes. To acchive this I have implemented a observer method on fastruby method object .
Each time a method is build, observers are created for each inlined method to rebuild when any of these observers are triggered, and each time a method changes all related observers are triggered.




lunes, 2 de enero de 2012

No more excuses for do not testing: picotest 0.0.1 released

Picotest is a gem which allows to write complete test suites in a few lines, targeted primarily to test separated methods, algorithms or little snippets of code

The first gem release (0.0.1) is only a proof on concept to explore the idea which is to avoid "It's not worth testing this" syndrome. This problem occurs when we develop really small features such as helper functions, calculation private methods, etc... and then we fall in "It's not worth testing this", "because test code will be much greater than code being tested".

The solution offered by picotest, is to provide a really small testing dsl to keep the "proportion" even in smaller cases, using hash syntax and other syntaxic sugar

Examples

For example, if you want to test Math.sqrt function, you could use the following:
require "picotest"

suite(1 => 1, 4 => 2, 9 => 3, 16 => 4).test(Math.method(:sqrt))

There is no need to write all cases one by one, if you known as to calculate the input from the output (the inverse of sqrt), in that cases you could use a reverse oracle
require "picotest"

suite( lambda{|x| x**2} => _set(1,2,3,4,5,6,7,8,9,10)).test(Math.method(:sqrt))
# the following is the same as the above example
suite( lambda{|x| x**2} => _set(*(1..10)).test(Math.method(:sqrt))

And mocking is as follows:
class X
attr_accessor :another_object
def foo(data)
data.split(";")[another_object.fieldno].to_f
end
end

# you can return mock objects using mock method again
s.test X.new.method(:foo).mock(:another_object => mock(:fieldno => 2) )
You can read more examples at https://github.com/tario/picotest/tree/master/samples and in the README

Install

gem install picotest

Links

martes, 1 de noviembre de 2011

Fastruby 0.0.15 released. Callcc and continuation objects!

Fastruby is a gem which allows to execute ruby code much faster than normal, currently in a state of transition between a spike and a usable gem, it is released when possible with incremental improvements.
The v0.0.15 release adds the support for continuation objects created using callcc function. To acchieve this, the implementation uses the non-lineal stack created on previous releases.
Also this release adds a few extra language support improvements such as method default arguments

Install

You can clone the repository at github:
git clone git://github.com/tario/fastruby.git
git checkout v0.0.15
Or install it using gem install:
gem install fastruby

New Features

Fixes

Examples of code being supported

Example of continuation calls, tail recursion


require "fastruby"

fastruby '
class X
def fact( n )
a = callcc { |k| [ n, 1, k ] }
n = a[0]
f = a[1]
k = a[2]
if ( n == 0 ) then return f
else
k.call n-1, n*f, k
end
end
end
'

p X.new.fact(6) # 720

Default Arguments
require "fastruby"

fastruby '
class X
def foo(a,b = a.reverse)
a + b
end
end
'

p X.new.foo("13") # 1331
p X.new.foo("xx", "yy") # xxyy

Passing proc objects as blocks
require "fastruby"

fastruby '
class X
def bar
yield(1)
yield(2)
yield(3)
end

def foo
pr = proc do |x|
p x
end

bar(&pr) # passing proc as a block
end
end
'

X.new.foo

Receiving blocks as proc objects
require "fastruby"

fastruby '
class X
def bar(&pr)
pr.call(1)
pr.call(2)
pr.call(3)
end

def foo
bar do |x|
p x
end
end
end
'

X.new.foo


domingo, 2 de octubre de 2011

Fastruby 0.0.11 released with support for retry, redo and foreach

Fastruby is a gem which allows to execute ruby code much faster than normal, currently in a state of transition between a spike and a usable gem, it is released when possible with incremental improvements.
The v0.0.11 release of fastruby make internal improvements regarding non-local jumps and implements foreach support, including retry and redo statements (both applicable to any kind of block, and retry applicable on rescue clause)

Install

You can clone the repository at github:
git clone git://github.com/tario/fastruby.git
git checkout v0.0.11
Or install it using gem install:
gem install fastruby

New Features


Examples of code being supported

For each with redo and retry

require "fastruby"

fastruby '
class X
def foo
sum = 0

for i in (4..8)
p i # 4 4 4 4 4 5 6 7 4 5 6 7 4 5 6 7 4 5 6 7 8
sum = sum + i
redo if sum < 20
retry if sum < 100 and i == 7
end

sum
end
end
'

x = X.new
x.foo # 46


miércoles, 28 de septiembre de 2011

Fastruby 0.0.9 released with support for proc

Fastruby is a gem which allows to execute ruby code much faster than normal, currently in a state of transition between a spike and a usable gem, it is released when possible with incremental improvements.
Following the release of version v0.0.8 which did implement several internal improvements including a new non-linear stack structure, the new release v0.0.9 implements the support for procedure objects based in the previous implementation

Install

You can clone the repository at github:
git clone git://github.com/tario/fastruby.git
git checkout v0.0.9
Or install it using gem install:
gem install fastruby

New Features


Examples of code being supported

return method from proc

require "fastruby"

fastruby '
class X
def foo
pr = Proc.new do
return "return foo from inside proc"
end

pr.call

return "return foo"
end
end
'

x = X.new
x.foo # this returns "return foo from inside proc"

The behavior of Proc objects (those created using Proc.new) are very similar to the behavior of lambdas, but with a few little differences:
  • It is possible to do non-local return's from Proc.new of the method where the block was defined, this non-local return will fail with LocalJumpError when the method scope is closed in the moment of invoking the return (in lambdas, return only acts as next; ie, only ends the lambda function)
  • Break jumps are illegal from Proc.new blocks and always fails with LocalJumpError (in lambdas, break acts as next)

As additional note, block objects created using proc method are exact the same block objects created using lambda method, only check the the ruby 1.8 source code to see something like that:

   ...
rb_define_global_function("proc", proc_lambda, 0);
rb_define_global_function("lambda", proc_lambda, 0);
...





domingo, 5 de junio de 2011

Shikashi v0.5.0 released

Shikashi is a sandbox for ruby that handles all ruby method calls executed in the interpreter to allow or deny these calls depending on the receiver object, the method name, the source file from where the call was originated

For more info about the project, visit the project page

You can install the gem by doing

gem install shikashi

New Enhancements

The primary focus of this latest release was improve the performance of execution of ruby code in the sandbox but maintaining the purity of ruby (i.e. without implementing anything with C or similar)

Code packets

In the previous version of shikashi (0.4.0), to execute code in the sandbox, you do:
Shikashi::Sandbox.run("1+1", Shikashi::Privileges.allow_method(:+))
This, internally, implies:
  • Parse the code
  • Process the syntax tree (evalhook)
  • Emulate the transformed tree (partialruby)
  • Execute the emulation code using eval...
This was a great performance problem in cases where you need to execute the same piece of ruby code many times in the sandbox (e.g. sandboxed web code executed for each incoming request)
To solve this, the concept of "code packet" was implemented; a "code packet" is the ultimate product of the above mentioned processing chain ready to be executed as many times as necessary without reprocess the code and tree. Example:
packet = Shikashi::Sandbox.packet(code)
# after that, you can run the "packet" as many times as you want without doing all
# parsing, tree processing and emulation stuff internally
packet.run(privileges)
packet.run(privileges)
packet.run(privileges)
packet.run(privileges)

The performance difference is of the order of 20X, you can check by doing the following benchmark:

require "rubygems"
require "shikashi"
require "benchmark"

code = "class X
def foo(n)
end
end
X.new.foo(1000)
"

s = Shikashi::Sandbox.new

Benchmark.bm(7) do |x|

x.report("normal") {
1000.times do
s.run(code, Shikashi::Privileges.allow_method(:new))
end
}

x.report("packet") {
packet = s.packet(code, Shikashi::Privileges.allow_method(:new))
1000.times do
packet.run
end
}

end

In my laptop, the result of this benchmark is:

            user     system      total        real
normal 5.870000 0.540000 6.410000 ( 6.436331)
packet 0.290000 0.020000 0.310000 ( 0.315351)

The repeated execution of code using packets is 20 times faster than normal

Refactor of evalhook internals

Evalhook is the support gem that provides the ability to hook events on the execution of ruby code. Evalhook implemenations initially was not desgined for performance, then it's possible to refactor the implementation to increase the performance by supressing unnecessary calls and other optimizations.

require "rubygems"
require "shikashi"
require "benchmark"

s = Shikashi::Sandbox.new

class NilClass
def foo
end
end

Benchmark.bm(7) do |x|

x.report {

code = "
500000.times {
nil.foo
}
"
s.run code, Shikashi::Privileges.allow_method(:times).allow_method(:foo)
}

end



The result without the optimizations:
            user     system      total        real
36.140000 1.420000 37.560000 ( 37.672708)


The result with the optimizations:
            user     system      total        real
27.460000 1.130000 28.590000 ( 28.658031)


The difference is not so significant because most of the time is used by privileges checking in both cases. Similar bechmarks executed against evalhook directly throws greater differences

Links

viernes, 29 de abril de 2011

Released Shikashi v0.4.0 (and dependencies)

Shikashi is an sandbox for ruby that handles all ruby method calls executed in the interpreter to allow or deny these calls depending on the receiver object, the method name, the source file from where the call was originated

For more info about the project, visit the project page

You can install the gem by doing


gem install shikashi


New Enhancements

removed evalmimic dependency

evalmimic is a gem to emulate the behavior of the binding argument of the eval method (the default binding), to allow the API to do this:

a = 5
Sandbox.run("a+1", Privileges.allow_method(:+)) # return 6


The implementation of evalmimic is somewhat complex because it relies in a C extension and even implements some low level hacks using unsupported features of MRI (e.g. evalmimic will not compile and install for ruby 1.9)

So it was decided to remove the evalmimic dependency from shikashi and evalhook, and remove the feature shown in the above example. The only difference now is that you must add the binding as parameter if you decide to execute the sandbox in that way.

a = 5
Sandbox.run("a+1", Privileges.allow_method(:+), binding) # return 6


And if you do not specify the binding, the default behavior is use the global binding nested in the sandbox namespace


Sugar Syntax

As seen in previous code examples, it's no longer necessary to instanciate lot of objects in order to execute code in the sandbox, only Sandbox.run and Privileges now use method chaining syntax. Example:

require "shikashi"

Sandbox.run('print "hello world\n"', Privileges.allow_method(:print))

$a = 1
Sandbox.run('print $a, "\n"',
Privileges.allow_method(:print).allow_global_read(:$a)
)



Control over read access of constants and global variables

Now, you must grant read privileges over global variables and constants in order to allow the read access to them. By default, trying to access to global variables and constants will result on SecurityError exceptions. Constants defined inside the base namespace of the sandbox are allowed by default (e.g. classes defined in the same code)

# this will work
include Shikashi
Sandbox.run("
class X
def foo
end
end
X.new.foo
", Privileges.allow_method(:new))

$a = 4
Sandbox.run("$a", Privileges.allow_global_read(:$a)) # 4

A = 4
Sandbox.run("A", Privileges.allow_const_read("A") # 4

Sandbox.run("$a") # raise SecurityError

Sandbox.run("A") # raise SecurityError




Interception of method calls using super on evalhook


Now, call to super methods are intercepted by evalhook and rejected by shikashi when appropiate

include Shikashi
#=begin
Sandbox.run("
class X
def system(*args)
super # raise SecurityError
end
end
X.new.system('ls -l')
", Privileges.allow_method(:new))


Refactor to use Ruby2Ruby on partialruby

Partialruby is the gem that emulates ruby using ruby to allow the changes to AST needed by evalhook for interceptions. Up to version 0.1.0, partialruby implements the emulation with abstract tree processing from scratch. Now, at released version 0.2.0, partialruby relies on more mature and stable gem Ruby2Ruby which converts AST to executable ruby source code

Links


martes, 19 de abril de 2011

Released ImageRuby - a flexible ruby gem for image processing

ImageRuby is a flexible gem for image processing, it's designed to be easy to install and use. The core of ImageRuby is written in pure ruby and the installation is as simple as execute "gem install imageruby". The API of the library take advantage of sugar syntax constructions specific of ruby language such as method chaining and blocks. e.g, the next code loads an image from "input.bmp", crop the rectangle from 0,0 to 49,49 (50x50 pixels), replace the orange color to red color, replace the black color to orange color and finally saves the image as "output.bmp"

require "imageruby"

ImageRuby::Image.from_file("input.bmp")[0..49,0..49].
color_replace(Color.orange,Color.red).
color_replace(Color.black,Color.orange).
save("output.bmp", :bmp)

Current Status

The ImageRuby had his first release (version 0.1.0) the last week with imageruby-c (extension to override a few methods with C methods) and imageruby-bmp (support for save and load images on BMP format),

You can install the gem by doing

gem install imageruby


The current version o imageruby has no dependencies, and it's installation should be as simple as that, optionally, you can install imageruby-c to improve the performance of draw and color_replace methods and imageruby-bmp to have support for bmp images.
The extensions take effect automagically by installing the gem, there is no need to do something specific in the ruby script (e.g. when install imageruby-c the methods will be optimized)

Goals of the project

  • Become synonymous of image processing in the ruby world, displacing other options in the "market" but re-using the existent stuff (giving credit, obviously)
  • Highlight deficiencies in the ruby language when used for heavy processing (e.g. draw method implemented in ruby is very slow)
  • Spike workarounds for the Ruby language issues
  • Spike "soft" dependency model or plugin gems to avoid the problems of "static" dependencies

Competitors

There is a range of gems for image processing on ruby, most of these are based on C libraries and implies the need for a compiler in order to install them. Some have "low level" bugs such memory leaks but most of them have many possibilities through their API

  • RMagick: Ruby wrapper of the archi-famous ImageMagick library. Has reported problems with low level memory handling and is not recommended for services environment (such as a Rails application) but is good for scripting
  • ImageScience: More stable alternative to RMagick, written on top of FreeImage and RubyInline (for more optimal code in C), an issue when using this library is the RubyInline dependency which implies the need for GCC installed on the system, something that is simple on Linux but is so tricky in other environments such Windows and Heroku
  • Camellia: A C/C++ library with a Ruby interface apparently directed to photo processing, there is no gem and the installation should be using the classic command sequence ./configure; make; make install.
  • Devil: Wrapper of C library of the same name. It has a wide range of operations over an image including blur and equalize. (see documentation)
  • Ruby-Processing: Well documented library mainly oriented to interactive media including video features. Has nice pencil functions
Future

New image formats

One of the next big steps will be develop decoder and encoders for common image formats, the encoders and decoders can be released as separated gems (like imageruby-bmp) without need for change imageruby core. In fact, anyone can make their own encoder or decoder.

Competitors become allies

Re-use the existing development around image processing in the ruby world, divided in two big groups: Interfaces and Implementations

Reusing interfaces implies the creation of "mocks" or "wrappers" of ImageRuby providing the same API of other library (e.g. RMagick or ImageScience) to reduce the learning curve of ImageRuby API and provide a method to replace that library with ImageScience transparently for code developed on top of that (e.g. a rails application using ImageMagick and then the ImageMagick gem is replaced by ImageRuby, app will not need to be modified)

Reusing implementation is about build features on top of other image libraries, but without setting a "hard" o "static" dependency with the library, but by creating a optional extension for ImageRuby (something like a port between libraries). Example: imageruby-devil.

Could be installed so:
gem install imageruby-devil

And used like
# is not necessary to make a explicit require of imageruby-devil
require "imageruby"

image = ImageRuby::Image.from_file("input.bmp")

image.color_replace!(Color.black, Color.purple) # use a method of ImageRuby
image.devil.blur(1) # use a method of Devil library on the image

image.save("output.bmp", :bmp)

Feedback for Ruby improvements

One of the goals of ImageRuby is to Highlight deficiencies in the ruby language , methods such draw or color_replace process hundred of thousands of pixels per image which in the ruby language implies still much more unnecessary rubymorphic calls just to access each bit of pixel data of the processed images. This produces a very very slow result, the current workaround for this issue is override the "heavy" methods with the imageruby-c extension. This was achieved as a optional extension to avoid the unnecessary hard dependency of have a compiler in the system, ImageRuby without imageruby-c installed HAS the draw and color_replace method, but with the imageruby-c gem installed, is much much more fast (near to 100x more faster)

Links

ImageRuby rdoc reference: http://tario.github.com/imageruby/doc/
ImageRuby GitHub site: https://github.com/tario/imageruby

lunes, 4 de abril de 2011

Evalhook v0.3.0 released and shikashi works on Heroku

Evalhook is a vital dependency of shikashi gem, provide the feature of execute managed ruby code in the way of allow "hooks" of common events like method calls, system calls, assignment of global variables and constants, etc...

Fixed Issues

With this release, problems of compatibility of actual version of shikashi (v0.3.1 at the moment of writing this post) were solved. The most important change was refactor of the hooking to make it more portable, less prone to error and implemented in pure ruby (the C extension was removed from the gem). this implies that now shikashi works on heroku and possibly in other environments

Update/Install

To update current installation of evalhook gem, run in terminal:

gem install evalhook

NOTE: To update the gem on heroku you must change Gemfile and Gemfile.lock files, then push this files to your application git repository, setting the content of Gemfile to something like that:

source :rubygems

gem 'shikashi'
gem 'evalhook', '0.3.0'
gem 'partialruby', '0.1.0'


Fundamentals

The most important concept behind evalhook is the AST hook technique which consist in change the Abstract Syntax Tree before it is interpreted and executed (view this other post for more info)

In previous versions of evalhook, the method to implement the AST hook was change the internal structures (representations of AST) on the version 1.8 of Matz Ruby Interpreter. This implementation breaks the encapsulation of MRI and depends on the specific implementation of that ruby interpreter making a very poor portability and being prone to error (e.g. the ruby interpreter crashes in heroku)

The reimplementation of the evalhook core is based on partialruby, a "para-interpreter" which executes ruby code by emulating it using ruby. Partialruby also exports the hooking services needed by evalhook thus avoiding the need for "hacks" to implement that service in a interpreter that does not have that. This concept is much more portable and theoretically should work with any interpreter of ruby 1.8 code.

Links

Shikashi Project description: http://tario-project.blogspot.com/2011/03/shikashi-flexible-sandbox-for-ruby.html
AST Hook fundamentals: http://tario-project.blogspot.com/2011/03/how-tree-hooks-works-and-what-are.html
Evalhook on Github https://github.com/tario/evalhook
Shikashi on Github: https://github.com/tario/shikashi
Heroku; http://heroku.com