Parallel, Concurrent and Using both of them in Ruby

Hi everyone,

I’ve been working on a food delivery platform over the 1 year and we some traveling salesman problem on the background. But this problem is little different one the original problem, we add some time constraints to routing. So for this type of problem we need make some fast calculations. For this purpose I’ve researched parallelism and concurrency usage in ruby.

This article is about the calculation speed in ruby. I want to show you, how we can increase the speed of calculation with small changes. Following scripts has some different procs for calculation. I’ll use these procs on below examples;

In the above runner.rb file I’ve give you example scripts for testing your calculation methods. concurrent_with_sleep_proc and calculate_with_sleep_proc have sleep in calculation. You can put there some io operation or some long running calculations. As you see on that file I did not run any calculate_with_sleep_proc with parallel like following;

 parallel_proc.call(arg_array, 24, calculate_with_sleep_proc) 

Because it takes time to calculate 10000 items to calculate in 24 processes with waiting 3 seconds.

When we use parallelism and concurrency together, it can be really fast as you see on the following. On this example I’ve used 10000 items with sleep in calculation. Following results come from with sleep part of runner.rb file.

With sleep;
              user     system      total        real
p&c10 :   0.004987   0.009114   8.745606 (  4.926278)
p&c11 :   0.007684   0.015370   9.212766 (  5.166559)
p&c12 :   0.005729   0.012166   8.855250 (  5.235328)
p&c13 :   0.007170   0.015991   8.578989 (  5.590199)
p&c14 :   0.006369   0.014589   7.640638 (  5.035913)
p&c15 :   0.005920   0.013606   7.590901 (  4.699597)
p&c16 :   0.006150   0.013080   7.686084 (  5.187887)
p&c17 :   0.006169   0.013702   7.179024 (  4.563827)
p&c18 :   0.008542   0.015548   7.505202 (  4.850113)
p&c19 :   0.007834   0.019239   7.105738 (  4.515940)
p&c20 :   0.007496   0.017252   6.859975 (  4.523301)
p&c21 :   0.010529   0.019268   7.220272 (  4.781848)
p&c22 :   0.011064   0.024596   6.510430 (  4.539044)
p&c23 :   0.011928   0.020285   6.786807 (  4.405131)
p&c24 :   0.008574   0.020050   6.672230 (  4.375151)
p&c25 :   0.009868   0.021247   6.598457 (  4.338372)
p&c26 :   0.011396   0.021693   6.415795 (  4.252782)
c     :   3.877854   1.743947   5.621801 ( 16.474863)

If you don’t have any long waiting like sleep in your calculation, pure parallelism can be more fast; Following results come from without sleep part of runner.rb file.

Without sleep;
              user     system      total        real
p&c10 :   0.004269   0.007311   3.289474 (  0.901183)
p&c11 :   0.004811   0.009777   3.833186 (  1.015616)
p&c12 :   0.004785   0.009611   3.402832 (  0.900057)
p&c13 :   0.006303   0.010684   3.168834 (  0.872175)
p&c14 :   0.007362   0.011854   3.595352 (  0.953701)
p&c15 :   0.006461   0.011326   3.381846 (  0.906724)
p&c16 :   0.007204   0.011498   3.400029 (  1.056723)
p&c17 :   0.012351   0.022792   3.645585 (  1.418485)
p&c18 :   0.010182   0.023389   3.846147 (  1.331361)
p&c19 :   0.008289   0.017057   3.363501 (  0.936982)
p&c20 :   0.008949   0.015332   3.747166 (  1.217274)
p&c21 :   0.009793   0.016687   3.752948 (  1.262270)
p&c22 :   0.012197   0.024683   3.566022 (  1.227817)
p&c23 :   0.009727   0.019681   3.953732 (  1.310838)
p&c24 :   0.008423   0.017569   3.603448 (  1.135166)
p&c25 :   0.009448   0.020335   4.148266 (  1.468371)
p&c26 :   0.010681   0.023786   4.596350 (  1.862336)
p10   :   0.004490   0.009256   0.150146 (  0.065307)
p11   :   0.004768   0.009506   0.182004 (  0.080961)
p12   :   0.004725   0.009574   0.177207 (  0.095620)
p13   :   0.005870   0.013676   0.122861 (  0.103468)
p14   :   0.004806   0.010406   0.117565 (  0.042819)
p15   :   0.005112   0.011147   0.129349 (  0.047156)
p16   :   0.005038   0.010570   0.136523 (  0.047802)
p17   :   0.005091   0.011161   0.144015 (  0.054687)
p18   :   0.006997   0.015194   0.179129 (  0.119613)
p19   :   0.007105   0.017206   0.204005 (  0.105792)
p20   :   0.007530   0.016820   0.214045 (  0.102703)
p21   :   0.008028   0.018449   0.271488 (  0.187290)
p22   :   0.008422   0.019139   0.244256 (  0.149529)
p23   :   0.007691   0.016469   0.259061 (  0.121158)
p24   :   0.008392   0.019364   0.267913 (  0.122444)
p25   :   0.013546   0.023487   0.285710 (  0.165075)
p26   :   0.009482   0.022572   0.209581 (  0.083306)
c     :   2.107816   1.099271   3.207087 (  2.729089)

So parallelism and concurrency can be really helpful for some calculations. I’ve also tested some bigger arrays and results are like following;

1_000_000 items array without sleep in calculation;


                user     system        total          real
p&c24  :     0.350523   0.154071  311.822112 ( 103.817200)
p&c100 :     0.362228   0.252416  291.592356 (  78.864338)
c      :  1827.912725 140.046733 1967.959458 (2047.510425)
p100   :     0.381085   0.134941    3.518979 (   1.227611)

10_000_000 items array without sleep in calculation;

               user     system      total        real
p24  :    25.753854   0.810221  43.390889 ( 28.204278)
p100 :    14.243962   0.682502  98.411622 ( 30.706949)

For large arrays parallelism is best for calculation. According to your problem you can choose concurrency, parallelism or both of them for calculation.

For bigger arrays 100_000_000 I’ve tried to split arrays into small parts like 100_000 item for each array element and calculate each group with parallel or concurrent. But for bigger arrays it takes time to split and merge array. So it did not solve real problem. For 1_000_000_000 elements, it is really problem calculate each item or item group with limited time.

I’ve shared my experiences over calculation speed for large arrays in ruby.

Happy coding..

Leave a Reply

  

  

  

*