Cron Jobs — Master Worker Strategy 🧮

Cron jobs using sidekiq in ruby on rails

Introduction

Imaginary problem time

The problem begins when we realise that these many calls will take long time to process and our server will always be busy. Hell, if we encounter one of those viral posts containing 100K comments, our processing will surpass a whole hour for it and we’ll not be able to do anything. SHAME ON US!

Technologies

  • Sidekiq-cron — Gem to run cron jobs (same task after certain interval, like daily/hourly).
  • Redis — An in-memory database to track state of our async jobs.

Setup and Codes

Gemfile
config/sidekiq.yml
config/sidekiq_cron.yml
config/intializers/sidekiq.rb
app/workers/comment_master.rb
app/workers/comment_fetcher.rb

Explanation

Coming to CommentMaster class, you’ll find it calling CommentFetcher’s perform_async method for all posts at once. This perform_async is a special method provided by sidekiq to call that method asynchronously. Ah! Sweet smell of JavaScript.

Looking at the bigger picture, 1000 jobs will be submitted at once to process our 1000 posts and will get scheduled in the queue. Notice the concurrency = 10, it means that 10 jobs from that queue will run in parallel.

Major benefit comes to light when we understand that any one job failing doesn’t impact others, no matter if its an alien attack error situation too. Had we ran with perform instead of perform_async, any uncatched error would have terminated the loop, because former is just like calling a method in loop. Paranoia is all good but it’s not possible to wrap up everything in exception rescue blocks, so some of our elves(workers) will die while processing job, leaving an error log for us to refer later.

Conclusion

Do follow for more Ruby on Rails posts.

To Connect

Currently working as Senior Ruby on Rails engineer at Sedin Technologies, India. I love Coding, Travelling and Teaching. Call me world's happiest engineer. 😇