Common mistakes with Sidekiq
Sidekiq is one of the most popular background job framework in Ruby/Ruby on Rails world. It’s easy to use, but I keep finding the same mistakes made by different people in different projects. Bellow I will describe those mistakes to help you avoiding them. As well as tips on how to avoid or deal with them. Also, I’ll try to make this article lean, and instead of writing additional/educational explanations here, I’ll give lots of links.
Don’t enqueue Job inside transaction
This is the easiest case. Just look at this example:
UserMailer.created(user_params).deliver_later # Rails will use Sidekiq to send email
raise "Let's imagine something went wrong here"end
As you can see, it creates user, writes logs and sends email. If something breaks inside that transaction it rollback user create and write log operations. But it won’t do anything with mailer job. Furthermore, even if there is no problem, the job may start before the data from the transaction is written to the database. So if you Job is depended on data from the database it can fail.
Sidekiq is asynchronous
Sidekiq processes jobs in the order they are created, but by default it processes multiple jobs simultaneously, and there is no guarantee that a given job will finish before another job is started. Here is what official FAQ tells us:
You can’t, by design. Sidekiq is designed for asynchronous processing of jobs that can be completed in isolation and independent of each other. Jobs will be popped off of Redis in the order in which they were pushed but there’s no guarantee that Job #1 will execute fully before Job #2 is started.
If you need serial execution, you should look into other systems which give those types of guarantees.
Sidekiq can lose Jobs
There are multiple scenarios of how you can lose a Sidekiq job.
The first one is when Sidekiq processing a job the data is dequeued from Redis and if the worker is interrupted and killed, the job will be lost. So just consider Sidekiq is not reliable and don’t use it for business-critical background jobs you cannot afford to lose. You might consider using Sidekiq Pro or something more reliable for business-critical background jobs.
Another frequent scenario is that Redis might fail somewhere around job processing. Fortunately, this case is quite easy to deal with. Just configure Redis persistence.
One more piece of advice. Be extra careful on deployments. Don’t just kill Sidekiq or Redis without ensuring Sidekiq finished all Workers.
Scaling Sidekiq with Redis Cluster
So, you want to scale your Sidekiq. You might think that along with increasing the number of Sidekiq instances, it would be a good idea to create a Redis Cluster. This is a bad idea because Sidekiq will not work with Redis Cluster. Here is what official FAQ tells us:
Redis offers many different topologies:
Single node — offers no fault tolerence
Redis Sentinel — offers fault tolerence, fails over to a replica in case of primary failure
Redis Cluster — multi-master keyspace spread across many instances
Cluster is designed for large-scale datasets, like caches, that can spread evenly across machines. Cluster is NOT appropriate for Sidekiq as Sidekiq has a few very hot keys which are constantly changing (aka queues). I recommend using Sentinel or use a Redis SaaS which has built-in support for failover.
You can consider other workarounds like vertical scaling, batch processing or sharding.
Don’t use the same Redis for caching and for Sidekiq
Usually people use Redis as a cache store in Rails, and it is important that production Sidekiq runs on a Redis instance that is configured as a persistent repository. And you can’t use Redis namespaces and have separate configurations for cache store and persistent store in separate namespaces, so you have to use separate Redis instance.
These were mistakes/bad practices I’ve seen most often. They are easy to detect and avoid, but fixing them after you’ve already implemented a feature with described problems can be difficult and time-consuming. Therefore, knowing them in advance, you can save a lot of time and effort.