Fixes #411 Wrap job invocation in Rails executor when using ActiveRecord

[tomgi]

See #411 first.

The connection is being released by que, because Rails.application.executor has this callback registered, being called here.

A smaller reproducible example is here (raises on master, doesn't raise on this branch anymore):

class TestQueJob < Que::Job
  def run(id)
    conn = ::ActiveRecord::Base.connection

    Que.pool.execute("SELECT 1;")
    
    if conn != ::ActiveRecord::Base.connection
      raise "Connection changed"
    end
  end
end

I think que should be executing the entire job code inside of Rails.application.executor, then the nested one in connection would be a noop and the connection would only be released after the whole job is finished.

That's what other background job libraries like sidekiq and shoryuken are doing, which follow the Rails guidelines for writing libraries here https://guides.rubyonrails.org/v7.1/threading_and_code_execution.html#wrapping-application-code

If you're writing a library or component that will invoke application code, you should wrap it with a call to the executor

[tomgi]

I've added a small fix to this branch that we discovered while testing it in production.

The problem is very similar to #393 - it happens when nesting a synchronous job execution inside of another Que job, e.g.:

class TestQueJob < Que::Job
  def run
    ActiveRecord::Base.transaction do
      ActiveRecord::Base.connection_pool.active_connection? # true
      ActiveRecord::Base.connection.transaction_open? # true

      TestQueJob2.run
      # ^ the job runs synchronously, invokes a nested instance of Que::ActiveRecord::Connection::JobMiddleware 
      # and calls ::ActiveRecord::Base.clear_active_connections!

      raise "lost connection" unless ActiveRecord::Base.connection_pool.active_connection?
      # ^ our connection with open transaction got returned back to the pool
    end
  end
end

class TestQueJob2 < Que::Job
  def run; end
end

That sample job would raise before the patch applied in 3d856c0

[owst]

This would be great to get merged - @tomgi any thoughts on the cause of the failing checks?

[owst]

Looks like #421 may well be related

[tomgi]

@oeoeaio @ZimbiX any chance we could get this merged into master?

We've been using it in production for over a year already and it does fix #411