feat(batch): Enabling job informer + job list pagination (#2322)

* Enabling job informer + job list pagination.
* Resolve gemini code review comments
* Add REDIS to allowed metadata list and move check to initialization instead of list_batch_jobs call

Signed-off-by: Jingyuan Zhang <jingyuan.zhang0929@bytedance.com>
Co-authored-by: Jingyuan Zhang <jingyuan.zhang0929@bytedance.com>