With those I am getting bellow logs Found 2 pods, using pod/opal-server-daemonset-lhrxk [2022-12-15 11:15:08 +0000] [1] [INFO] Starting gunicorn 20.1.0 [2022-12-15 11:15:08 +0000] [1] [INFO] Listening at: http://0.0.0.0:7002 (1) [2022-12-15 11:15:08 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker [2022-12-15 11:15:08 +0000] [10] [INFO] Booting worker with pid: 10 [2022-12-15 11:15:08 +0000] [11] [INFO] Booting worker with pid: 11 [2022-12-15 11:15:08 +0000] [14] [INFO] Booting worker with pid: 14 2022-12-15T11:15:09.239244+0000 | 10 | opal_server.server | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests! 2022-12-15T11:15:09.239620+0000 | 10 | opal_server.pubsub | INFO | Initializing broadcaster for server<->server communication 2022-12-15T11:15:09.282074+0000 | 10 | opal_server.server | INFO | *** OPAL Server Startup *** 2022-12-15T11:15:09.282462+0000 | 10 | opal_server.server | INFO | leadership lock acquired, leader pid: 10 2022-12-15T11:15:09.282582+0000 | 10 | opal_server.server | INFO | listening on webhook topic: 'webhook' 2022-12-15T11:15:09.282924+0000 | 10 | opal_server.server | INFO | Policy repo will be cloned to: /opal/regoclone/opal_repo_clone 2022-12-15T11:15:09.283328+0000 | 10 | fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': 'ac53673313274dd2a5c23705a3d25508', 'subscriber_id': '81d8ce186af347c690f4e3f919b69a4a', 'topic': 'webhook', 'callback': functools.partial(, ), 'notifier_id': None} 2022-12-15T11:15:09.283435+0000 | 10 | opal_server.policy.watcher.task | INFO | Launching policy watcher 2022-12-15T11:15:09.283506+0000 | 10 | opal_common.topics.publisher | INFO | started broadcaster keepalive task: topic is '__broadcast_session_keepalive__', interval is 3600 seconds 2022-12-15T11:15:09.283892+0000 | 10 | opal_server.data.data_update_publisher | INFO | [10] Starting Polling Updates 2022-12-15T11:15:09.285781+0000 | 10 | opal_common.git.repo_cloner | INFO | Cloning repo from 'https://github.com/sumeshes07/opal-example-policy-repo-new' to '/opal/regoclone/opal_repo_clone' (branch: 'master') 2022-12-15T11:15:09.362981+0000 | 11 | opal_server.server | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests! 2022-12-15T11:15:09.363394+0000 | 11 | opal_server.pubsub | INFO | Initializing broadcaster for server<->server communication 2022-12-15T11:15:09.425370+0000 | 11 | opal_server.server | INFO | *** OPAL Server Startup *** 2022-12-15T11:15:09.451634+0000 | 14 | opal_server.server | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests! 2022-12-15T11:15:09.453634+0000 | 14 | opal_server.pubsub | INFO | Initializing broadcaster for server<->server communication 2022-12-15T11:15:09.495282+0000 | 14 | opal_server.server | INFO | *** OPAL Server Startup *** 2022-12-15T11:15:10.392344+0000 | 10 | opal_common.git.repo_cloner | INFO | Clone succeeded 2022-12-15T11:15:10.395792+0000 | 10 | opal_common.sources.base_policy_source | INFO | Polling task is off 2022-12-15T11:16:50.556036+0000 | 14 | uvicorn.protocols.http.httptools_impl | INFO | 127.0.0.1:56486 - "GET / HTTP/1.1" 200 2022-12-15T11:17:14.770293+0000 | 14 | uvicorn.protocols.http.httptools_impl | INFO | 183.83.252.7:0 - "GET / HTTP/1.1" 200 2022-12-15T11:17:14.897240+0000 | 14 | uvicorn.protocols.http.httptools_impl | INFO | 183.83.252.7:0 - "GET /favicon.ico HTTP/1.1" 404 2022-12-15T11:20:03.727629+0000 | 14 | uvicorn.protocols.http.httptools_impl | INFO | 122.50.209.215:0 - "GET / HTTP/1.1" 200 2022-12-15T11:20:16.388342+0000 | 14 | opal_server.policy.webhook.api | INFO | triggered webhook on repo: https://github.com/sumeshes07/opal-example-policy-repo-new 2022-12-15T11:20:16.397782+0000 | 14 | fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': '329c85860fd14d5e8e0225db84a70692', 'subscriber_id': '16918c566d6445fa9e56175696d2ba57', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': >, 'notifier_id': None} 2022-12-15T11:20:16.397988+0000 | 14 | fastapi_websocket_pubsub.event_notifier | INFO | calling subscription callbacks: topic=webhook (ALL_TOPICS), subscription_id=329c85860fd14d5e8e0225db84a70692, subscriber_id=16918c566d6445fa9e56175696d2ba57 2022-12-15T11:20:16.398060+0000 | 14 | fastapi_websocket_pubsub.event_broadc...| INFO | Broadcasting incoming event: {'topic': 'webhook', 'notifier_id': '16918c566d6445fa9e56175696d2ba57'} 2022-12-15T11:20:37.448343+0000 | 14 | fastapi_websocket_pubsub.event_notifier |ERROR | Failed to notify subscriber sub_id=16918c566d6445fa9e56175696d2ba57 with topic=webhook Traceback (most recent call last): File "/usr/local/bin/gunicorn", line 33, in sys.exit(load_entry_point('gunicorn==20.1.0', 'console_scripts', 'gunicorn')()) │ │ └ │ └ File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 67, in run WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run() └ File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 231, in run super().run() File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 72, in run Arbiter(self).run() │ └ File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 202, in run self.manage_workers() │ └ File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 551, in manage_workers self.spawn_workers() │ └ File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers self.spawn_worker() │ └ File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker worker.init_process() │ └ File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process super(UvicornWorker, self).init_process() │ └ File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 142, in init_process self.run() │ └ File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 83, in run return asyncio.run(self._serve()) │ │ │ └ │ │ └ │ └ File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) │ │ └ │ └ > File "/usr/local/lib/python3.10/site-packages/fastapi_websocket_pubsub/event_notifier.py", line 220, in callback_subscribers await self.trigger_callback(data, topic, subscriber_id, event) │ │ │ │ │ └ Subscription(id='329c85860fd14d5e8e0225db84a70692', subscriber_id='16918c566d6445fa9e56175696d2ba57', topic='webhook', callba... │ │ │ │ └ '16918c566d6445fa9e56175696d2ba57' │ │ │ └ 'webhook' │ │ └ None │ └ File "/usr/local/lib/python3.10/site-packages/fastapi_websocket_pubsub/event_notifier.py", line 178, in trigger_callback await subscription.callback(subscription, data) │ │ │ └ None │ │ └ Subscription(id='329c85860fd14d5e8e0225db84a70692', subscriber_id='16918c566d6445fa9e56175696d2ba57', topic='webhook', callba... │ └ File "/usr/local/lib/python3.10/site-packages/broadcaster/_base.py", line 55, in __aenter__ await self.connect() │ └ File "/usr/local/lib/python3.10/site-packages/broadcaster/_base.py", line 62, in connect await self._backend.connect() │ │ └ │ └ File "/usr/local/lib/python3.10/site-packages/broadcaster/_backends/postgres.py", line 15, in connect self._conn = await asyncpg.connect(self._url) │ │ │ │ └ 'postgres://postgres:postgres@10.96.116.182:5432/postgres' │ │ │ └ │ │ └ │ └ File "/usr/local/lib/python3.10/site-packages/asyncpg/connection.py", line 2093, in connect return await connect_utils._connect( │ └ File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 903, in _connect raise last_error └ ConnectionRefusedError(111, 'Connection refused') File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 889, in _connect return await _connect_addr( └ File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 781, in _connect_addr return await __connect_addr(params, timeout, True, *args) │ │ │ └ (('10.96.116.182', 5432), , ConnectionConfiguration(command_timeout=None, ... │ │ └ 60 │ └ ConnectionParameters(user='postgres', password='postgres', database='postgres', ssl=... └ File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 833, in __connect_addr tr, pr = await compat.wait_for(connector, timeout=timeout) │ │ │ └ 60 │ │ └ File "/usr/local/lib/python3.10/site-packages/asyncpg/compat.py", line 66, in wait_for return await asyncio.wait_for(fut, timeout) │ │ │ └ 60 │ │ └ File "/usr/local/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() │ └ File "uvloop/loop.pyx", line 2039, in create_connection raise exceptions[0] File "uvloop/loop.pyx", line 2016, in uvloop.loop.Loop.create_connection await waiter ConnectionRefusedError: [Errno 111] Connection refused 2022-12-15T11:20:37.453760+0000 | 14 | fastapi_websocket_pubsub.event_notifier | INFO | Removing Subscription of topic='__EventNotifier_ALL_TOPICS__' for subscriber=16918c566d6445fa9e56175696d2ba57