RetroSearch Browse

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://github.com/celery/celery/discussions/7106 below:

Add a configuration option to disable prefetch completely · celery/celery · Discussion #7106 · GitHub

Checklist

I have verified that the issue exists against the master branch of Celery.
This has already been asked to the discussion group first.
I have read the relevant section in the
contribution guide
on reporting bugs.
I have checked the issues list
for similar or identical bug reports.
I have checked the pull requests list
for existing proposed fixes.
I have checked the commit log
to find out if the bug was already fixed in the master branch.
I have included all related issues and possible duplicate issues
in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.
(if you are not able to do this, then at least specify the Celery
version affected).
I have verified that the issue exists against the master branch of Celery.
I have included the contents of pip freeze in the issue.
I have included all the versions of all the external dependencies required
to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version
and/or implementation.
I have tried reproducing the issue on more than one message broker and/or
result backend.
I have tried reproducing the issue on more than one version of the message
broker and/or result backend.
I have tried reproducing the issue on more than one operating system.
I have tried reproducing the issue on more than one workers pool.
I have tried reproducing the issue with autoscaling, retries,
ETA/Countdown & rate limits disabled.
I have tried reproducing the issue after downgrading
and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 4.3.0 (rhubarb)

celery report Output:

software -> celery:4.3.0 (rhubarb) kombu:4.6.11 py:3.6.12
            billiard:3.6.3.0 redis:3.3.11
platform -> system:Linux arch:64bit, ELF
            kernel version:4.14.106-97.85.amzn2.x86_64 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://query-runner-redis-master:6379/0

BROKER_URL: 'redis://query-runner-redis-master:6379/0'
CELERY_RESULT_BACKEND: 'redis://query-runner-redis-master:6379/0'
CELERY_INCLUDE: ['tasks.query']
CELERY_TIMEZONE: 'UTC'
CELERY_TASK_SERIALIZER: 'json'
CELERY_IGNORE_RESULT: True
CELERY_TASK_ACKS_LATE: True
CELERYD_PREFETCH_MULTIPLIER: 1
CELERY_ROUTES: {
    'tasks.query.kill_query': {'queue': 'meta'},
    'tasks.query.run_query': {'queue': 'default'}}
CELERY_ALWAYS_EAGER: False
CELERYBEAT_SCHEDULE: {
 }
CELERY_TASK_CREATE_MISSING_QUEUES: True

Steps to Reproduce Required Dependencies

Minimal Python Version: N/A or Unknown
Minimal Celery Version: N/A or Unknown
Minimal Kombu Version: N/A or Unknown
Minimal Broker Version: N/A or Unknown
Minimal Result Backend Version: N/A or Unknown
Minimal OS and/or Kernel Version: N/A or Unknown
Minimal Broker Client Version: N/A or Unknown
Minimal Result Backend Client Version: N/A or Unknown

Python Packages pip freeze Output:

amqp==2.6.1
billiard==3.6.3.0
blinker==1.4
boto==2.49.0
boto3==1.16.21
botocore==1.19.21
celery==4.3.0
celery-prometheus-exporter==1.7.0
certifi==2020.11.8
chardet==3.0.4
cheroot==8.4.7
CherryPy==18.6.0
click==7.1.2
datadog==0.39.0
decorator==4.4.2
env-excavator==1.5.0
Flask==1.1.2
Flask-Cors==3.0.9
flask-swagger==0.2.14
flask-swagger-ui==3.36.0
flower==0.9.5
future==0.18.2
gunicorn==20.0.4
humanize==3.1.0
idna==2.10
importlib-metadata==2.0.0
importlib-resources==3.3.0
inflection==0.5.1
influxdb==5.3.1
itsdangerous==1.1.0
jaraco.classes==3.1.0
jaraco.collections==3.0.0
jaraco.functools==3.0.1
jaraco.text==3.2.0
JayDeBeApi==1.1.1
Jinja2==2.11.2
jmespath==0.10.0
JPype1==0.6.3
kombu==4.6.11
MarkupSafe==1.1.1
more-itertools==8.6.0
msgpack==1.0.0
numpy==1.17.0
portend==2.7.0
prometheus-client==0.9.0
prometheus-flask-exporter==0.18.1
psycopg2==2.8.6
pycrypto==2.6.1
PyHive==0.6.3
PyMySQL==0.10.1
python-dateutil==2.8.1
pytz==2020.4
PyYAML==5.3.1
qds-sdk==1.16.1
raven==6.10.0
redis==3.3.11
redlock==1.2.0
requests==2.25.0
s3transfer==0.3.3
six==1.15.0
tempora==4.0.1
thrift==0.13.0
tornado==6.1
urllib3==1.26.2
vine==1.3.0
Werkzeug==1.0.1
zc.lockfile==2.0
zipp==3.4.0

Other Dependencies

N/A

Minimally Reproducible Test Case Expected Behavior

I was hoping by specifying -Ofair that tasks would not be reserved by workers as stated here https://docs.celeryproject.org/en/v4.3.0/userguide/optimizing.html?highlight=optimization#prefork-pool-prefetch-settings

Actual Behavior

Workers have reserved tasks while other workers are free to do work.

I am going to paste the state we are seeing in prod when a lot of tasks come in. Notice how query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj has a reserved task while celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9 is empty and could run that task. Please note the name of the worker is the name of the queue they are reading from.

celery inspect active and celery inspect reserved Output:


root@query-runner-celery-datalake-687847cbb6-mxrnc:/opt/workdir# celery -A tasks inspect active
2020-11-24 21:03:59,312 [WARNING] datadog.api: No agent or invalid configuration file found
2020-11-24 21:03:59,316 [INFO] tasks.datadog_helper: initialized with
2020-11-24 21:03:59,316 [INFO] tasks.datadog_helper: {'statsd_host': '172.29.156.91', 'statsd_port': 8125, 'statsd_namespace': 'query_runner_celery_tasks'}
-> celery@query-runner-celery-default-6c9bd65fd7-4cdz6: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-v8w22: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-68xqn: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-lsk2z: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-mxrnc: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-9f8r2: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-hnrhw: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-9jp2n: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-4dx2b: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-q82hx: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk: OK
    * {'id': '93a32a04-7a6d-4a32-a2ee-8b88ca4f9049', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_aa42f1c1970bc1ac2778a65c60b2e08b', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3107', 'caller_json': {'queryId': 3107, 'interval': 'HOURLY', 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606251603.8548772, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': '747a1dd7-209a-4870-a636-c18176587723', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_373fafa1388d047c83cae9d40f249655', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3149 on dash 616', 'caller_json': {'queryId': 3149, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606251609.6809852, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
    * {'id': '28ec904f-197f-4aef-99e2-0887f6f85ec7', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_829e5787764013fbf5eb0958eec86a8f', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2400 on dash 609', 'caller_json': {'queryId': 2400, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606251604.9346218, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '705d4f46-662d-4930-9011-265e1065346d', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_04cd73d13695b2a7d9503148018c2650', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3179 on dash 616', 'caller_json': {'queryId': 3179, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606240820.285725, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
-> celery@query-runner-celery-default-6c9bd65fd7-4s4lz: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc: OK
    * {'id': '7afd7a99-368e-4995-8e5a-a1802dff7476', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_9fe008b693e6dc12999bf58212430afa', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2403 on dash 609', 'caller_json': {'queryId': 2403, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc', 'time_start': 1606251605.1041203, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': '4d59fc3f-40b9-4d51-ad41-e241518c629f', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_9a6f5ab9f25ab21bddcc128aee8677d6', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2396 on dash 609', 'caller_json': {'queryId': 2396, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc', 'time_start': 1606251604.2693179, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '05bc6789-2183-47d2-abe0-7093bad200a5', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_a937a6def4c1010c15dd3c5e1b488d2d', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3116 on dash 616', 'caller_json': {'queryId': 3116, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc', 'time_start': 1606251609.2829497, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc: OK
    * {'id': 'd6ddf309-e58a-4f2d-be71-1824e618716e', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_b7253439941188da3776ffdc8bab2b07', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2401 on dash 609', 'caller_json': {'queryId': 2401, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251605.0180476, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '62bb08c5-d6a6-4966-8de1-0d872faba09f', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_b75d10ba197e1bb1ab787217ca5daa96', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2395 on dash 609', 'caller_json': {'queryId': 2395, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251604.1448753, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': '82240c16-ec84-4279-9cd4-694a73a145eb', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_6435b6004863fd5a10de1abc8bf294cd', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3188 on dash 624', 'caller_json': {'queryId': 3188, 'interval': 'HOURLY', 'dashboard_id': 624, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251609.9815636, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
    * {'id': '9c062ae6-bc8d-44d1-b861-068cd1032c38', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_6912fef8aee81d8aa0a68f1326e5bbf6', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3117 on dash 616', 'caller_json': {'queryId': 3117, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251609.2058136, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-94qs4: OK
    * {'id': '87228aca-e02f-48a3-9096-5540c7b4fb22', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_e97a5e28903b3652a4a2c386fc287cc6', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3115 on dash 616', 'caller_json': {'queryId': 3115, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-94qs4', 'time_start': 1606251609.4260392, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj: OK
    * {'id': '1a8982d5-32cd-4fcf-a134-72a79d7e5da7', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_02758ef185f77a7397ead9e8e90e9743', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2397 on dash 609', 'caller_json': {'queryId': 2397, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj', 'time_start': 1606251604.4064894, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': 'b2804139-6d51-4ffe-817b-8b9efc7a9cd4', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_1920e1f67f84ac24fb3d2b8e8bb4bd22', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3114 on dash 616', 'caller_json': {'queryId': 3114, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj', 'time_start': 1606251609.3467886, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
-> celery@query-runner-celery-ga-55cdb77d4-j6994: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-m8wfp: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-drdl2: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-lgstj: OK
    * {'id': '741f2751-ca24-4bcd-b78c-1d92e679952d', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_e526e7f0e7a7c28271dc8f08ee19504b', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3021 on dash 595', 'caller_json': {'queryId': 3021, 'interval': 'HOURLY', 'dashboard_id': 595, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-lgstj', 'time_start': 1606251616.69369, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
-> celery@query-runner-celery-ga-55cdb77d4-6jxzz: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh: OK
    * {'id': '650009cc-4d6b-42c4-96b1-9a3112a1e5c8', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_711f6cdfe13c22710887f00f5678baeb', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2399 on dash 609', 'caller_json': {'queryId': 2399, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh', 'time_start': 1606251604.6513581, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': 'd74e3272-45f0-44ee-bcab-7b67de60b407', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_5ca350ca8b4b185ad42f94c6dd3b6fc9', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3189 on dash 624', 'caller_json': {'queryId': 3189, 'interval': 'HOURLY', 'dashboard_id': 624, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh', 'time_start': 1606251611.4904687, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
    * {'id': '6f61b237-fa9e-4b30-8131-8d8ee75306c5', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_9bc92aabee38e8d6d607bbb24785d814', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2137 on dash 557', 'caller_json': {'queryId': 2137, 'interval': 'HOURLY', 'dashboard_id': 557, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh', 'time_start': 1606248030.1126661, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
-> celery@query-runner-celery-datalake-687847cbb6-dvdpl: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-pdxsk: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-vl2t6: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-g55gl: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-7spd4: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj: OK
    * {'id': 'ab45784f-8d85-452c-b690-43ead1a010c6', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_bb8a680bab37f6bc86eb9f5c7435d656', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2816 on dash 609', 'caller_json': {'queryId': 2816, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606251605.576281, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
    * {'id': '4e64157e-9356-4362-88af-318013c0c4de', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_4859064fc4b96df99eec68e94f733157', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3143 on dash 616', 'caller_json': {'queryId': 3143, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606251609.5426545, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
    * {'id': 'ac0c403b-ade6-4377-9a41-6d305268204b', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_b04a82ce9376c9c6618c3a300abe4182', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3109', 'caller_json': {'queryId': 3109, 'interval': 'HOURLY', 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606240815.8285584, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': 'e9c368de-d47f-4a84-b05d-81cbda00a394', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_4d03e965dbd57a05d6071c0b5cbf699d', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2398 on dash 609', 'caller_json': {'queryId': 2398, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606251604.517987, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
-> celery@query-runner-celery-default-6c9bd65fd7-8zdvm: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc: OK
    * {'id': '03e786df-44b1-412a-8354-3acbc762528f', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_0687aa16794011f4d1a0ee1023173bf8', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3105 on dash 557', 'caller_json': {'queryId': 3105, 'interval': 'HOURLY', 'dashboard_id': 557, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc', 'time_start': 1606251618.9102547, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '646617ce-9835-4b60-b5fd-c4a4a5631749', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_ed09cf8c1166115a4ed1a5aa33b032c7', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3144 on dash 616', 'caller_json': {'queryId': 3144, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc', 'time_start': 1606251609.6082683, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
root@query-runner-celery-datalake-687847cbb6-mxrnc:/opt/workdir# celery -A tasks inspect reserved
2020-11-24 21:04:03,851 [WARNING] datadog.api: No agent or invalid configuration file found
2020-11-24 21:04:03,854 [INFO] tasks.datadog_helper: initialized with
2020-11-24 21:04:03,855 [INFO] tasks.datadog_helper: {'statsd_host': '172.29.156.91', 'statsd_port': 8125, 'statsd_namespace': 'query_runner_celery_tasks'}
-> celery@query-runner-celery-default-6c9bd65fd7-4cdz6: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-v8w22: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-mxrnc: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-68xqn: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-q82hx: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-lsk2z: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-4dx2b: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-hnrhw: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-lgstj: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-9f8r2: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-j6994: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-9jp2n: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-4s4lz: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-94qs4: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-m8wfp: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-6jxzz: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-dvdpl: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-g55gl: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-drdl2: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-vl2t6: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-8zdvm: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj: OK
    * {'id': '3f501bbe-6c56-48e2-ba64-9b1adac30695', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_6eb04958c42a71ebd014f9be9e116f73', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2823 on dash 557', 'caller_json': {'queryId': 2823, 'interval': 'HOURLY', 'dashboard_id': 557, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': None, 'acknowledged': False, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': None}
-> celery@query-runner-celery-datalake-687847cbb6-7spd4: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-pdxsk: OK
    - empty -

Also, here is the output of celery inspect stats where you can see fair is getting in properly

celery inspect stats for worker query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9 Output:

``` -> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9: OK { "broker": { "alternates": [], "connect_timeout": 4, "failover_strategy": "round-robin", "heartbeat": 120.0, "hostname": "query-runner-redis-master", "insist": false, "login_method": null, "port": 6379, "ssl": false, "transport": "redis", "transport_options": {}, "uri_prefix": null, "userid": null, "virtual_host": "0" }, "clock": "8600271", "pid": 8, "pool": { "max-concurrency": 4, "max-tasks-per-child": "N/A", "processes": [ 13, 14, 15, 16 ], "put-guarded-by-semaphore": false, "timeouts": [ 0, 0 ], "writes": { "all": "42.86%, 42.86%, 14.29%", "avg": "33.33%", "inqueues": { "active": 0, "total": 4 }, "raw": "3, 3, 1", "strategy": "fair", "total": 7 } }, "prefetch_count": 4, "rusage": { "idrss": 0, "inblock": 72400, "isrss": 0, "ixrss": 0, "majflt": 158, "maxrss": 68800, "minflt": 69557, "msgrcv": 0, "msgsnd": 0, "nivcsw": 2942, "nsignals": 0, "nswap": 0, "nvcsw": 334278, "oublock": 1584, "stime": 7.775837, "utime": 152.889586 }, "total": { "tasks.query.run_query": 7 } } ```

Generally, we have a lot of tasks long running tasks. This issue is making our tasks take double the amount of time because they are unnecessarily waiting behind other tasks without utilizing the other workers.

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4