How to configure Celery using Elastic Beanstalk with Amazon Linux 2

celery_configs

# In .ebextensions/01_celery.config

files:
    "/etc/systemd/system/celery.service":
        mode: "000644"
        owner: celery
        group: celery  
        content: |
            [Unit]
            Description=Celery Service
            After=network.target

            [Service]
            # I saw some other tutorials suggesting using Type=simple, but that didn't work for me. Type=forking works
            # as long as you're using an instance with at least 2.0 Gigs of RAM, but on a t2.micro instance it was running out
            # of memory and crashing.
            Type=forking
            Restart=on-failure
            RestartSec=10
            User=celery
            Group=celery
            # You can have multiple EnvironmentFile= variables declared if you have files with variables.
            # The celery docs on daemonizing celery with systemd put their environment variables in a file called
            # /etc/conf.d/celery, but I'm choosing to instead set the celery variables as environment variables so that
            # celery can also access the necessary variables for interacting with Django.
            EnvironmentFile=/opt/elasticbeanstalk/deployment/env
            WorkingDirectory=/var/app/current
            ExecStart=/bin/sh -c '${CELERY_BIN} multi start worker \
              -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
              --logfile=${CELERYD_LOG_FILE} --loglevel=INFO --time-limit=300 --concurrency=2'
            ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait worker \
              --pidfile=${CELERYD_PID_FILE}'
            ExecReload=/bin/sh -c '${CELERY_BIN} multi restart worker \
              -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
              --logfile=${CELERYD_LOG_FILE} --loglevel=INFO --time-limit=300 --concurrency=2'

            [Install]
            WantedBy=multi-user.target
    "/etc/tmpfiles.d/celery.conf":
        mode: "000755"
        owner: celery
        group: celery
        content: |
            d /var/run/celery 0755 celery celery -
            d /var/log/celery 0755 celery celery -

container_commands:
    01_create_celery_log_file_directories:
        command: mkdir -p /var/log/celery /var/run/celery
    02_give_celery_user_ownership_of_directories:
        command: chown -R celery:celery /var/log/celery /var/run/celery
    03_change_mode_of_celery_directories:
        command: chmod -R 755 /var/log/celery /var/run/celery
    04_reload_settings:
        command: systemctl daemon-reload
        

# In .platform/hooks/postdeploy/01_start_celery.sh

#!/bin/bash

(cd /var/app/current; systemctl stop celery)
(cd /var/app/current; systemctl start celery)
(cd /var/app/current; systemctl enable celery.service)

@edchelstephens No I have not gotten beat to start on deployment. I can start it manually after SSH into the instance.

@Alex3917 sorry to bother you on this, but I tried to add a file to start beat after starting celery - and after restarting the instance I'm somehow getting the same error as typefox09 'celery is not a valid user name'. Should I add this to my 01_celery.config file?

commands:
00_add_user_celery:
test: test ! "id -u celery 2> /dev/null"
command: useradd -d /opt/python/celery -g celery -u 1502 celery
ignoreErrors: false

Is this a permissions issue somehow? Any help you can provide would be very much appreciated.

@Alex3917 I've added a 01_python.config file like you mentioned in your first response to appli-intramuros, however should the path change from /opt/python/celery to something else in AL2?

I'm getting 'Created group celery successfully' but then 'Command 00_add_user_celery (useradd -d /opt/python/celery -g celery -u 1501 celery) failed' error..

When I SSH in and try to run the command 'useradd -d /opt/python/celery -g celery -u 1501 celery'
I get '-bash: /usr/sbin/useradd: Permission denied'

Thanks in advance.

@edchelstephens I was able to get beat to start on deployment. I added another file in .ebextensions/01_celery.config; similar to "/etc/systemd/system/celery.service":, but instead called "/etc/systemd/system/celerybeat.service". This is the file (underneath celery.service file):

 "/etc/systemd/system/celerybeat.service":
        mode: "000644"
        owner: celery
        group: celery
        content: |
            [Unit]
            Description=Celery Beat Service
            After=network.target

            [Service]
            Type=forking
            Restart=on-failure
            RestartSec=10
            User=celery
            Group=celery
            EnvironmentFile=/opt/elasticbeanstalk/deployment/env
            WorkingDirectory=/var/app/current
            ExecStart=/bin/sh -c '/var/app/venv/staging-LQM1lest/bin/celery beat -A video.celery  \
            --pidfile=/tmp/celerybeat.pid \
            --logfile=/var/log/celery/celerybeat.log \
            --loglevel=INFO -s /tmp/celerybeat-schedule'

            [Install]
            WantedBy=multi-user.target

After that, I added:

(cd /var/app/current; systemctl stop celerybeat)
(cd /var/app/current; systemctl start celerybeat)
(cd /var/app/current; systemctl enable celerybeat.service)

to the 01_start_celery.sh file that had the same language for celery. Hope this helps.

Take a look at this gist. I have successfully deployed Django Rest App with celery, AWS sqs in Elastic Beanstalk with Amazon Linux 2

https://gist.github.com/MahirMahbub/f9c226dbc0a01da22c8c539cf9c9bcc9