May 7, 2021

Fixing Issue DB Open edX Notifier Celery Workers is Locked

Activities, Android Developer, Ansible, Apache2, Atlassian, Ayo Belajar Linux, Bestpath Network, BLC Telkom Klaten, BSD, Caddy Server, Case Study, Cisco, Cisco Indonesia, Cloud Computing, Cockpit, Custom Weapons, Docker, E-Learning, Engenius, Error, FreeBSD, FreeBSD Indonesia, Komunitas Pengguna Linux Indonesia, KPLI Bulukumba, KPLI Klaten, Lets Encrypt, Linux, MacOS, Microsoft Azure, Microsoft SQL Server, MikroTik, MikroTik Indonesia, MySQL, Nginx, Open edX, OpenSID, Others, PHP, phpMyAdmin, PostgreSQL, Proxmox, Python, Redash, Sendy, SSH, Stories, Subnetting, TP-Link, Ubiquiti, Unix, Virtualization, Windows, X-Mosque, Faizar Septiawan, Icar, siBunglonGanteng, Orang Ganteng, siBunglonLabs, Programmer, SysAdmin, Site Reliability Engineer, Developer, Palugada, Makassar, Ganteng, Gila, Cyclist, Panglima, Setan, Panglima Setan, sibunglon, Ganteng

Apart from the issue user celery does not exist after migrated or upgrade the instance, there are other possible issues that may be encountered after migrating instances, i.e. service notifier-celery-workers failed to start (FATAL, Exited too quickly).

Issue notifier-celery-workers failing to start may be encountered after migrate the instances because before migration, the service has not been turned off while the service is serving several transactional queries so the database used by the service is locked (especially if you are still using SQLite), which means that the service cannot read or write data to the database.

I have encountered this issue and when I look at the log service notifier-celery-workers, I find logs like the following:

$ sudo tail /edx/var/log/supervisor/notifier-celery-workers-std{out,err}.log
==> /edx/var/log/supervisor/notifier-celery-workers-stdout.log <==
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery


[tasks]
  . notifier.tasks.do_forums_digests
  . notifier.tasks.generate_and_send_digests


==> /edx/var/log/supervisor/notifier-celery-workers-stderr.log <==
    return self.cursor.execute(sql, params)
  File "/edx/app/notifier/virtualenvs/notifier/local/lib/python2.7/site-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/edx/app/notifier/virtualenvs/notifier/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/edx/app/notifier/virtualenvs/notifier/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 318, in execute
    return Database.Cursor.execute(self, query, params)
OperationalError: database is locked

The issue was successfully fixed with steps:

  1. From the current user on the server, change to user notifier.
$ sudo -H -u notifier bash
  1. Change to the database notifier directory and back up then delete the currently used database.
(notifier) $ cd ~/db
(notifier) $ cp notifier.db notifier.db.bak
(notifier) $ rm -f notifier.db
  1. Run the database migration, so that a new database will be created.
(notifier) $ cd ~/src
(notifier) $ python manage.py migrate
  1. Go back to the database directory, change the file permissions so that the service can write data and return to the user that was previously used (user non edX).
(notifier) $ cd ~/db
(notifier) $ chmod 664 notifier.db
(notifier) $ exit
  1. Restart all services of Open edX.
$ sudo /edx/bin/supervisorctl restart all

Check the service status periodically and pay attention to the log service, make sure the service is run or not error.