Fixing Issue DB Open edX Notifier Celery Workers is Locked

Apart from the issue user celery does not exist after migrated or upgrade the instance, there are other possible issues that may be encountered after migrating instances, i.e. service notifier-celery-workers failed to start (FATAL, Exited too quickly).

Issue notifier-celery-workers failing to start may be encountered after migrate the instances because before migration, the service has not been turned off while the service is serving several transactional queries so the database used by the service is locked (especially if you are still using SQLite), which means that the service cannot read or write data to the database.

I have encountered this issue and when I look at the log service notifier-celery-workers, I find logs like the following:

$ sudo tail /edx/var/log/supervisor/notifier-celery-workers-std{out,err}.log
==> /edx/var/log/supervisor/notifier-celery-workers-stdout.log <==
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery


[tasks]
  . notifier.tasks.do_forums_digests
  . notifier.tasks.generate_and_send_digests


==> /edx/var/log/supervisor/notifier-celery-workers-stderr.log <==
    return self.cursor.execute(sql, params)
  File "/edx/app/notifier/virtualenvs/notifier/local/lib/python2.7/site-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/edx/app/notifier/virtualenvs/notifier/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/edx/app/notifier/virtualenvs/notifier/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 318, in execute
    return Database.Cursor.execute(self, query, params)
OperationalError: database is locked

The issue was successfully fixed with steps:

1. From the current user on the server, change to user notifier.

$ sudo -H -u notifier bash

2. Change to the database notifier directory and back up then delete the currently used database.

(notifier) $ cd ~/db
(notifier) $ cp notifier.db notifier.db.bak
(notifier) $ rm -f notifier.db

3. Run the database migration, so that a new database will be created.

(notifier) $ cd ~/src
(notifier) $ python manage.py migrate

4. Go back to the database directory, change the file permissions so that the service can write data and return to the user that was previously used (user non edX).

(notifier) $ cd ~/db
(notifier) $ chmod 664 notifier.db
(notifier) $ exit

5. Restart all services of Open edX.

$ sudo /edx/bin/supervisorctl restart all

Check the service status periodically and pay attention to the log service, make sure the service is run or not error.