Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.
After a Jira's database was copied from a production to a test environment (to create an up to date test environment), Jira wouldn't start anymore. Instead the logs showed the following informational events:
2020-09-08 07:55:49,543 JIRA-Bootstrap INFO [c.a.plugin.manager.DefaultPluginManager] Plugin system earlyStartup ended
2020-09-08 07:55:49,554 JIRA-Bootstrap INFO [c.a.jira.i18n.CachingI18nFactory] [i18n-caching-factory] Starting
2020-09-08 07:55:49,699 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/application-data/jira/dbconfig.xml
2020-09-08 07:55:49,712 JIRA-Bootstrap INFO [c.a.j.instrumentation.external.DatabaseExternalGauges] Installing DBCP monitoring instruments: DatabaseExternalGauges.JiraDbcpInstruments[instruments=[DBCP_MAX, DBCP_ACTIVE, DBCP_IDLE],objectName=com.atlassian.jira:name=BasicDataSource]
2020-09-08 07:55:49,769 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] Running JIRA startup checks.
2020-09-08 07:55:49,769 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] JIRA pre-database startup checks completed successfully.
2020-09-08 07:55:49,836 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigurationManagerImpl] The database is not yet configured. Enqueuing Database Checklist Launcher on post-database-configured-but-pre-database-activated queue
2020-09-08 07:55:49,836 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigurationManagerImpl] The database is not yet configured. Enqueuing Post database-configuration launchers on post-database-activated queue
2020-09-08 07:55:49,861 JIRA-Bootstrap INFO [c.a.jira.startup.LauncherContextListener] Memory Usage:
---------------------------------------------------------------------------------
Heap memory : Used: 777 MiB. Committed: 1591 MiB. Max: 4066 MiB
Non-heap memory : Used: 72 MiB. Committed: 88 MiB. Max: 1536 MiB
---------------------------------------------------------------------------------
TOTAL : Used: 849 MiB. Committed: 1679 MiB. Max: 5602 MiB
---------------------------------------------------------------------------------
Important hint here: The database is not yet configured. This makes no sense obviously as the database is coming right from a Jira running in production.
When browsing to the URL of this test instance, the Jira setup page would be shown:
Obviously there had been a couple of migrations happening to create an up to date test environment:
A quick look into /var/atlassian/application-data/jira revealed something very interesting:
root@jiratest:~# ll /var/atlassian/application-data/jira/
total 76
drwx------ 2 jira jira 4096 Sep 7 15:23 analytics-logs
drwx------ 6 jira jira 4096 May 22 2019 caches
drwxr-x--- 5 jira jira 4096 Sep 7 09:36 customisations
drwx------ 5 jira jira 4096 May 22 2019 data
-rw------- 1 root root 1157 Sep 7 15:21 dbconfig.xml
-rw------- 1 jira jira 1157 Mar 21 2018 dbconfig.xml.20200907
drwx------ 3 jira jira 4096 Dec 9 2019 export
drwx------ 4 jira jira 4096 Apr 26 2017 import
-rw------- 1 jira jira 93 May 22 2019 jira-config.properties
-rw------- 1 jira jira 412 Aug 21 2018 jira-healthcheck-eol.json
drwx------ 2 jira jira 16384 Sep 7 14:41 log
drwxr-x--- 2 jira jira 4096 Jan 14 2020 logos
drwx------ 3 jira jira 4096 May 2 2017 logs
drwx------ 2 jira jira 4096 Apr 26 2017 monitor
drwx------ 6 jira jira 4096 Jun 13 2018 plugins
drwx------ 3 jira jira 4096 Apr 24 2017 tmp
When dbconfig.xml was adjusted, a backup copy was made. But the ownership of dbconfig.xml now changed from jira to root. The current permissions being 0600 don't allow the application (launched as jira user) to read that file. Meaning: From the point of view of Jira, the dbconfig.xml file does not yet exist and needs to be created - hence the setup page showing up.
After an ownership change, Jira started correctly again:
root@jiratest:~# chown jira:jira /var/atlassian/application-data/jira/dbconfig.xml
root@jiratest:~# /etc/init.d/jira start