Postgres won't start

Today, after shutting down Postgres, to add memory to the machine, it refused to start.

Notes like:

Mar 2 03:50:31 pgsql postgres[1234]: [3-1]: FATAL: the database system is starting up

scrolled by quickly and no apparent progress was made. Eventually, cntl-c stopped things. Inspecting the log showed nothing (literally, empty log file)

Single user mode

After a bit of googling (mostly useless, no logs to narrow down anything), I went back to reading the man page for postgres. It offered a single user mode of operation (similar in concept to the single user mode that UNIX offers). I ended up using the following options:

–single: single user mode

-D: path to the data directory

-P: disable system indexes

-d 1: set debug output to 1

$ /usr/local/bin/postgres --single -D /usr/local/pgsql/data -P -d 1
LOG:  database system was shut down in recovery at 2017-03-02 03:53:05 UTC
DEBUG:  checkpoint record is at 1A5/EEA5ED0
DEBUG:  redo record is at 1A4/FB0DC7F8; shutdown FALSE
DEBUG:  next transaction ID: 0:69624573; next OID: 191021
DEBUG:  next MultiXactId: 17; next MultiXactOffset: 33
DEBUG:  oldest unfrozen transaction ID: 853, in database 12716
DEBUG:  oldest MultiXactId: 1, in database 27036
DEBUG:  commit timestamp Xid oldest/newest: 0/0
DEBUG:  transaction ID wrap limit is 2147484500, limited by database with OID 12716
DEBUG:  MultiXactId wrap limit is 2147483648, limited by database with OID 27036
DEBUG:  starting up replication slots
LOG:  database system was not properly shut down; automatic recovery in progress
DEBUG:  resetting unlogged relations: cleanup 1 init 0
LOG:  redo starts at 1A4/FB0DC7F8
LOG:  invalid record length at 1A5/23DF9F30: wanted 24, got 0
LOG:  redo done at 1A5/23DF9F08
LOG:  last completed transaction was at log time 2017-03-02 03:26:45.090211+00
DEBUG:  resetting unlogged relations: cleanup 0 init 1
LOG:  checkpoint starting: end-of-recovery immediate
DEBUG:  performing replication slot checkpoint
DEBUG:  checkpoint sync: number=1 file=base/16762/77539 time=0.030 msec
DEBUG:  checkpoint sync: number=2 file=base/16762/77902 time=0.024 msec
DEBUG:  checkpoint sync: number=3 file=base/16762/77569 time=0.020 msec
DEBUG:  checkpoint sync: number=4 file=base/16762/77877 time=0.021 msec
DEBUG:  checkpoint sync: number=5 file=base/82382/71522 time=0.024 msec
DEBUG:  checkpoint sync: number=6 file=base/82382/71522_fsm time=0.047 msec
DEBUG:  checkpoint sync: number=7 file=base/16385/69196 time=0.039 msec
DEBUG:  checkpoint sync: number=8 file=base/16762/77878 time=0.022 msec
DEBUG:  checkpoint sync: number=9 file=base/82382/82468 time=0.020 msec
DEBUG:  checkpoint sync: number=10 file=base/133179/71525 time=0.023 msec
DEBUG:  checkpoint sync: number=11 file=base/133179/71525_fsm time=0.022 msec
DEBUG:  checkpoint sync: number=12 file=base/133179/71525_vm time=0.020 msec
DEBUG:  checkpoint sync: number=12 file=base/133179/71525_vm time=0.020 msec
DEBUG:  checkpoint sync: number=13 file=base/82382/83786 time=0.006 msec
DEBUG:  checkpoint sync: number=14 file=base/16762/77444 time=0.020 msec
DEBUG:  checkpoint sync: number=15 file=base/16762/77370 time=0.019 msec
DEBUG:  checkpoint sync: number=16 file=base/16762/77370_vm time=0.019 msec
DEBUG:  checkpoint sync: number=17 file=base/16762/77527 time=0.019 msec
DEBUG:  checkpoint sync: number=18 file=base/16762/77527_fsm time=0.020 msec
DEBUG:  checkpoint sync: number=19 file=base/16762/77527_vm time=0.020 msec
DEBUG:  checkpoint sync: number=20 file=base/82382/83536 time=0.020 msec
DEBUG:  checkpoint sync: number=21 file=base/16762/77873 time=0.020 msec
DEBUG:  checkpoint sync: number=22 file=base/16762/77420 time=0.019 msec
DEBUG:  checkpoint sync: number=23 file=base/16762/77420_fsm time=0.019 msec
DEBUG:  checkpoint sync: number=24 file=base/16762/77420_vm time=0.019 msec
DEBUG:  checkpoint sync: number=25 file=base/27036/68257 time=0.020 msec
DEBUG:  checkpoint sync: number=26 file=base/27036/68257_fsm time=0.019 msec
DEBUG:  checkpoint sync: number=27 file=base/27036/68257_vm time=0.019 msec
DEBUG:  checkpoint sync: number=28 file=base/27036/68269 time=0.019 msec
DEBUG:  checkpoint sync: number=29 file=base/27036/68269_fsm time=0.018 msec
DEBUG:  checkpoint sync: number=30 file=base/27036/68269_vm time=0.018 msec
DEBUG:  checkpoint sync: number=31 file=base/82382/83458 time=0.020 msec
DEBUG:  checkpoint sync: number=32 file=base/16762/69411 time=0.019 msec
DEBUG:  checkpoint sync: number=33 file=base/16762/69411_fsm time=0.018 msec
DEBUG:  checkpoint sync: number=34 file=base/16762/69411_vm time=0.018 msec
DEBUG:  checkpoint sync: number=35 file=base/82382/83803 time=0.018 msec
DEBUG:  checkpoint sync: number=36 file=base/82382/83803_fsm time=0.019 msec
DEBUG:  checkpoint sync: number=37 file=base/16762/77566 time=0.019 msec
DEBUG:  checkpoint sync: number=38 file=base/16762/77471 time=0.020 msec
DEBUG:  checkpoint sync: number=39 file=base/16762/77471_vm time=0.019 msec
DEBUG:  checkpoint sync: number=40 file=base/16762/77875 time=0.019 msec
DEBUG:  checkpoint sync: number=41 file=base/82382/82465 time=0.018 msec
DEBUG:  checkpoint sync: number=42 file=base/16762/77860 time=0.019 msec
DEBUG:  checkpoint sync: number=43 file=base/16762/77860_fsm time=0.019 msec
DEBUG:  checkpoint sync: number=44 file=base/16762/77860_vm time=0.019 msec
DEBUG:  checkpoint sync: number=45 file=base/16762/77418 time=0.018 msec
DEBUG:  checkpoint sync: number=46 file=base/82382/83482 time=0.019 msec
DEBUG:  checkpoint sync: number=47 file=base/82382/83823 time=0.018 msec
DEBUG:  checkpoint sync: number=48 file=base/16762/69547 time=0.018 msec
DEBUG:  checkpoint sync: number=49 file=base/16762/77447 time=0.020 msec
DEBUG:  checkpoint sync: number=50 file=base/82382/83558 time=0.018 msec
DEBUG:  checkpoint sync: number=51 file=base/27036/186065 time=289.835 msec
DEBUG:  checkpoint sync: number=52 file=base/27036/68623 time=0.026 msec
DEBUG:  checkpoint sync: number=53 file=base/16762/77541 time=0.020 msec
DEBUG:  checkpoint sync: number=54 file=base/82382/82467 time=0.022 msec
DEBUG:  checkpoint sync: number=55 file=base/82382/82463 time=0.019 msec
DEBUG:  checkpoint sync: number=56 file=base/16762/77876 time=0.019 msec
DEBUG:  checkpoint sync: number=57 file=base/82382/82689 time=0.020 msec
DEBUG:  checkpoint sync: number=58 file=base/82382/82689_fsm time=0.019 msec
DEBUG:  checkpoint sync: number=59 file=base/82382/83901 time=0.018 msec
DEBUG:  checkpoint sync: number=60 file=base/133179/71522 time=0.019 msec
DEBUG:  checkpoint sync: number=61 file=base/133179/71522_fsm time=0.020 msec
DEBUG:  checkpoint sync: number=62 file=base/133179/71522_vm time=0.018 msec
DEBUG:  checkpoint sync: number=63 file=base/16762/77903 time=0.018 msec
DEBUG:  checkpoint sync: number=64 file=base/16762/77635 time=0.019 msec
DEBUG:  checkpoint sync: number=65 file=base/16762/77635_fsm time=0.018 msec
DEBUG:  checkpoint sync: number=66 file=base/16762/77635_vm time=0.019 msec
DEBUG:  checkpoint sync: number=67 file=base/82382/82590 time=0.019 msec
DEBUG:  checkpoint sync: number=68 file=base/82382/83438 time=0.019 msec
DEBUG:  checkpoint sync: number=69 file=base/82382/83438_fsm time=0.019 msec
DEBUG:  checkpoint sync: number=70 file=base/82382/83438_vm time=0.019 msec
DEBUG:  checkpoint sync: number=71 file=base/82382/83538 time=0.018 msec
DEBUG:  checkpoint sync: number=72 file=base/82382/83789 time=0.018 msec
DEBUG:  checkpoint sync: number=73 file=base/82382/83825 time=0.018 msec
DEBUG:  checkpoint sync: number=74 file=base/16762/77446 time=0.019 msec
DEBUG:  checkpoint sync: number=75 file=base/82382/83452 time=0.018 msec
DEBUG:  checkpoint sync: number=76 file=base/82382/83452_fsm time=0.018 msec
DEBUG:  checkpoint sync: number=77 file=base/16762/77880 time=0.025 msec
DEBUG:  checkpoint sync: number=78 file=base/16762/77880_fsm time=0.018 msec
DEBUG:  checkpoint sync: number=79 file=base/16762/77880_vm time=0.020 msec
DEBUG:  checkpoint sync: number=80 file=base/16385/68648 time=0.019 msec
DEBUG:  checkpoint sync: number=81 file=base/16385/68648_fsm time=0.019 msec
DEBUG:  checkpoint sync: number=82 file=base/16385/68648_vm time=0.019 msec
DEBUG:  checkpoint sync: number=83 file=base/82382/83667 time=0.018 msec
DEBUG:  checkpoint sync: number=84 file=base/27036/68247 time=0.018 msec
DEBUG:  checkpoint sync: number=85 file=base/27036/68247_vm time=0.019 msec
DEBUG:  checkpoint sync: number=86 file=base/16762/69408 time=0.019 msec
DEBUG:  checkpoint sync: number=87 file=base/16762/69408_fsm time=0.019 msec
DEBUG:  checkpoint sync: number=88 file=base/16762/69408_vm time=0.018 msec
DEBUG:  checkpoint sync: number=89 file=base/82382/83526 time=0.019 msec
DEBUG:  checkpoint sync: number=90 file=base/82382/83526_fsm time=0.018 msec
DEBUG:  checkpoint sync: number=91 file=base/82382/83548 time=0.018 msec
DEBUG:  checkpoint sync: number=92 file=base/82382/83669 time=0.019 msec
DEBUG:  checkpoint sync: number=93 file=base/16762/77568 time=0.018 msec
DEBUG:  checkpoint sync: number=94 file=base/82382/82466 time=0.019 msec
DEBUG:  checkpoint sync: number=95 file=base/16762/77552 time=0.019 msec
DEBUG:  checkpoint sync: number=96 file=base/16762/77552_fsm time=0.018 msec
DEBUG:  checkpoint sync: number=97 file=base/16762/77552_vm time=0.018 msec
DEBUG:  checkpoint sync: number=98 file=base/16762/77633 time=0.018 msec
DEBUG:  checkpoint sync: number=99 file=base/27036/68262 time=0.019 msec
DEBUG:  checkpoint sync: number=100 file=base/16762/77641 time=0.018 msec
DEBUG:  checkpoint sync: number=101 file=base/16762/77899 time=0.019 msec
DEBUG:  checkpoint sync: number=102 file=base/82382/83459 time=0.019 msec
DEBUG:  checkpoint sync: number=103 file=base/27036/68263 time=0.019 msec
DEBUG:  checkpoint sync: number=104 file=base/16762/77550 time=0.019 msec
DEBUG:  checkpoint sync: number=104 file=base/16762/77550 time=0.019 msec
DEBUG:  checkpoint sync: number=105 file=base/82382/83472 time=0.019 msec
DEBUG:  checkpoint sync: number=106 file=base/82382/83788 time=0.018 msec
DEBUG:  checkpoint sync: number=107 file=base/82382/71528 time=0.020 msec
DEBUG:  checkpoint sync: number=108 file=base/133179/71646 time=0.018 msec
DEBUG:  checkpoint sync: number=109 file=base/16385/68645 time=0.018 msec
DEBUG:  checkpoint sync: number=110 file=base/16385/68645_fsm time=0.026 msec
DEBUG:  checkpoint sync: number=111 file=base/16385/68645_vm time=0.019 msec
DEBUG:  checkpoint sync: number=112 file=base/133179/71527 time=0.018 msec
DEBUG:  checkpoint sync: number=113 file=base/82382/82575 time=0.019 msec
DEBUG:  checkpoint sync: number=114 file=base/16762/77901 time=0.018 msec
DEBUG:  checkpoint sync: number=115 file=base/27036/68625 time=0.020 msec
DEBUG:  checkpoint sync: number=116 file=base/133179/71528 time=0.020 msec
DEBUG:  checkpoint sync: number=117 file=base/27036/186066 time=0.907 msec
DEBUG:  checkpoint sync: number=118 file=base/82382/83445.1 time=0.022 msec
DEBUG:  checkpoint sync: number=119 file=base/82382/83445.2 time=0.019 msec
DEBUG:  checkpoint sync: number=120 file=base/82382/83445_fsm time=0.021 msec
DEBUG:  checkpoint sync: number=121 file=base/16385/68650 time=0.021 msec
DEBUG:  checkpoint sync: number=122 file=base/82382/83473 time=0.018 msec
DEBUG:  checkpoint sync: number=123 file=base/82382/83473_fsm time=0.109 msec
DEBUG:  checkpoint sync: number=124 file=base/27036/68624 time=0.019 msec
DEBUG:  checkpoint sync: number=125 file=base/82382/71646 time=0.021 msec
DEBUG:  checkpoint sync: number=126 file=base/82382/83682 time=0.018 msec
DEBUG:  checkpoint sync: number=127 file=base/16762/77858 time=0.019 msec
DEBUG:  checkpoint sync: number=128 file=base/27036/68620 time=0.019 msec
DEBUG:  checkpoint sync: number=129 file=base/27036/68620_vm time=0.019 msec
DEBUG:  checkpoint sync: number=130 file=base/16385/68657 time=0.019 msec
DEBUG:  checkpoint sync: number=131 file=base/16385/68657_vm time=0.019 msec
DEBUG:  checkpoint sync: number=132 file=base/16762/77442 time=0.018 msec
DEBUG:  checkpoint sync: number=133 file=base/16762/69413 time=0.018 msec
DEBUG:  checkpoint sync: number=134 file=base/27036/68511 time=0.025 msec
DEBUG:  checkpoint sync: number=135 file=base/27036/68511_vm time=0.019 msec
DEBUG:  checkpoint sync: number=136 file=base/82382/83607 time=0.028 msec
DEBUG:  checkpoint sync: number=137 file=base/82382/83607_vm time=0.018 msec
DEBUG:  checkpoint sync: number=138 file=base/82382/83451 time=0.019 msec
DEBUG:  checkpoint sync: number=139 file=base/82382/83451.1 time=0.018 msec
DEBUG:  checkpoint sync: number=140 file=base/82382/83451.2 time=0.018 msec
DEBUG:  checkpoint sync: number=141 file=base/82382/83451.3 time=0.019 msec
DEBUG:  checkpoint sync: number=142 file=base/16762/77643 time=0.019 msec
DEBUG:  checkpoint sync: number=143 file=base/82382/83539 time=0.017 msec
DEBUG:  checkpoint sync: number=144 file=base/16762/77448 time=0.019 msec
DEBUG:  checkpoint sync: number=145 file=base/16762/77644 time=0.018 msec
DEBUG:  checkpoint sync: number=146 file=base/82382/82383 time=0.019 msec
DEBUG:  checkpoint sync: number=147 file=base/82382/83550 time=0.018 msec
DEBUG:  checkpoint sync: number=148 file=base/82382/83550_fsm time=0.018 msec
DEBUG:  checkpoint sync: number=149 file=base/27036/186067 time=1.065 msec
DEBUG:  checkpoint sync: number=150 file=base/82382/83444 time=0.021 msec
DEBUG:  checkpoint sync: number=151 file=base/82382/83540 time=0.019 msec
DEBUG:  checkpoint sync: number=152 file=base/82382/83540_fsm time=0.019 msec
DEBUG:  checkpoint sync: number=153 file=base/27036/68260 time=0.022 msec
DEBUG:  checkpoint sync: number=154 file=base/27036/68260_fsm time=0.019 msec
DEBUG:  checkpoint sync: number=155 file=base/27036/68260_vm time=0.019 msec
DEBUG:  checkpoint sync: number=156 file=base/16762/77900 time=0.019 msec
DEBUG:  checkpoint sync: number=157 file=base/16762/77897 time=0.019 msec
DEBUG:  checkpoint sync: number=158 file=base/16762/69414 time=0.018 msec
DEBUG:  checkpoint sync: number=159 file=base/82382/83661 time=0.018 msec
DEBUG:  checkpoint sync: number=160 file=base/82382/83661_fsm time=0.019 msec
DEBUG:  checkpoint sync: number=161 file=base/82382/83661_vm time=0.018 msec
DEBUG:  checkpoint sync: number=162 file=base/82382/82469 time=0.019 msec
DEBUG:  checkpoint sync: number=163 file=base/16762/77445 time=0.019 msec
DEBUG:  checkpoint sync: number=164 file=base/82382/83780 time=0.018 msec
DEBUG:  checkpoint sync: number=165 file=base/82382/83780_fsm time=0.019 msec
DEBUG:  checkpoint sync: number=166 file=base/82382/83780_vm time=0.018 msec
DEBUG:  checkpoint sync: number=167 file=base/27036/68514 time=0.019 msec
DEBUG:  checkpoint sync: number=168 file=base/27036/68514_fsm time=0.018 msec
DEBUG:  checkpoint sync: number=169 file=base/27036/68514_vm time=0.019 msec
DEBUG:  checkpoint sync: number=170 file=base/82382/83817 time=0.018 msec
DEBUG:  checkpoint sync: number=171 file=base/82382/83817_fsm time=0.028 msec
DEBUG:  checkpoint sync: number=172 file=base/82382/82422 time=0.018 msec
DEBUG:  checkpoint sync: number=173 file=base/82382/82422_fsm time=0.018 msec
LOG:  checkpoint complete: wrote 95206 buffers (18.2%); 0 transaction log file(s) added, 0 removed, 40 recycled; write=523.261 s, sync=0.314 s, total=524.054 s; sync files=173, lo
ngest=0.289 s, average=0.001 s; distance=668789 kB, estimate=668789 kB
DEBUG:  MultiXactId wrap limit is 2147483648, limited by database with OID 27036
DEBUG:  oldest MultiXactId member is at offset 1
DEBUG:  MultiXact member stop limit is now 4294914944 based on MultiXact 1
FATAL:  database "pgsql" does not exist
NOTICE:  shutting down
LOG:  checkpoint starting: shutdown immediate
DEBUG:  performing replication slot checkpoint
LOG:  checkpoint complete: wrote 0 buffers (0.0%); 0 transaction log file(s) added, 0 removed, 40 recycled; write=0.015 s, sync=0.000 s, total=0.068 s; sync files=0, longest=0.000
 s, average=0.000 s; distance=0 kB, estimate=601910 kB
NOTICE:  database system is shut down
$

After this completed gracefully, starting the database with the normal process worked again.

For some reason, even though I used the intended process to shut things down, something got stuck/fell over w/o logging something, and then decided not to recover on startup.