Turns out the problem I was having with everything except the front page was due to permalink rewrites not working. It seems three things were incorrectly setup.

First, the WordPress folder wasn’t writable by the www-user.

Secondly, this needed to be done:

You have to edit the Apache conf file (in Ubuntu 14.04: /etc/apache2/apache2.conf) and change the AllowOverride setting from None to FileInfo

<Directory /var/www/>
Options Indexes FollowSymLinks
AllowOverride FileInfo
Require all granted
</Directory>

You may need to enable the Rewrite module:

sudo a2enmod rewrite

– Brian Milnes on StackOverflow