Project Titanicarus: NGINX, SPDY and PageSpeed = FAST!

Web ServersSince I posted the original post on building NGINX servers with Google PageSpeed there have been a bunch of additional shiny things to add to your NGINX build, the biggest and shiniest one being SPDY. I’m building a new development environment at the moment, so I’ve taken the opportunity to update this post with the additional instructions for including SPDY. I’ve written a script that installs most of the major stuff. The script is pretty well commented and explains what its doing throughout. To use this script, put it into a file called install.sh and run “sh install.sh” at the command line.

Web Server Install Script

# Update apt
apt-get update 
# Install VMware Tools
apt-get -y install open-vm-tools
# Install NTP 
apt-get -y install ntp 
# OS Upgrades 
# Upgrade ulimits 
echo "*  soft  no file 9000" >> /etc/security/limits.conf  
echo "* hard  no file 65000" >> /etc/security/limits.conf  
echo "session required pam_limits.so" >> /etc/pam.d/common-session 
# Make /tmp nice & fast
echo "tmpfs /tmp tmpfs defaults,noexec,nosuid 0 0" >> /etc/fstab 
mount -a 
# Install packages required for NGINX install 
apt-get -y install build-essential zlib1g-dev libpcre3 libpcre3-dev unzip libssl-dev libxslt1-dev libgd2-xpm-dev libgeoip-dev libperl-dev postfix unzip 
# Prepare ngx_pagespeed - check for updated versions at https://developers.google.com/speed/pagespeed/module/build_ngx_pagespeed_from_source
cd /usr/src/
NPS_VERSION=1.8.31.4
wget https://github.com/pagespeed/ngx_pagespeed/archive/release-${NPS_VERSION}-beta.zip
unzip release-${NPS_VERSION}-beta.zip
cd ngx_pagespeed-release-${NPS_VERSION}-beta/
wget https://dl.google.com/dl/page-speed/psol/${NPS_VERSION}.tar.gz
tar -xzvf ${NPS_VERSION}.tar.gz  # extracts to psol/
cd /usr/src/
# check http://nginx.org/en/download.html for the latest version
NGINX_VERSION=1.7.3
wget http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz
tar -xvzf nginx-${NGINX_VERSION}.tar.gz
cd nginx-${NGINX_VERSION}
# Configure NGINX for local environment 
./configure --add-module=/usr/src/ngx_pagespeed-release-${NPS_VERSION}-beta --sbin-path=/usr/sbin --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-http_dav_module --with-http_gzip_static_module --with-http_realip_module --with-http_ssl_module --with-ipv6 --with-http_spdy_module 

# Compile NGINX 
make 
# Install NGINX 
make install 

mkdir /var/lib/nginx/ 
mkdir /var/lib/nginx/body 
# Make tmpfs mount for pagespeed data 
echo "tmpfs /var/cache/pagespeed tmpfs size=256m,mode=0775,uid=www-data,gid=www-data 0 0" >> /etc/fstab 
mkdir /var/cache/pagespeed 
chown www-data:www-data /var/cache/pagespeed 
mount /var/cache/pagespeed 

# Create init script 
cd /etc/init.d/ 
wget http://www.hooton.org/downloads/scripts/nginx.txt -Onginx 
chmod +x nginx 
update-rc.d nginx defaults 

# Install php-fpm 
apt-get install -y php5 php5-xmlrpc php5-mysql php5-mcrypt php5-intl php5-gd php5-dev php5-curl php5-common php5-cli php5-cgi php-pear php5-mysql php-apc php5-fpm php5-imap php5-memcache php5-memcached libssh2-php php5-tidy php5-json 

# Install varnish cache & memcached 
apt-get -y install varnish memcached 

# Make varnish faster 
echo "tmpfs /var/lib/varnish tmpfs size=256m,mode=0775,uid=root,gid=root 0 0" >> /etc/fstab 

# Copy default configs to BTSYNC share (Skip if this is not the first server) 
cd /data/config/ 
wget http://www.hooton.org/downloads/configs/appserverconfig.tgz 
tar -zxvf appserverconfig.tgz 

# Create symlinks for configs 
cd /etc 
rm -rf nginx
rm -rf php5 
rm -rf varnish
ln -s /data/config/etc/nginx/ 
ln -s /data/config/etc/varnish/ 
ln -s /data/config/etc/php5/ 
cd /etc/default 
rm -rf varnish 
rm -rf memcached 
ln -s /data/config/etc/default/varnish 
ln -s /data/config/etc/default/memcached 
cd /etc/ 
rm -rf memcached.conf 
ln -s /data/config/etc/memcached.conf

Once you’ve finished that script, reboot the server and it should come up running a default install of everything.

Kernel Tuning

These are a few kernel adjustments I’ve found to be useful on web servers, your mileage may vary.

net.ipv4.ip_local_port_range = 2000 65000
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_max_syn_backlog = 3240000
net.core.somaxconn = 3240000
net.ipv4.tcp_max_tw_buckets = 1440000
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_congestion_control = cubic

To apply the changes

sysctl -p /etc/sysctl.conf

Thats it!

You’ve now got a working NGINX, PHP-FPM, Google PageSpeed & SPDY web server running. You’ll probably want to create a couple of virtual hosts, a template for a virtual host with pagespeed enabled can be found here. To create a new virtual host, modify the template file for your site and add it to /data/config/etc/nginx/sites-available/ then symlink to it in /data/config/etc/nginx/sites-enabled/ and restart nginx.

Last updated by at .