How to handle a tech security incident



One of my favourite apps had a security incident this week. Buffer is a social media management tool that allows you to schedule posts into a “buffer” that posts on a predetermined schedule so you aren’t bombarding people with all your content at once. Its a great app and I’ve been using it for over a year.

One of the things that differentiates the buffer experience from competing products is that their management team are AMAZING communicators. They haven’t lost touch with who they are as things have grown and this weeks experience is no exception. Buffers handling of what could be a fatal experience for a startup is an awesome example of why they are going to be very very successful in the future. Continue reading “How to handle a tech security incident”

Project Titanicarus: Part 9 – Building the Email Servers

You've got mail

Email servers.. the bane of every sysadmins existence. The second something goes wrong with an email server, you’re guaranteed to get 100 phone calls and people dropping by your office to say “My emails aren’t working”. This is one part of your hosting infrastructure you want to get right.

I’ve decided to build my infrastructure on Postfix & Dovecot with a MySQL user database. My previous email setup was built using this howto. One of the major issues I ran into was with Courier’s inability to handle large mailboxes so I’ve decided to use a similar setup only with Dovecot in place of Courier and there are a couple of other major differences:

  1. This is going to be a highly distributed configuration (ie multiple servers in multiple datacentres)
  2. This is going to sit behind load balancers (brings interesting spam filtering and security issues)
  3. This is going to use a clustered MySQL backend

So the goal of todays blog post is to deliver:

  • Multi-server & multi-datacentre replicated mail stores
  • Fault tolerance (pull a server out at any time of the day and mail keeps flowing)
  • POP3 & IMAP user access
  • Authenticated SMTP Submission

Continue reading “Project Titanicarus: Part 9 – Building the Email Servers”

Project Titanicarus: How to configure SFTP and FTP with ProFTPD

FTP Server

I personally hate & never use FTP, but some people prefer/need it for their development tools to work. Today we’re going to install ProFTPd on our servers using MySQL based virtual users. The following instructions are adapted from this really good howto, if I’ve missed something you may want to check the original version which I’ve recreated here just in case the other one goes away.
Continue reading “Project Titanicarus: How to configure SFTP and FTP with ProFTPD”

Project Titanicarus: Part 7 – Building the Web Servers

Web Servers

I am building two app boxes per site. They will host mail, web and DNS for all applications I’m hosting. If I was building a larger implementation I’d separate those tasks out but scale doesn’t justify it just yet.

This week we’re going to take one of the app servers we built previously and install the web server components. I am using NGINX compiled from source as I want to include a plugin called Google PageSpeed that helps make things very quick.

Continue reading “Project Titanicarus: Part 7 – Building the Web Servers”

Project Titanicarus: Part 6 – Building the MySQL Cluster


Before I’d dealt with the filers, I had written this weeks tasks up as being the most difficult part of the project.

I have a bunch of experience working with standard MySQL servers using replication, but I’ve never played with MySQL Cluster server before. Learning how to make it work was made difficult by a lack of packages in the Ubuntu repositories, I also struggled to find documentation that was simple enough to understand and complete the task without having to fill in blanks that were left by those documenting their learnings.

I’ve decided to write up the process I used to build a two node MySQL Cluster, hopefully I can fill in the gaps for others trying to make this kind of project happen for themselves. I’m building one cluster per island on a pair of servers. Inter-island replication is something I’m going to have to experiment with as the MySQL cluster docco seems to say that it gets cranky when asked to replicate over the internet.

Continue reading “Project Titanicarus: Part 6 – Building the MySQL Cluster”

Project Titanicarus: Part 5 – Building the Filers or “Welcome to the Pit of Despair”

This part of the project is the one I have the least experience with and the one which I’ve spent the most time trying to find a solution that works the way that I need.

To put it bluntly I don’t know if a solution exists that is capable of doing what I want with the level of simplicity I want. Almost every solution I have found has its own unique set of shortcomings, almost all of those are performance or complexity related.

I have been through several levels of insanity trying to get a viable solution implemented, including a momentary period of complete lunacy in which I planned to write my own solution.

Lets look at what I am looking for in a backend filesystem:

  • Multi-chassis striping (for performance & redundancy)
  • Self healing in the event of failure without admin intervention
  • Able to scale up by adding more storage servers
  • It must perform well with lots of small files
  • It must be fast enough that web applications don’t lag
  • Replication over WAN to multiple datacentres
  • Capable of continuing to function when partitioned (WAN down)
  • POSIX style locking (not mandatory, but ideal)

Continue reading “Project Titanicarus: Part 5 – Building the Filers or “Welcome to the Pit of Despair””

Project Titanicarus: Part 4 – Building Load Balancers

Load Balancer

Zen Load Balancer

I am using Zen Load Balancer as the front end to all of the infrastructure in this project. It is simple, robust, provides a nice air gap between application servers and the big bad world and it also doubles as a choke point that we can apply security policies and traffic monitoring when required.

I chose Zen over better known load balancers as it can handle TCP and UDP and it will also do SSL offloading, freeing up some CPU on the app servers if we need it later on.

Zen Load Balancer will sit in front of every service we allow to be seen by the outside world – HTTP, HTTPS, DNS, SMTP, POP, IMAP. Continue reading “Project Titanicarus: Part 4 – Building Load Balancers”

Project Titanicarus: Part 3 – Building the PFSense Firewall


PFSense is a FreeBSD firewall distro that is primarily focussed on delivering a very simple and secure firewall solution. I am using it because I’ve used it before in production environments and it proved to be a really reliable workhorse. Installation is incredibly simple, I followed the installation guide on their wiki here. I have set the box up with 2 network interfaces, one for internet access and one for internal network access. Continue reading “Project Titanicarus: Part 3 – Building the PFSense Firewall”

Last updated by at .