One of my favourite apps had a security incident this week. Buffer is a social media management tool that allows you to schedule posts into a “buffer” that posts on a predetermined schedule so you aren’t bombarding people with all your content at once. Its a great app and I’ve been using it for over a year.
One of the things that differentiates the buffer experience from competing products is that their management team are AMAZING communicators. They haven’t lost touch with who they are as things have grown and this weeks experience is no exception. Buffers handling of what could be a fatal experience for a startup is an awesome example of why they are going to be very very successful in the future. Continue reading “How to handle a tech security incident”
Email servers.. the bane of every sysadmins existence. The second something goes wrong with an email server, you’re guaranteed to get 100 phone calls and people dropping by your office to say “My emails aren’t working”. This is one part of your hosting infrastructure you want to get right.
I’ve decided to build my infrastructure on Postfix & Dovecot with a MySQL user database. My previous email setup was built using this howto. One of the major issues I ran into was with Courier’s inability to handle large mailboxes so I’ve decided to use a similar setup only with Dovecot in place of Courier and there are a couple of other major differences:
- This is going to be a highly distributed configuration (ie multiple servers in multiple datacentres)
- This is going to sit behind load balancers (brings interesting spam filtering and security issues)
- This is going to use a clustered MySQL backend
So the goal of todays blog post is to deliver:
- Multi-server & multi-datacentre replicated mail stores
- Fault tolerance (pull a server out at any time of the day and mail keeps flowing)
- POP3 & IMAP user access
- Authenticated SMTP Submission
Continue reading “Project Titanicarus: Part 9 – Building the Email Servers”
I personally hate & never use FTP, but some people prefer/need it for their development tools to work. Today we’re going to install ProFTPd on our servers using MySQL based virtual users. The following instructions are adapted from this really good howto, if I’ve missed something you may want to check the original version which I’ve recreated here just in case the other one goes away.
Continue reading “Project Titanicarus: Howto configure ProFTPD for FTP and SFTP”
I am building two app boxes per site. They will host mail, web and DNS for all applications I’m hosting. If I was building a larger implementation I’d separate those tasks out but scale doesn’t justify it just yet.
This week we’re going to take one of the app servers we built previously and install the web server components. I am using NGINX compiled from source as I want to include a plugin called Google PageSpeed that helps make things very quick.
Continue reading “Project Titanicarus: Part 7 – Building the Web Servers”
Before I’d dealt with the filers, I had written this weeks tasks up as being the most difficult part of the project.
I have a bunch of experience working with standard MySQL servers using replication, but I’ve never played with MySQL Cluster server before. Learning how to make it work was made difficult by a lack of packages in the Ubuntu repositories, I also struggled to find documentation that was simple enough to understand and complete the task without having to fill in blanks that were left by those documenting their learnings.
I’ve decided to write up the process I used to build a two node MySQL Cluster, hopefully I can fill in the gaps for others trying to make this kind of project happen for themselves. I’m building one cluster per island on a pair of servers. Inter-island replication is something I’m going to have to experiment with as the MySQL cluster docco seems to say that it gets cranky when asked to replicate over the internet.
Continue reading “Project Titanicarus: Part 6 – Building the MySQL Cluster”
This part of the project is the one I have the least experience with and the one which I’ve spent the most time trying to find a solution that works the way that I need.
To put it bluntly I don’t know if a solution exists that is capable of doing what I want with the level of simplicity I want. Almost every solution I have found has its own unique set of shortcomings, almost all of those are performance or complexity related.
I have been through several levels of insanity trying to get a viable solution implemented, including a momentary period of complete lunacy in which I planned to write my own solution.
Lets look at what I am looking for in a backend filesystem:
- Multi-chassis striping (for performance & redundancy)
- Self healing in the event of failure without admin intervention
- Able to scale up by adding more storage servers
- It must perform well with lots of small files
- It must be fast enough that web applications don’t lag
- Replication over WAN to multiple datacentres
- Capable of continuing to function when partitioned (WAN down)
- POSIX style locking (not mandatory, but ideal)
Continue reading “Project Titanicarus: Part 5 – Building the Filers or “Welcome to the Pit of Despair””
Zen Load Balancer
I am using Zen Load Balancer as the front end to all of the infrastructure in this project. It is simple, robust, provides a nice air gap between application servers and the big bad world and it also doubles as a choke point that we can apply security policies and traffic monitoring when required.
I chose Zen over better known load balancers as it can handle TCP and UDP and it will also do SSL offloading, freeing up some CPU on the app servers if we need it later on.
Zen Load Balancer will sit in front of every service we allow to be seen by the outside world – HTTP, HTTPS, DNS, SMTP, POP, IMAP. Continue reading “Project Titanicarus: Part 4 – Building Load Balancers”
PFSense is a FreeBSD firewall distro that is primarily focussed on delivering a very simple and secure firewall solution. I am using it because I’ve used it before in production environments and it proved to be a really reliable workhorse. Installation is incredibly simple, I followed the installation guide on their wiki here. I have set the box up with 2 network interfaces, one for internet access and one for internal network access. Continue reading “Project Titanicarus: Part 3 – Building the PFSense Firewall”
You will remember that each island in the design requires 9 servers. All servers are going to run Ubuntu, with the exception of the load balancers (Zen Loadbalancer Distro) and the firewall which I’ll be using PFSense for as it has a bunch of pre-built bells & whistles and a simple GUI to drive it with.
Here’s a copy of the island design to refresh your mind:
Continue reading “Project Titanicarus: Part 2 – Building the Servers”
The generous boys at Simtronic have just given me a bunch of new server capacity to stick my personal web infrastructure on, so I thought I’d have a go at building something really scalable, fault tolerant, easy to maintain and of course wildly over spec’d for what I need :-)
My current web infrastructure is a series of virtual machines all over the place (AWS, Customer/Friends Networks etc), the goal is to build myself a series of self healing “islands” that can operate independently if required or together when everything is operating ok. I hope that this will eventually become an infrastructure blueprint for other ventures I get myself tangled up in.
The name – yup it’s a mouthful, but it means something. Titanicarus was inspired by the recent Clickfrenzy debacle in which a local web hosting provider failed to properly scale their infrastructure for the hammering of a lifetime. The name is a combination of Titanic, the unsinkable ship and Icarus, the man who flew too close to the sun, melting his wings and falling to his death.
Web infrastructure needs to be stable, and able to adapt quickly. I’m trying to build this infrastructure so it can scale up and down quickly, reacting to whatever Icebergs might come our way while maintaining a reasonable cost overhead so we don’t melt our wings.
Continue reading “Project Titanicarus: Part 1 – Building a better web infrastructure”